html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538793817,IC_kwDOCGYnMM5buCFZ,9599,simonw,2023-05-08T17:55:10Z,2023-05-08T17:55:10Z,OWNER,"Confirmed - I added this test and it fails:
```python
def test_upsert_all_not_null(fresh_db):
# https://github.com/simonw/sqlite-utils/issues/538
fresh_db[""comments""].upsert_all(
[{""id"": 1, ""name"": ""Cleo""}],
pk=""id"",
not_null=[""name""],
)
assert list(fresh_db[""comments""].rows) == [{""id"": 1, ""name"": ""Cleo""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538801855,IC_kwDOCGYnMM5buEC_,9599,simonw,2023-05-08T18:00:17Z,2023-05-08T18:00:17Z,OWNER,"From time in the debugger, after creating the table it ends up doing this:
```
(Pdb) queries_and_params
[
('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]),
('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538887361,IC_kwDOCGYnMM5buY7B,9599,simonw,2023-05-08T19:01:20Z,2023-05-08T19:01:20Z,OWNER,"Here's the problem:
```python
import sqlite3
db = sqlite3.connect("":memory:"")
db.execute('create table foo (id integer primary key, name not null)')
db.execute('insert into foo (id) values (1)')
```
Produces:
```
IntegrityError: NOT NULL constraint failed: foo.name
```
But this:
```python
db.execute('insert or ignore into foo (id) values (1)')
```
Completes without an exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538889482,IC_kwDOCGYnMM5buZcK,9599,simonw,2023-05-08T19:02:38Z,2023-05-08T19:02:38Z,OWNER,"Here's the code at fault:
https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538893329,IC_kwDOCGYnMM5buaYR,9599,simonw,2023-05-08T19:04:47Z,2023-05-08T19:04:47Z,OWNER,"This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`.
One fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html
But...
> UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).
I still want to support SQLite versions earlier than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538903556,IC_kwDOCGYnMM5buc4E,9599,simonw,2023-05-08T19:11:24Z,2023-05-08T19:13:23Z,OWNER,"I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though.
This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`:
https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538910894,IC_kwDOCGYnMM5buequ,9599,simonw,2023-05-08T19:16:52Z,2023-05-08T19:17:00Z,OWNER,"How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`?
Something like this:
```python
[
('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']),
('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538921774,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538921774,IC_kwDOCGYnMM5buhUu,9599,simonw,2023-05-08T19:24:41Z,2023-05-08T19:24:41Z,OWNER,That fix seems to work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538975545,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538975545,IC_kwDOCGYnMM5buuc5,1231935,xavdid,2023-05-08T20:06:35Z,2023-05-08T20:06:35Z,NONE,"perfect, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,