issue_comments

That fix seems to work!

How about if I had logic which checked that all not-null columns were provided in the call to upsert_all() - and if they were, modified the INSERT OR IGNORE INTO to include a placeholder value for those columns that would then be fixed by the later UPDATE?

Something like this:

python [ ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ]

I could detect if this happens using cursor.rowcount - not sure how I would recover from it though.

This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in .insert_chunk():

https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878

This feels like a fundamental flaw in the way upserts are implemented by sqlite-utils.

One fix would be to switch to using the UPSERT feature in SQLite: https://www.sqlite.org/lang_UPSERT.html

But...

UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).

I still want to support SQLite versions earlier than that.

Here's the code at fault: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788

Here's the problem: python import sqlite3 db = sqlite3.connect(":memory:") db.execute('create table foo (id integer primary key, name not null)') db.execute('insert into foo (id) values (1)') Produces: IntegrityError: NOT NULL constraint failed: foo.name But this: python db.execute('insert or ignore into foo (id) values (1)') Completes without an exception.

From time in the debugger, after creating the table it ends up doing this: (Pdb) queries_and_params [ ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ]

Confirmed - I added this test and it fails: python def test_upsert_all_not_null(fresh_db): # https://github.com/simonw/sqlite-utils/issues/538 fresh_db["comments"].upsert_all( [{"id": 1, "name": "Cleo"}], pk="id", not_null=["name"], ) assert list(fresh_db["comments"].rows) == [{"id": 1, "name": "Cleo"}]