issue_comments

Here's an example from the new tests:

https://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307

This is being handled in: - #520

Hey, isn't this essentially the same issue as #448 ?

Yes it is, good catch!

That original example passes against main now.

I like the lambda-{uuid} idea.

Seeing as sqlite-utils doesn't currently provide mechanisms for adding check constraints like this I'm going to leave this - I'm happy with the fix I put in for the not null constraints.

Applying the fix from the PR here doesn't fix the above problem either: - https://github.com/simonw/sqlite-utils/pull/515

So it looks like these kinds of check constraints currently aren't compatible with how upsert() works.

OK, this fails silently: python import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute('''CREATE TABLE employees ( id INTEGER PRIMARY KEY, name TEXT, age INTEGER, salary REAL, CHECK (salary is not null and salary > 0) );''') db["employees"].upsert({"id": 1, "name": "Bob"}, pk="id") list(db["employees"].rows) ```` It outputs:python [] ```

This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted.

Huh... on that basis, it's possible my fix in https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304 is incomplete. I only covered the 'not null' case.

Dupe of: - #538

I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304

This is a really neat fix, thank you.

Thanks!

I fixed this in: - #527

Will fully remove this misfeature in: - #542

Relevant commits (will mostly revert these): - https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd - https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2

Updated documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#convert

I'm going to go with --no-skip-false as the CLI option. It's ugly, but this whole thing is ugly. I'm going to make a note to remove this misfeature in sqlite-utils 4.

OK, I implemented that at the Python API level. I need to decide how it should work for the sqlite-utils convert command too: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#convert

I think the natural place to add these in the Python library API would be https://sqlite-utils.datasette.io/en/stable/python-api.html#adding-foreign-key-constraints - maybe something like this:

python db["books"].add_foreign_key("author_id", "authors", "id", on_delete=RESTRICT)

Then for the CLI tool could be added to https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key

sqlite-utils add-foreign-key my.db books author_id authors id --on-update SET_NULL I wouldn't support these for the other methods of adding foreign keys - foreign_keys(...) for the various .insert() etc methods and the add_foreign_keys(...) bulk menthod.

Wow, this is a neat feature I didn't know about. Looks like there are a bunch of options:

• NO ACTION (default)
• RESTRICT: application is prohibited from deleting a parent key when there exists one or more child keys mapped to it
• SET NULL: when a parent key is deleted the child key columns of all rows in the child table that mapped to the parent key are set to contain SQL NULL values
• SET DEFAULT: set a specific default
• CASCADE: propagates the delete or update operation on the parent key to each dependent child key

Here's an improvement: ``` % sqlite-utils insert /tmp/b.db blah /tmp/blah.txt [####################################] 100% Error: Invalid JSON - use --csv for CSV or --tsv for TSV files

JSON error: Expecting value: line 1 column 1 (char 0) ```

Was this a newline-delimited JSON file perhaps?

There are 8 failing tests left:

==== short test summary info ==== FAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Table.convert.<locals>.convert_value at 0x106bcaca0> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Table.convert.<locals>.convert_value at 0x106bc9620> ERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> ==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ==== Full traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e

On macOS this shows the same warning: % python -Wdefault $(which sqlite-utils) insert dogs.db dogs dogs.csv --csv [############------------------------] 35% [####################################] 100%/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs.csv' encoding='utf-8-sig'> insert_upsert_implementation( ResourceWarning: Enable tracemalloc to get the object allocation traceback The file itself is a click.File which is automatically closed - https://click.palletsprojects.com/en/8.1.x/api/#click.File - but it looks like it's the _io.TextIOWrapper which is not being closed:

https://github.com/simonw/sqlite-utils/blob/2376c452a56b0c3e75e7ca698273434e32945304/sqlite_utils/cli.py#L949-L956

That fix seems to work!

How about if I had logic which checked that all not-null columns were provided in the call to upsert_all() - and if they were, modified the INSERT OR IGNORE INTO to include a placeholder value for those columns that would then be fixed by the later UPDATE?

Something like this:

python [ ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ]

I could detect if this happens using cursor.rowcount - not sure how I would recover from it though.

This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in .insert_chunk():

https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878

This feels like a fundamental flaw in the way upserts are implemented by sqlite-utils.

One fix would be to switch to using the UPSERT feature in SQLite: https://www.sqlite.org/lang_UPSERT.html

But...

UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).

I still want to support SQLite versions earlier than that.

Here's the code at fault: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788

Here's the problem: python import sqlite3 db = sqlite3.connect(":memory:") db.execute('create table foo (id integer primary key, name not null)') db.execute('insert into foo (id) values (1)') Produces: IntegrityError: NOT NULL constraint failed: foo.name But this: python db.execute('insert or ignore into foo (id) values (1)') Completes without an exception.

From time in the debugger, after creating the table it ends up doing this: (Pdb) queries_and_params [ ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ]

Confirmed - I added this test and it fails: python def test_upsert_all_not_null(fresh_db): # https://github.com/simonw/sqlite-utils/issues/538 fresh_db["comments"].upsert_all( [{"id": 1, "name": "Cleo"}], pk="id", not_null=["name"], ) assert list(fresh_db["comments"].rows) == [{"id": 1, "name": "Cleo"}]