issue_comments

Shipped in 2.5 - documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#connecting-to-or-creating-a-database

I wonder if it should delete an recreate the file or if it would be safer to drop every table instead?

Dropping tables gets messy: then you need to drop triggers and views, and you need to run vacuum to clean up the space.

My worry with deleting and recreating the file is that it could trigger errors in other processes that are currently attached to that database file. But... if you know that's going to be likely, maybe you shouldn't use the recreate=True feature?

Interesting thought. I've run into this myself a lot - many of my scripts intend to create the database from scratch, so I end up running !rm /tmp/blah.db in Jupyter and occasionally getting errors if the file doesn't exist.

I think adding recreate=True could make sense. It could throw an error if you attempt to use it after passing in something other than a path to a file on disk.

Implementation plan: .insert_all() and .upsert_all() should only set .last_rowid and last_pk if they were called with a single item.

Bit trick from an implementation point of view this, since we want to be able to handle input that is a generator - so we can't scan through the input to validate that every dictionary has the same exact keys without consuming the entire iterator.

The alternative would be to raise an error the first time we spot a dictionary with keys that differ... but that's weird because we commit changes in batches, so we may end up only applying half of the changes before exiting with the error.

On that basis, I'm going to leave this as-is and mark this as wontfix.

I think I'm going to leave this as intended behaviour. Or maybe passing multiple dictionaries to .upsert_all() with different numbers of keys should raise an error?

In mucking around with sqlite3 it looks like result.lastrowid is indeed populated for UPDATE - in this case with the last inserted rowid in the table. This differs from the documented behaviour I linked to above.

``` In [1]: import sqlite3

In [2]: c = sqlite3.connect(":memory:")

In [3]: c
Out[3]: <sqlite3.Connection at 0x103c4d490>

In [4]: c.execute('create table foo (bar integer);')
Out[4]: <sqlite3.Cursor at 0x103f760a0>

In [5]: c.execute('insert into foo (bar) values (1)')
Out[5]: <sqlite3.Cursor at 0x103f76650>

In [6]: c.execute('select * from foo').fetchall()
Out[6]: [(1,)]

In [7]: c.execute('insert into foo (bar) values (1)')
Out[7]: <sqlite3.Cursor at 0x103f766c0>

In [8]: c.execute('select * from foo').fetchall()
Out[8]: [(1,), (1,)]

In [9]: c.execute('insert into foo (bar) values (1)').lastrowid
Out[9]: 3

In [10]: c.execute('select * from foo').fetchall()
Out[10]: [(1,), (1,), (1,)]

In [11]: c.execute('select rowid, bar from foo').fetchall()
Out[11]: [(1, 1), (2, 1), (3, 1)]

In [12]: c.execute('insert into foo (bar) values (1)').lastrowid
Out[12]: 4

In [13]: c.execute('select rowid, bar from foo').fetchall()
Out[13]: [(1, 1), (2, 1), (3, 1), (4, 1)]

In [14]: r = c.execute('update foo set bar =2 where rowid = 1')

In [15]: r.lastrowid
Out[15]: 4

In [16]: c.execute('select rowid, bar from foo').fetchall()
Out[16]: [(1, 2), (2, 1), (3, 1), (4, 1)]

In [17]: r = c.execute('select rowid, bar from foo')

In [18]: r.fetchall()
Out[18]: [(1, 2), (2, 1), (3, 1), (4, 1)]

In [19]: r.lastrowid
Out[19]: 4 ```

Why do I even care about lastrowid here?

I'm trying to ensure that after you insert or upsert a row you can use table.last_pk to start doing things like building additional foreign key relationships.

So maybe it doesn't make sense to make .last_pk available at all for cases where you called .upsert_all() or .insert_all() - it should just be populated for .upsert() and .insert().

The documentation doesn't say it should work for .upsert_all() - it's only documented for the single actions.

https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124

I have a hunch that the root of the problem here is that accessing result.lastrowid during my version of an .upsert() doesn't actually make sense:

https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113

In the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point:

(Pdb) query 'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?' (Pdb) params ['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv'] (Pdb) result.lastrowid 100 But here's the weird thing... there's no row in the table with a rowid of 100! (Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] So what the heck is going on?

The last SQL statement I executed here was an UPDATE. The lastrowid docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid

This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None.

So where did that 100 come from? It should be None!