html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/73#issuecomment-571138093,https://api.github.com/repos/simonw/sqlite-utils/issues/73,571138093,MDEyOklzc3VlQ29tbWVudDU3MTEzODA5Mw==,82988,2020-01-06T13:28:31Z,2020-01-06T13:28:31Z,NONE,"I think I actually had several issues in play... The missing key was one, but I think there is also an issue as per below. For example, in the following: ```python def init_testdb(dbname='test.db'): if os.path.exists(dbname): os.remove(dbname) conn = sqlite3.connect(dbname) db = Database(conn) return conn, db conn, db = init_testdb() c = conn.cursor() c.executescript('CREATE TABLE ""test1"" (""Col1"" TEXT, ""Col2"" TEXT, PRIMARY KEY (""Col1""));') c.executescript('CREATE TABLE ""test2"" (""Col1"" TEXT, ""Col2"" TEXT, PRIMARY KEY (""Col1""));') print('Test 1...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) print('Test 2...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, {'Col1':'c','Col2':'x'}], pk=('Col1')) print('Done...') --------------------------------------------------------------------------- Test 1... Test 2... IndexError: list index out of range --------------------------------------------------------------------------- IndexError Traceback (most recent call last) in 22 print('Test 2...') 23 for i in range(3): ---> 24 db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) 25 db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, 26 {'Col1':'c','Col2':'x'}], pk=('Col1')) /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts) 1157 alter=alter, 1158 extracts=extracts, -> 1159 upsert=True, 1160 ) 1161 /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert) 1097 # self.last_rowid will be 0 if a ""INSERT OR IGNORE"" happened 1098 if (hash_id or pk) and self.last_rowid: -> 1099 row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0] 1100 if hash_id: 1101 self.last_pk = row[hash_id] IndexError: list index out of range ``` the first test works but the second fails. Is the length of the list of items being upserted leaking somewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-573047321,https://api.github.com/repos/simonw/sqlite-utils/issues/73,573047321,MDEyOklzc3VlQ29tbWVudDU3MzA0NzMyMQ==,82988,2020-01-10T14:02:56Z,2020-01-10T14:09:23Z,NONE,"Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop). Could it be sqlite3? I'm on 3.30.1. UPDATE: just tried it on jupyter.org/try and I get the error there, too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-580745213,https://api.github.com/repos/simonw/sqlite-utils/issues/73,580745213,MDEyOklzc3VlQ29tbWVudDU4MDc0NTIxMw==,82988,2020-01-31T14:02:38Z,2020-01-31T14:21:09Z,NONE,"So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error. If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly. It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/pull/203#issuecomment-1033641009,https://api.github.com/repos/simonw/sqlite-utils/issues/203,1033641009,IC_kwDOCGYnMM49nBwx,82988,2022-02-09T11:06:18Z,2022-02-09T11:06:18Z,NONE,"Is there any progress elsewhere on the handling of compound / composite foreign keys, or is this PR still effectively open?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1041313679,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1041313679,IC_kwDOCGYnMM4-ES-P,82988,2022-02-16T09:59:51Z,2022-02-16T10:00:10Z,NONE,"The `CustomColumnType()` approach looks good. This pushes you into the mindspace that you are defining and working with a custom column type. When creating the table, you could then error, or at least warn, if someone wasn't setting a column on a `type` or a custom column type, which I guess is where `mypy` comes in?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1041325398,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1041325398,IC_kwDOCGYnMM4-EV1W,82988,2022-02-16T10:12:48Z,2022-02-16T10:18:55Z,NONE,"> My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns Other possible pairs: unconventional date/datetime and timezone pairs eg `2022-02-16::17.00, London`; or more generally, numerical value and unit of measurement pairs (eg if you want to cast into and out of different measurement units using packages like `pint`) or currencies etc. Actually, in that case, I guess you may be presenting things that are unit typed already, and so a conversion would need to parse things into an appropriate, possibly two column `value, unit` format. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1041363433,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1041363433,IC_kwDOCGYnMM4-EfHp,82988,2022-02-16T10:57:03Z,2022-02-16T10:57:19Z,NONE,"Wondering if this actually relates to https://github.com/simonw/sqlite-utils/issues/402 ? I also wonder if this would be a sensible approach for eg registering `pint` based quantity conversions into and out of the db, perhaps storing the quantity as a serialised `magnitude measurement` single column string?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1248440137,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1248440137,IC_kwDOCGYnMM5Kaa9J,82988,2022-09-15T18:13:50Z,2022-09-15T18:13:50Z,NONE,"I was wondering if you have any more thoughts on this? I have a tangible use case now: adding a ""vector"" column to a database to support semantic search using doc2vec embeddings ([example](https://psychemedia.github.io/storynotes/Lang_Doc2Vec.html); note that the `vtfunc` package may no longer be reliable...).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,