html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727400,https://api.github.com/repos/simonw/sqlite-utils/issues/99,612727400,MDEyOklzc3VlQ29tbWVudDYxMjcyNzQwMA==,9599,2020-04-13T03:03:09Z,2020-04-13T03:03:09Z,OWNER,I think I'm going to leave this as intended behaviour. Or maybe passing multiple dictionaries to `.upsert_all()` with different numbers of keys should raise an error?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598640234, https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727814,https://api.github.com/repos/simonw/sqlite-utils/issues/99,612727814,MDEyOklzc3VlQ29tbWVudDYxMjcyNzgxNA==,9599,2020-04-13T03:05:04Z,2020-04-13T03:05:04Z,OWNER,"Bit trick from an implementation point of view this, since we want to be able to handle input that is a generator - so we can't scan through the input to validate that every dictionary has the same exact keys without consuming the entire iterator. The alternative would be to raise an error the first time we spot a dictionary with keys that differ... but that's weird because we commit changes in batches, so we may end up only applying half of the changes before exiting with the error. On that basis, I'm going to leave this as-is and mark this as wontfix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598640234, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612173156,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612173156,MDEyOklzc3VlQ29tbWVudDYxMjE3MzE1Ng==,9599,2020-04-10T19:03:32Z,2020-04-10T23:08:28Z,OWNER,"Investigate this traceback: ``` Traceback (most recent call last): File ""fetch_projects.py"", line 60, in fetch_projects(db, token) File ""fetch_projects.py"", line 41, in fetch_projects db[""projects""].upsert(project, pk=""id"") File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1139, in upsert conversions=conversions, File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1168, in upsert_all upsert=True, File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1107, in insert_all row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0] IndexError: list index out of range ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612258687,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612258687,MDEyOklzc3VlQ29tbWVudDYxMjI1ODY4Nw==,9599,2020-04-10T23:08:48Z,2020-04-10T23:08:48Z,OWNER,I need a test that reproduces this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707293,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612707293,MDEyOklzc3VlQ29tbWVudDYxMjcwNzI5Mw==,9599,2020-04-13T01:21:22Z,2020-04-13T01:21:22Z,OWNER,"I have a hunch that the root of the problem here is that accessing `result.lastrowid` during my version of an `.upsert()` doesn't actually make sense: https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113 In the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point: ``` (Pdb) query 'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?' (Pdb) params ['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv'] (Pdb) result.lastrowid 100 ``` But here's the weird thing... there's no row in the table with a rowid of 100! ``` (Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] ``` So what the heck is going on? The last SQL statement I executed here was an `UPDATE`. The `lastrowid` docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid > This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None. So where did that `100` come from? It should be `None`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707828,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612707828,MDEyOklzc3VlQ29tbWVudDYxMjcwNzgyOA==,9599,2020-04-13T01:24:05Z,2020-04-13T01:24:16Z,OWNER,"Why do I even care about `lastrowid` here? I'm trying to ensure that after you insert or upsert a row you can use `table.last_pk` to start doing things like building additional foreign key relationships. So maybe it doesn't make sense to make `.last_pk` available _at all_ for cases where you called `.upsert_all()` or `.insert_all()` - it should just be populated for `.upsert()` and `.insert()`. The documentation doesn't say it should work for `.upsert_all()` - it's only documented for the single actions. https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612708274,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612708274,MDEyOklzc3VlQ29tbWVudDYxMjcwODI3NA==,9599,2020-04-13T01:25:59Z,2020-04-13T01:26:11Z,OWNER,"In mucking around with `sqlite3` it looks like `result.lastrowid` is indeed populated for `UPDATE` - in this case with the last inserted rowid in the table. This differs from the documented behaviour I linked to above. ``` In [1]: import sqlite3 In [2]: c = sqlite3.connect("":memory:"") In [3]: c Out[3]: In [4]: c.execute('create table foo (bar integer);') Out[4]: In [5]: c.execute('insert into foo (bar) values (1)') Out[5]: In [6]: c.execute('select * from foo').fetchall() Out[6]: [(1,)] In [7]: c.execute('insert into foo (bar) values (1)') Out[7]: In [8]: c.execute('select * from foo').fetchall() Out[8]: [(1,), (1,)] In [9]: c.execute('insert into foo (bar) values (1)').lastrowid Out[9]: 3 In [10]: c.execute('select * from foo').fetchall() Out[10]: [(1,), (1,), (1,)] In [11]: c.execute('select rowid, bar from foo').fetchall() Out[11]: [(1, 1), (2, 1), (3, 1)] In [12]: c.execute('insert into foo (bar) values (1)').lastrowid Out[12]: 4 In [13]: c.execute('select rowid, bar from foo').fetchall() Out[13]: [(1, 1), (2, 1), (3, 1), (4, 1)] In [14]: r = c.execute('update foo set bar =2 where rowid = 1') In [15]: r.lastrowid Out[15]: 4 In [16]: c.execute('select rowid, bar from foo').fetchall() Out[16]: [(1, 2), (2, 1), (3, 1), (4, 1)] In [17]: r = c.execute('select rowid, bar from foo') In [18]: r.fetchall() Out[18]: [(1, 2), (2, 1), (3, 1), (4, 1)] In [19]: r.lastrowid Out[19]: 4 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612728047,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612728047,MDEyOklzc3VlQ29tbWVudDYxMjcyODA0Nw==,9599,2020-04-13T03:06:10Z,2020-04-13T03:06:10Z,OWNER,Implementation plan: `.insert_all()` and `.upsert_all()` should only set `.last_rowid` and `last_pk` if they were called with a single item.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-928790381,https://api.github.com/repos/simonw/sqlite-utils/issues/98,928790381,IC_kwDOCGYnMM43XDdt,36834097,2021-09-28T04:38:44Z,2021-09-28T04:38:44Z,NONE,"Hi @simonw - wondering if you might be able to shed some light here. I've seemed to reproduce this issue. Here's the stacktrace: ``` ... db[""potholes""].insert(pothole, pk='id', alter=True, replace=True) ... Traceback (most recent call last): File """", line 3, in File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2481, in insert return self.insert_all( File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2596, in insert_all self.insert_chunk( File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2424, in insert_chunk row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0] IndexError: list index out of range ``` Interesting enough, I found that omitting the `pk` param does not throw the error. Let me know how I can help out! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518, https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732129,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612732129,MDEyOklzc3VlQ29tbWVudDYxMjczMjEyOQ==,9599,2020-04-13T03:25:29Z,2020-04-13T03:25:29Z,OWNER,"Interesting thought. I've run into this myself a lot - many of my scripts intend to create the database from scratch, so I end up running `!rm /tmp/blah.db` in Jupyter and occasionally getting errors if the file doesn't exist. I think adding `recreate=True` could make sense. It could throw an error if you attempt to use it after passing in something other than a path to a file on disk.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293, https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732453,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612732453,MDEyOklzc3VlQ29tbWVudDYxMjczMjQ1Mw==,9599,2020-04-13T03:26:46Z,2020-04-13T03:26:46Z,OWNER,"I wonder if it should delete an recreate the file or if it would be safer to drop every table instead? Dropping tables gets messy: then you need to drop triggers and views, and you need to run `vacuum` to clean up the space. My worry with deleting and recreating the file is that it could trigger errors in other processes that are currently attached to that database file. But... if you know that's going to be likely, maybe you shouldn't use the `recreate=True` feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293, https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612738311,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612738311,MDEyOklzc3VlQ29tbWVudDYxMjczODMxMQ==,9599,2020-04-13T03:55:11Z,2020-04-13T03:55:11Z,OWNER,Shipped in 2.5 - documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#connecting-to-or-creating-a-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293, https://github.com/simonw/sqlite-utils/issues/97#issuecomment-614073859,https://api.github.com/repos/simonw/sqlite-utils/issues/97,614073859,MDEyOklzc3VlQ29tbWVudDYxNDA3Mzg1OQ==,1448859,2020-04-15T14:29:30Z,2020-04-15T14:29:30Z,NONE,"Woah! Thanks a lot. Next time I will add a more obvious/explicit ""if you like this idea let me know I'd love to work on it to get my feet wet here"" :D","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293, https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394349,https://api.github.com/repos/simonw/sqlite-utils/issues/96,606394349,MDEyOklzc3VlQ29tbWVudDYwNjM5NDM0OQ==,9599,2020-03-31T04:37:16Z,2020-03-31T04:37:16Z,OWNER,Test failure was just a Black formatting issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",589801352, https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394619,https://api.github.com/repos/simonw/sqlite-utils/issues/96,606394619,MDEyOklzc3VlQ29tbWVudDYwNjM5NDYxOQ==,9599,2020-03-31T04:38:17Z,2020-03-31T04:40:23Z,OWNER,I wonder if there are any other Pandas conversions we should be doing? https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#dtypes ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",589801352, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125455,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599125455,MDEyOklzc3VlQ29tbWVudDU5OTEyNTQ1NQ==,9599,2020-03-14T19:34:35Z,2020-03-14T19:34:35Z,OWNER,"From https://www.sqlite.org/datatype3.html it looks like `FLOAT` is a supported keyword for creating tables but `REAL` is the correct keyword. So actually `sqlite-utils` gets this wrong, because when we create a table we turn Python `float` values into a `FLOAT` column. Looks like the correct behaviour would be to turn them into a `REAL` column. https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L28-L48","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125557,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599125557,MDEyOklzc3VlQ29tbWVudDU5OTEyNTU1Nw==,9599,2020-03-14T19:35:29Z,2020-03-14T19:35:29Z,OWNER,"Fixing that would technically constitute a breaking change for library consumers, so it should be a major version release. I'm not inclined to release `3.0` just for this one issue, so I'm going to hold back on fixing that and address the smaller issue in this bug as a dot release instead for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599126831,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599126831,MDEyOklzc3VlQ29tbWVudDU5OTEyNjgzMQ==,9599,2020-03-14T19:45:28Z,2020-03-14T19:45:28Z,OWNER,"Turns out there are a TON of valid column definitions that aren't being considered yet - https://www.sqlite.org/datatype3.html#affinity_name_examples - stuff like `VARYING CHARACTER(255)` and `DECIMAL(10,5)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127197,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599127197,MDEyOklzc3VlQ29tbWVudDU5OTEyNzE5Nw==,9599,2020-03-14T19:48:06Z,2020-03-14T19:48:06Z,OWNER,"Actually it looks like I should implement the exact rules described in https://www.sqlite.org/datatype3.html#determination_of_column_affinity > The affinity of a column is determined by the declared type of the column, according to the following rules in the order shown: > > 1. If the declared type contains the string ""INT"" then it is assigned INTEGER affinity. > 2. If the declared type of the column contains any of the strings ""CHAR"", ""CLOB"", or ""TEXT"" then that column has TEXT affinity. Notice that the type VARCHAR contains the string ""CHAR"" and is thus assigned TEXT affinity. > 3. If the declared type for a column contains the string ""BLOB"" or if no type is specified then the column has affinity BLOB. > 4. If the declared type for a column contains any of the strings ""REAL"", ""FLOA"", or ""DOUB"" then the column has REAL affinity. > 5. Otherwise, the affinity is NUMERIC. > > Note that the order of the rules for determining column affinity is important. A column whose declared type is ""CHARINT"" will match both rules 1 and 2 but the first rule takes precedence and so the column affinity will be INTEGER.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127453,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599127453,MDEyOklzc3VlQ29tbWVudDU5OTEyNzQ1Mw==,9599,2020-03-14T19:50:08Z,2020-03-14T19:50:08Z,OWNER,"> If the declared type for a column contains the string ""BLOB"" or if no type is specified then the column has affinity BLOB I currently treat those as `str` - it sounds like I should treat them as `bytes`: https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L69 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599128891,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599128891,MDEyOklzc3VlQ29tbWVudDU5OTEyODg5MQ==,9599,2020-03-14T20:03:45Z,2020-03-14T20:03:45Z,OWNER,I'm going to keep treating them as `str`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599247833,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599247833,MDEyOklzc3VlQ29tbWVudDU5OTI0NzgzMw==,9599,2020-03-15T18:37:28Z,2020-03-15T18:37:43Z,OWNER,Released in 2.4.2.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961, https://github.com/simonw/sqlite-utils/issues/91#issuecomment-723350956,https://api.github.com/repos/simonw/sqlite-utils/issues/91,723350956,MDEyOklzc3VlQ29tbWVudDcyMzM1MDk1Ng==,9599,2020-11-06T23:53:25Z,2020-11-06T23:53:25Z,OWNER,"This is now possible, for both FTS4 and FTS5 - see #197.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",577302229, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-593122605,https://api.github.com/repos/simonw/sqlite-utils/issues/89,593122605,MDEyOklzc3VlQ29tbWVudDU5MzEyMjYwNQ==,35075,2020-03-01T17:33:11Z,2020-03-01T17:33:11Z,NONE,"If you're happy with the proposed implementation, I have code & tests written that I'll get ready for a PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-615515867,https://api.github.com/repos/simonw/sqlite-utils/issues/89,615515867,MDEyOklzc3VlQ29tbWVudDYxNTUxNTg2Nw==,9599,2020-04-18T00:00:41Z,2020-04-18T00:00:41Z,OWNER,"Yes pleas, I'd love to see that pull request!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242,https://api.github.com/repos/simonw/sqlite-utils/issues/89,710460242,MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==,9599,2020-10-16T19:17:27Z,2020-10-16T19:17:50Z,OWNER,"I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works: ```python fresh_db.table(""tree"", extracts=[Extract( columns=(""CommonName"", ""LatinName""), table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} )]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548, https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591768604,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591768604,MDEyOklzc3VlQ29tbWVudDU5MTc2ODYwNA==,9599,2020-02-27T04:03:03Z,2020-02-27T04:03:03Z,OWNER,"`drop table resources_fts` drops the FTS table and the other ones that it created (resources_fts_data, resources_fts_idx, resources_fts_docsize, resources_fts_config) - but keeps the triggers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300, https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769046,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769046,MDEyOklzc3VlQ29tbWVudDU5MTc2OTA0Ng==,9599,2020-02-27T04:05:15Z,2020-02-27T04:05:15Z,OWNER,I can reliably get the list of triggers to delete from `select name from sqlite_master where type = 'trigger' and tbl_name = 'resources';`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300, https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769171,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769171,MDEyOklzc3VlQ29tbWVudDU5MTc2OTE3MQ==,9599,2020-02-27T04:05:58Z,2020-02-27T04:26:31Z,OWNER,"Strange - https://www.sqlite.org/lang_droptrigger.html says ""Note that triggers are automatically dropped when the associated table is dropped"" but that doesn't seem to be true in my experimenting. UPDATE: no that makes sense - the triggers are on `resources` which still exists, it was `resources_fts` that was dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300, https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769373,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769373,MDEyOklzc3VlQ29tbWVudDU5MTc2OTM3Mw==,9599,2020-02-27T04:06:47Z,2020-02-27T04:06:47Z,OWNER,Looks like safest option is to loop through those trigger names and run `DROP TRIGGER IF EXISTS foo` on each one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300, https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769759,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769759,MDEyOklzc3VlQ29tbWVudDU5MTc2OTc1OQ==,9599,2020-02-27T04:08:29Z,2020-02-27T04:08:29Z,OWNER,"I think the method should be called `table.disable_fts()` - the opposite of `table.enable_fts(...)`. There should be a `sqlite-utils disable-fts database.db tablename` command to match it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300, https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661250,https://api.github.com/repos/simonw/sqlite-utils/issues/87,586661250,MDEyOklzc3VlQ29tbWVudDU4NjY2MTI1MA==,9599,2020-02-16T02:19:33Z,2020-02-16T02:19:33Z,OWNER,"Here's the code: https://github.com/simonw/sqlite-utils/blob/5e0000609f9be6efafea1b96f610988eb18d6d89/sqlite_utils/utils.py#L18-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",565837965, https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661276,https://api.github.com/repos/simonw/sqlite-utils/issues/87,586661276,MDEyOklzc3VlQ29tbWVudDU4NjY2MTI3Ng==,9599,2020-02-16T02:20:14Z,2020-02-16T02:20:14Z,OWNER,"`OrderedDict` is actually a subclass of `dict` - so a smart fix would be for this logic to check and see if the type `t` is a subclass of one of `list`, `tuple` or `dict`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",565837965, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586661934,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586661934,MDEyOklzc3VlQ29tbWVudDU4NjY2MTkzNA==,9599,2020-02-16T02:33:07Z,2020-02-16T02:33:07Z,OWNER,"Thanks for the example file - looks like it can be trimmed down to just these two lines to replicate the bug: ```csv ""MTU (CET)"",""Day-ahead Price [EUR/MWh]"" ""01.01.2016 00:00 - 01.01.2016 01:00"",""23.86"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586662404,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586662404,MDEyOklzc3VlQ29tbWVudDU4NjY2MjQwNA==,9599,2020-02-16T02:43:12Z,2020-02-16T02:43:12Z,OWNER,"https://stackoverflow.com/a/22694438 looks like the answer: > When using square brackets, it is not possible to have these characters in the identifier. > > When using double quotes, you can escape them in the name by doubling them: > > `CREATE TABLE ""hello """"world""""""(key INTEGER PRIMARY KEY);`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676640,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586676640,MDEyOklzc3VlQ29tbWVudDU4NjY3NjY0MA==,9599,2020-02-16T07:16:31Z,2020-02-16T07:16:31Z,OWNER,"There's something weird about this. I created a test database file like so: ``` sqlite3 /tmp/demo.db < .schema CREATE TABLE IF NOT EXISTS ""data"" ( ""MTU (CET)"" TEXT, ""Day-ahead Price [EUR/MWh]"" TEXT ); sqlite> .headers on sqlite> select * from data; MTU (CET)|Day-ahead Price [EUR/MWh] 01.01.2016 00:00 - 01.01.2016 01:00|23.86 sqlite> ``` BUT... if I open the same database in Python, something weird happens: ``` In [1]: import sqlite3 In [2]: conn = sqlite3.connect(""/tmp/demo.db"") In [3]: cursor = conn.cursor() In [4]: cursor.execute(""select * from data"") Out[4]: In [5]: cursor.fetchall() Out[5]: [('01.01.2016 00:00 - 01.01.2016 01:00', '23.86')] In [6]: cursor.description Out[6]: (('MTU (CET)', None, None, None, None, None, None), ('Day-ahead Price', None, None, None, None, None, None)) In [7]: conn.row_factory = sqlite3.Row In [8]: cursor = conn.cursor() In [9]: cursor.execute(""select * from data"") Out[9]: In [10]: row = cursor.fetchall() In [12]: row Out[12]: In [15]: row.keys() Out[15]: ['MTU (CET)', 'Day-ahead Price'] ``` Note that in `cursor.description` AND in `row.keys()` above the second column is displayed as `'Day-ahead Price'` - when we would expect it to be displayed as `Day-ahead Price [EUR/MWh]` So.... it looks like there may be a bug in Python's `sqlite3` module where columns with square braces in them have that portion of the name stripped out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586676856,MDEyOklzc3VlQ29tbWVudDU4NjY3Njg1Ng==,9599,2020-02-16T07:20:34Z,2020-02-16T07:20:34Z,OWNER,"I'm not sure what to do about this one. I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again. So... I could do one of the following: - Throw an error if a column name includes those characters. That's my preferred option I think. - Automatically replace `[` in column names with `(` and `]` with `)` - Do the automatic replacement but show a user-visible warning when I do it - Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix. Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586683572,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586683572,MDEyOklzc3VlQ29tbWVudDU4NjY4MzU3Mg==,8149512,2020-02-16T09:03:54Z,2020-02-16T09:03:54Z,NONE,"Probably the best option to just throw the error. Is there any active dev chan where we could post the issue to python sqlite3?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586729798,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586729798,MDEyOklzc3VlQ29tbWVudDU4NjcyOTc5OA==,9599,2020-02-16T17:11:02Z,2020-02-16T17:11:02Z,OWNER,I filed a bug in the Python issue tracker here: https://bugs.python.org/issue39652,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-591770623,https://api.github.com/repos/simonw/sqlite-utils/issues/86,591770623,MDEyOklzc3VlQ29tbWVudDU5MTc3MDYyMw==,9599,2020-02-27T04:12:39Z,2020-02-27T04:12:39Z,OWNER,"I pushed a branch with my experiment in it, but I'm going to fix this by throwing an error on `[` or `]` in a column name instead - I won't implement the changes from that branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430, https://github.com/simonw/sqlite-utils/issues/85#issuecomment-584426938,https://api.github.com/repos/simonw/sqlite-utils/issues/85,584426938,MDEyOklzc3VlQ29tbWVudDU4NDQyNjkzOA==,9599,2020-02-11T00:35:09Z,2020-02-11T00:35:09Z,OWNER,"Here's why: https://github.com/simonw/sqlite-utils/blob/0c2451e0690c5f4e6463a2f339b0a280e30ed806/sqlite_utils/db.py#L627-L636","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",562911863, https://github.com/simonw/sqlite-utils/issues/83#issuecomment-583789015,https://api.github.com/repos/simonw/sqlite-utils/issues/83,583789015,MDEyOklzc3VlQ29tbWVudDU4Mzc4OTAxNQ==,9599,2020-02-08T23:58:35Z,2020-02-08T23:58:35Z,OWNER,Shipped as 2.3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559374410, https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581651409,https://api.github.com/repos/simonw/sqlite-utils/issues/82,581651409,MDEyOklzc3VlQ29tbWVudDU4MTY1MTQwOQ==,9599,2020-02-03T22:32:41Z,2020-02-03T22:32:41Z,OWNER,This should work - the data should be chunked automatically. It looks like this is a bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745, https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581652388,https://api.github.com/repos/simonw/sqlite-utils/issues/82,581652388,MDEyOklzc3VlQ29tbWVudDU4MTY1MjM4OA==,9599,2020-02-03T22:35:44Z,2020-02-03T22:35:44Z,OWNER,"I can't replicate this problem: ``` /tmp $ sqlite-utils --version sqlite-utils, version 2.2 /tmp $ curl ""https://data.nasa.gov/resource/y77d-th95.json"" | sqlite-utils insert meteorites.db meteorites - --pk=id % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 240k 0 240k 0 0 185k 0 --:--:-- 0:00:01 --:--:-- 185k ``` Could you run `sqlite-utils --version` and tell me what you get?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745, https://github.com/simonw/sqlite-utils/issues/82#issuecomment-591771532,https://api.github.com/repos/simonw/sqlite-utils/issues/82,591771532,MDEyOklzc3VlQ29tbWVudDU5MTc3MTUzMg==,9599,2020-02-27T04:16:30Z,2020-02-27T04:16:30Z,OWNER,Closing as can't reproduce.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745, https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071010,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071010,MDEyOklzc3VlQ29tbWVudDU4MTA3MTAxMA==,9599,2020-02-01T21:27:00Z,2020-02-01T21:27:00Z,OWNER,"Here's the current method: https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/sqlite_utils/db.py#L823-L845 If I make it a utility function instead of a class method I could ensure it is directly importable like so: ```python from sqlite_utils import detect_column_types ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274, https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071116,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071116,MDEyOklzc3VlQ29tbWVudDU4MTA3MTExNg==,9599,2020-02-01T21:28:35Z,2020-02-01T21:28:53Z,OWNER,"Should I keep `table.detect_column_types()` working so as not to break existing code? If it was part of the documented API then I wouldn't break that without bumping to 3.x. Since it's undocumented I'm going to make it as a breaking change instead (and bump the `geojson-to-sqlite` dependency version).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274, https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071235,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071235,MDEyOklzc3VlQ29tbWVudDU4MTA3MTIzNQ==,9599,2020-02-01T21:30:09Z,2020-02-01T21:30:09Z,OWNER,Actually I'll put it in the `utils.py` module.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274, https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071434,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071434,MDEyOklzc3VlQ29tbWVudDU4MTA3MTQzNA==,9599,2020-02-01T21:32:34Z,2020-02-01T21:32:34Z,OWNER,While I'm at it I think I'll rename it to `suggest_column_types` - it's not really detecting them since the input is just a list of dictionaries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567505,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567505,MDEyOklzc3VlQ29tbWVudDU4MDU2NzUwNQ==,9599,2020-01-31T03:39:19Z,2020-01-31T03:39:19Z,OWNER,"Still needs documentation and tests. Also I'm not certain that this should be an argument you can pass to the `.table()` constructor, need to think that over.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567604,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567604,MDEyOklzc3VlQ29tbWVudDU4MDU2NzYwNA==,9599,2020-01-31T03:39:58Z,2020-01-31T03:39:58Z,OWNER,Perhaps this should be called `after_create` instead of `on_create`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567886,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567886,MDEyOklzc3VlQ29tbWVudDU4MDU2Nzg4Ng==,9599,2020-01-31T03:41:31Z,2020-01-31T03:41:31Z,OWNER,I think it does make sense to be able to pass it to the `.table()` constructor.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580569059,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580569059,MDEyOklzc3VlQ29tbWVudDU4MDU2OTA1OQ==,9599,2020-01-31T03:48:41Z,2020-01-31T03:48:41Z,OWNER,"This may not be the right feature after all, see https://github.com/simonw/geojson-to-sqlite/issues/6#issuecomment-580569002","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580584269,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580584269,MDEyOklzc3VlQ29tbWVudDU4MDU4NDI2OQ==,9599,2020-01-31T05:08:04Z,2020-01-31T05:08:04Z,OWNER,Ditching this since it won't actually solve my problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819, https://github.com/simonw/sqlite-utils/issues/8#issuecomment-464341721,https://api.github.com/repos/simonw/sqlite-utils/issues/8,464341721,MDEyOklzc3VlQ29tbWVudDQ2NDM0MTcyMQ==,82988,2019-02-16T12:08:41Z,2019-02-16T12:08:41Z,NONE,We also get an error if a column name contains a `.`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403922644, https://github.com/simonw/sqlite-utils/issues/8#issuecomment-466695500,https://api.github.com/repos/simonw/sqlite-utils/issues/8,466695500,MDEyOklzc3VlQ29tbWVudDQ2NjY5NTUwMA==,9599,2019-02-23T21:09:03Z,2019-02-23T21:09:03Z,OWNER,"Fixed in https://github.com/simonw/sqlite-utils/commit/228d595f7d10994f34e948888093c2cd290267c4 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403922644, https://github.com/simonw/sqlite-utils/issues/8#issuecomment-482994231,https://api.github.com/repos/simonw/sqlite-utils/issues/8,482994231,MDEyOklzc3VlQ29tbWVudDQ4Mjk5NDIzMQ==,82988,2019-04-14T15:04:07Z,2019-04-14T15:29:33Z,NONE," PLEASE IGNORE THE BELOW... I did a package update and rebuilt the kernel I was working in... may just have been an old version of sqlite_utils, seems to be working now. (Too many containers / too many environments!) Has an issue been reintroduced here with FTS? eg I'm getting an error thrown by spaces in column names here: ``` /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) def enable_fts(self, columns, fts_version=""FTS5""): --> 329 ""Enables FTS on the specified columns"" 330 sql = """""" 331 CREATE VIRTUAL TABLE ""{table}_fts"" USING {fts_version} ( ``` when trying an `insert_all`. Also, if a col has a `.` in it, I seem to get: ``` /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid OperationalError: near ""."": syntax error ``` (Can't post a worked minimal example right now; racing trying to build something against a live timing screen that will stop until next weekend in an hour or two...) PS Hmmm I did a test and they seem to work; I must be messing up s/where else... ``` import sqlite3 from sqlite_utils import Database dbname='testingDB_sqlite_utils.db' #!rm $dbname conn = sqlite3.connect(dbname, timeout=10) #Setup database tables c = conn.cursor() setup=''' CREATE TABLE IF NOT EXISTS ""test1"" ( ""NO"" INTEGER, ""NAME"" TEXT ); CREATE TABLE IF NOT EXISTS ""test2"" ( ""NO"" INTEGER, `TIME OF DAY` TEXT ); CREATE TABLE IF NOT EXISTS ""test3"" ( ""NO"" INTEGER, `AVG. SPEED (MPH)` FLOAT ); ''' c.executescript(setup) DB = Database(conn) import pandas as pd df1 = pd.DataFrame({'NO':[1,2],'NAME':['a','b']}) DB['test1'].insert_all(df1.to_dict(orient='records')) df2 = pd.DataFrame({'NO':[1,2],'TIME OF DAY':['early on','late']}) DB['test2'].insert_all(df2.to_dict(orient='records')) df3 = pd.DataFrame({'NO':[1,2],'AVG. SPEED (MPH)':['123.3','123.4']}) DB['test3'].insert_all(df3.to_dict(orient='records')) ``` all seem to work ok. I'm still getting errors in my set up though, which is not too different to the text cases?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403922644, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012158895,IC_kwDOCGYnMM48VFGv,25778,2022-01-13T13:55:59Z,2022-01-13T13:55:59Z,CONTRIBUTOR,"Came here to add this. I might pick it up. Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012230212,IC_kwDOCGYnMM48VWhE,25778,2022-01-13T15:15:13Z,2022-01-13T15:15:13Z,CONTRIBUTOR,"Some proposals I'd add to sqlite-utils: Some version of this, from [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite/blob/main/geojson_to_sqlite/utils.py#L124-L130): ```python def init_spatialite(db, lib): db.conn.enable_load_extension(True) db.conn.load_extension(lib) # Initialize SpatiaLite if not yet initialized if ""spatial_ref_sys"" in db.table_names(): return db.conn.execute(""select InitSpatialMetadata(1)"") ``` Also a function for creating a spatial index: ```python db.conn.execute(""select CreateSpatialIndex(?, ?)"", [table, ""geometry""]) ``` I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like: ```sh sqlite-utils spatial-index spatial.db table-name column-name ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012253198,IC_kwDOCGYnMM48VcIO,25778,2022-01-13T15:39:14Z,2022-01-13T15:39:14Z,CONTRIBUTOR,"Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (`gis.py` or something) just so it's easier to find later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1012413729,IC_kwDOCGYnMM48WDUh,25778,2022-01-13T18:50:00Z,2022-01-13T18:50:00Z,CONTRIBUTOR,"One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1013698557,IC_kwDOCGYnMM48a8_9,25778,2022-01-15T15:15:22Z,2022-01-15T15:15:22Z,CONTRIBUTOR,@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029317527,IC_kwDOCGYnMM49WiOX,25778,2022-02-03T19:18:02Z,2022-02-03T19:18:02Z,CONTRIBUTOR,"Taking part of the conversation from #385 here. > Would sqlite-utils add-geometry-column ... be a good CLI enhancement. for example? Yes. And also `sqlite-utils create-spatial-index` would be great to have. My plan would be to add those once the Python API is settled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029683977,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029683977,IC_kwDOCGYnMM49X7sJ,9599,2022-02-04T05:58:15Z,2022-02-04T05:58:15Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#spatialite-helpers,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029703503,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029703503,IC_kwDOCGYnMM49YAdP,9599,2022-02-04T06:46:32Z,2022-02-04T06:46:32Z,OWNER,Shipped in 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580515506,https://api.github.com/repos/simonw/sqlite-utils/issues/77,580515506,MDEyOklzc3VlQ29tbWVudDU4MDUxNTUwNg==,9599,2020-01-30T23:48:41Z,2020-01-30T23:48:41Z,OWNER,"Potential design: a `conversions={}` option. Used like this: ```python db[table].insert(record, conversions={""geom"": ""GeomFromText(?, 4326)""}) ``` The `conversions=` key would be supported on `.insert()`, `.insert_all()`, `.upsert()` etc. It could also be passed to the `db.table()` constructor function: ```python table = db.table( ""features"", pk=""id"", conversions={ ""geom"": ""GeomFromText(?, 4326)"" } ) # Then used like this: table.insert(record) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557825032, https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580527238,https://api.github.com/repos/simonw/sqlite-utils/issues/77,580527238,MDEyOklzc3VlQ29tbWVudDU4MDUyNzIzOA==,9599,2020-01-31T00:34:02Z,2020-01-31T00:34:02Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#python-api-conversions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557825032, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614354219,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614354219,MDEyOklzc3VlQ29tbWVudDYxNDM1NDIxOQ==,9599,2020-04-16T01:01:34Z,2020-04-16T01:01:34Z,OWNER,"I think a neat way to do this would be with an optional argument for `.rows_where()`: ```python rows = db[""table""].rows_where(""age > 10"", order_by=""age desc"") ``` If you want everything you can use this: ```python rows = db[""table""].rows_where(order_by=""age desc"") ``` It's a tiny bit weird calling `.rows_where()` without a where clause, but I think it makes sense here - especially since `.rows` is a property that can't take any arguments - though under the hood it actually does this: https://github.com/simonw/sqlite-utils/blob/ad6ac19470a67867b96cb4c086450b8e4e46bf02/sqlite_utils/db.py#L436-L443","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400454,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614400454,MDEyOklzc3VlQ29tbWVudDYxNDQwMDQ1NA==,9599,2020-04-16T03:51:01Z,2020-04-16T03:51:01Z,OWNER,Released in 2.6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400533,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614400533,MDEyOklzc3VlQ29tbWVudDYxNDQwMDUzMw==,9599,2020-04-16T03:51:26Z,2020-04-16T03:51:26Z,OWNER,Documentation here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#listing-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614440032,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614440032,MDEyOklzc3VlQ29tbWVudDYxNDQ0MDAzMg==,10501166,2020-04-16T06:23:29Z,2020-04-16T06:23:29Z,NONE,Thanks for your hard work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310, https://github.com/simonw/sqlite-utils/pull/75#issuecomment-580523995,https://api.github.com/repos/simonw/sqlite-utils/issues/75,580523995,MDEyOklzc3VlQ29tbWVudDU4MDUyMzk5NQ==,9599,2020-01-31T00:21:11Z,2020-01-31T00:21:11Z,OWNER,"This makes sense, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546078359, https://github.com/simonw/sqlite-utils/issues/74#issuecomment-572871797,https://api.github.com/repos/simonw/sqlite-utils/issues/74,572871797,MDEyOklzc3VlQ29tbWVudDU3Mjg3MTc5Nw==,9599,2020-01-10T04:47:55Z,2020-01-10T04:47:55Z,OWNER,"This is odd. I'd love to see more about that result object. Could you try running `pytest --pdb` and then `result.exit_code, result.exception` in the PDB prompt, something like this? ``` $ pytest --pdb ========================================================= test session starts ========================================================= platform darwin -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 rootdir: /Users/simonw/Dropbox/Development/sqlite-utils plugins: cov-2.8.1 collected 216 items tests/test_black.py s [ 0%] tests/test_cli.py F >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> db_path = '/private/var/folders/bl/5x847xbj2yb7xmp7f2tz7l280000gn/T/pytest-of-simonw/pytest-3/test_tables0/test.db' def test_tables(db_path): result = CliRunner().invoke(cli.cli, [""tables1"", db_path]) > assert '[{""table"": ""Gosh""},\n {""table"": ""Gosh2""}]' == result.output.strip() E assert '[{""table"": ""...e"": ""Gosh2""}]' == '' E - [{""table"": ""Gosh""}, E - {""table"": ""Gosh2""}] tests/test_cli.py:28: AssertionError >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /Users/simonw/Dropbox/Development/sqlite-utils/tests/test_cli.py(28)test_tables() -> assert '[{""table"": ""Gosh""},\n {""table"": ""Gosh2""}]' == result.output.strip() (Pdb) result.exit_code, result.exception (1, OperationalError('near ""/"": syntax error')) ``` That should show the exception that caused the script to fail to run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980, https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573388052,https://api.github.com/repos/simonw/sqlite-utils/issues/74,573388052,MDEyOklzc3VlQ29tbWVudDU3MzM4ODA1Mg==,15092,2020-01-12T06:51:30Z,2020-01-12T06:51:30Z,CONTRIBUTOR,"Thanks. That showed me that there was a click cli runner error, and setting `export LANG=en_US.UTF-8` fixed it. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980, https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573389669,https://api.github.com/repos/simonw/sqlite-utils/issues/74,573389669,MDEyOklzc3VlQ29tbWVudDU3MzM4OTY2OQ==,15092,2020-01-12T07:21:17Z,2020-01-12T07:21:17Z,CONTRIBUTOR,"I guess there is some extra flag for ` CliRunner.invoke` to check exitcode and raise the exception, or that should be an extra assert added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570930239,https://api.github.com/repos/simonw/sqlite-utils/issues/73,570930239,MDEyOklzc3VlQ29tbWVudDU3MDkzMDIzOQ==,9599,2020-01-05T17:15:18Z,2020-01-05T17:15:18Z,OWNER,I think this is because you forgot to include a `pk=` argument. I'll change the code to throw a more useful error in this case.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570931650,https://api.github.com/repos/simonw/sqlite-utils/issues/73,570931650,MDEyOklzc3VlQ29tbWVudDU3MDkzMTY1MA==,9599,2020-01-05T17:34:33Z,2020-01-05T17:34:33Z,OWNER,Released as 2.0.1 https://github.com/simonw/sqlite-utils/releases/tag/2.0.1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-571138093,https://api.github.com/repos/simonw/sqlite-utils/issues/73,571138093,MDEyOklzc3VlQ29tbWVudDU3MTEzODA5Mw==,82988,2020-01-06T13:28:31Z,2020-01-06T13:28:31Z,NONE,"I think I actually had several issues in play... The missing key was one, but I think there is also an issue as per below. For example, in the following: ```python def init_testdb(dbname='test.db'): if os.path.exists(dbname): os.remove(dbname) conn = sqlite3.connect(dbname) db = Database(conn) return conn, db conn, db = init_testdb() c = conn.cursor() c.executescript('CREATE TABLE ""test1"" (""Col1"" TEXT, ""Col2"" TEXT, PRIMARY KEY (""Col1""));') c.executescript('CREATE TABLE ""test2"" (""Col1"" TEXT, ""Col2"" TEXT, PRIMARY KEY (""Col1""));') print('Test 1...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) print('Test 2...') for i in range(3): db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, {'Col1':'c','Col2':'x'}], pk=('Col1')) print('Done...') --------------------------------------------------------------------------- Test 1... Test 2... IndexError: list index out of range --------------------------------------------------------------------------- IndexError Traceback (most recent call last) in 22 print('Test 2...') 23 for i in range(3): ---> 24 db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1')) 25 db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}, 26 {'Col1':'c','Col2':'x'}], pk=('Col1')) /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts) 1157 alter=alter, 1158 extracts=extracts, -> 1159 upsert=True, 1160 ) 1161 /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert) 1097 # self.last_rowid will be 0 if a ""INSERT OR IGNORE"" happened 1098 if (hash_id or pk) and self.last_rowid: -> 1099 row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0] 1100 if hash_id: 1101 self.last_pk = row[hash_id] IndexError: list index out of range ``` the first test works but the second fails. Is the length of the list of items being upserted leaking somewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-572870032,https://api.github.com/repos/simonw/sqlite-utils/issues/73,572870032,MDEyOklzc3VlQ29tbWVudDU3Mjg3MDAzMg==,9599,2020-01-10T04:38:41Z,2020-01-10T04:38:41Z,OWNER,"Odd.. I'm not able to replicate that error. Here's what I got: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-573047321,https://api.github.com/repos/simonw/sqlite-utils/issues/73,573047321,MDEyOklzc3VlQ29tbWVudDU3MzA0NzMyMQ==,82988,2020-01-10T14:02:56Z,2020-01-10T14:09:23Z,NONE,"Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop). Could it be sqlite3? I'm on 3.30.1. UPDATE: just tried it on jupyter.org/try and I get the error there, too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-580745213,https://api.github.com/repos/simonw/sqlite-utils/issues/73,580745213,MDEyOklzc3VlQ29tbWVudDU4MDc0NTIxMw==,82988,2020-01-31T14:02:38Z,2020-01-31T14:21:09Z,NONE,"So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error. If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly. It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569233996,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569233996,MDEyOklzc3VlQ29tbWVudDU2OTIzMzk5Ng==,9599,2019-12-27T09:45:17Z,2019-12-27T09:45:17Z,OWNER,"It looks like those backports no longer include sqlite3 - Google Searches still find it but when you click through to launchpad you get 404s: https://launchpad.net/~jonathonf/+archive/ubuntu/codelite/+build/10511920 Maybe Travis have a newer Ubuntu I can use that ships with FTS5 in its SQLite?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234096,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569234096,MDEyOklzc3VlQ29tbWVudDU2OTIzNDA5Ng==,9599,2019-12-27T09:45:52Z,2019-12-27T09:45:52Z,OWNER,I'll try `bionic`: https://docs.travis-ci.com/user/reference/bionic/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234571,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569234571,MDEyOklzc3VlQ29tbWVudDU2OTIzNDU3MQ==,9599,2019-12-27T09:48:48Z,2019-12-27T09:48:48Z,OWNER,That fixed it: https://travis-ci.com/simonw/sqlite-utils/builds/142443259,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756, https://github.com/simonw/sqlite-utils/issues/70#issuecomment-569130037,https://api.github.com/repos/simonw/sqlite-utils/issues/70,569130037,MDEyOklzc3VlQ29tbWVudDU2OTEzMDAzNw==,9599,2019-12-26T20:39:04Z,2019-12-26T20:39:04Z,OWNER,"I hadn't thought about those at all. Are you suggesting a utility mechanism in the library for setting it up so that, for a specific foreign key, rows are deleted from other tables if the row they are pointing at is deleted?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",539204432, https://github.com/simonw/sqlite-utils/issues/70#issuecomment-575799104,https://api.github.com/repos/simonw/sqlite-utils/issues/70,575799104,MDEyOklzc3VlQ29tbWVudDU3NTc5OTEwNA==,26292069,2020-01-17T21:20:17Z,2020-01-17T21:20:17Z,NONE,"Omg sorry I took so long to reply! On SQL we can say how the foreign key behaves when it is deleted or updated on the parent table (see https://www.sqlitetutorial.net/sqlite-foreign-key/ for more details). I did not see clearly how to create tables with this feature on sqlite-utils library.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",539204432, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-457980966,https://api.github.com/repos/simonw/sqlite-utils/issues/7,457980966,MDEyOklzc3VlQ29tbWVudDQ1Nzk4MDk2Ng==,9599,2019-01-28T02:29:32Z,2019-01-28T02:29:32Z,OWNER,"Remember to remove this TODO (and turn the `[]` into `()` on this line) as part of this task: https://github.com/simonw/sqlite-utils/blob/5309c5c7755818323a0f5353bad0de98ecc866be/sqlite_utils/cli.py#L78-L80","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011885,https://api.github.com/repos/simonw/sqlite-utils/issues/7,458011885,MDEyOklzc3VlQ29tbWVudDQ1ODAxMTg4NQ==,9599,2019-01-28T06:25:48Z,2019-01-28T06:25:48Z,OWNER,Re-opening for the second bit involving the cli tool.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011906,https://api.github.com/repos/simonw/sqlite-utils/issues/7,458011906,MDEyOklzc3VlQ29tbWVudDQ1ODAxMTkwNg==,9599,2019-01-28T06:25:55Z,2019-01-28T06:25:55Z,OWNER,"I tested this with a script called `churn_em_out.py` ``` i = 0 while True: i += 1 print( '{""id"": I, ""another"": ""row"", ""number"": J}'.replace(""I"", str(i)).replace( ""J"", str(i + 1) ) ) ``` Then I ran this: ``` python churn_em_out.py | \ sqlite-utils insert /tmp/getbig.db stats - \ --nl --batch-size=10000 ``` And used `watch 'ls -lah /tmp/getbig.db'` to watch the file growing as it had 10,000 lines of junk committed in batches. The memory used by the process never grew about around 50MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-569130196,https://api.github.com/repos/simonw/sqlite-utils/issues/69,569130196,MDEyOklzc3VlQ29tbWVudDU2OTEzMDE5Ng==,9599,2019-12-26T20:40:21Z,2019-12-26T20:40:21Z,OWNER,"This is a good idea. Datasette has this in the form of the `--load-extension` CLI argument, e.g. for SpatiaLite here: https://datasette.readthedocs.io/en/stable/spatialite.html#installation Having that available for `sqlite-utils` definitely makes sense.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",534507142, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710405658,MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==,9599,2020-10-16T18:42:48Z,2020-10-16T18:42:48Z,OWNER,"Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710768396,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710768396,MDEyOklzc3VlQ29tbWVudDcxMDc2ODM5Ng==,30607,2020-10-17T07:46:59Z,2020-10-17T07:46:59Z,NONE,Great @simonw thank you very much,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142, https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776,https://api.github.com/repos/simonw/sqlite-utils/issues/68,695695776,MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng==,9599,2020-09-20T04:25:47Z,2020-09-20T04:25:47Z,OWNER,This is a dupe of #130 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531583658, https://github.com/simonw/sqlite-utils/pull/67#issuecomment-559108591,https://api.github.com/repos/simonw/sqlite-utils/issues/67,559108591,MDEyOklzc3VlQ29tbWVudDU1OTEwODU5MQ==,9599,2019-11-27T14:24:59Z,2019-11-27T14:24:59Z,OWNER,Failed due to black testing dependency: https://travis-ci.com/simonw/sqlite-utils/jobs/260995814,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",529376481, https://github.com/simonw/sqlite-utils/pull/67#issuecomment-569844320,https://api.github.com/repos/simonw/sqlite-utils/issues/67,569844320,MDEyOklzc3VlQ29tbWVudDU2OTg0NDMyMA==,9599,2019-12-31T01:29:43Z,2019-12-31T01:29:43Z,OWNER,I don't really care about 3.5 any more.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",529376481, https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553170650,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553170650,MDEyOklzc3VlQ29tbWVudDU1MzE3MDY1MA==,9599,2019-11-12T23:49:29Z,2019-11-12T23:49:29Z,OWNER,This relates to this bug: https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864, https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171011,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553171011,MDEyOklzc3VlQ29tbWVudDU1MzE3MTAxMQ==,9599,2019-11-12T23:50:52Z,2019-11-12T23:50:52Z,OWNER,"Fixing this is going to be a real pain. There's lots of code out there that uses `sqlite-utils` with the expectation that `upsert()` behaves as it currently does. Maybe I need to introduce new terms for both of these different patterns and deprecate the existing `.upsert()` and `.upsert_all()` since their behaviour can't be changed? Or maybe I fix this and ship `sqlite-utils 2.0` with a breaking change?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864, https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171414,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553171414,MDEyOklzc3VlQ29tbWVudDU1MzE3MTQxNA==,9599,2019-11-12T23:52:35Z,2019-11-12T23:52:35Z,OWNER,"If I do implement the correct definition of `.upsert()` I think I'll use this pattern, since it works in versions of SQLite prior to 3.24: ```sql INSERT OR IGNORE INTO book(id) VALUES(1001); UPDATE book SET name = 'Programming' WHERE id = 1001; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864, https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553526685,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553526685,MDEyOklzc3VlQ29tbWVudDU1MzUyNjY4NQ==,9599,2019-11-13T17:58:59Z,2019-11-13T17:58:59Z,OWNER,"This warrants making a backwards compatible change, which means I'll need to bump the major version number and release 2.0. I'm going to rename the existing `upsert()` and `upsert_all()` methods to `replace()` and `replace_all()` - then write new `upsert()` and `upsert_all()` methods that implement the correct behavior.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864, https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553527384,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553527384,MDEyOklzc3VlQ29tbWVudDU1MzUyNzM4NA==,9599,2019-11-13T18:00:41Z,2019-11-13T18:00:41Z,OWNER,Is `replace()` a good name here? It doesn't really convey the idea that a brand new record will be created if there isn't an existing one to replace.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864, https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528386,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553528386,MDEyOklzc3VlQ29tbWVudDU1MzUyODM4Ng==,9599,2019-11-13T18:03:10Z,2019-11-13T18:03:54Z,OWNER,"Maybe `inplace()` (combining ""insert"" and ""replace"")? It could be an alias for `.insert(..., replace=True)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,