html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727400,https://api.github.com/repos/simonw/sqlite-utils/issues/99,612727400,MDEyOklzc3VlQ29tbWVudDYxMjcyNzQwMA==,9599,simonw,2020-04-13T03:03:09Z,2020-04-13T03:03:09Z,OWNER,I think I'm going to leave this as intended behaviour. Or maybe passing multiple dictionaries to `.upsert_all()` with different numbers of keys should raise an error?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598640234,.upsert_all() should maybe error if dictionaries passed to it do not have the same keys, https://github.com/simonw/sqlite-utils/issues/99#issuecomment-612727814,https://api.github.com/repos/simonw/sqlite-utils/issues/99,612727814,MDEyOklzc3VlQ29tbWVudDYxMjcyNzgxNA==,9599,simonw,2020-04-13T03:05:04Z,2020-04-13T03:05:04Z,OWNER,"Bit trick from an implementation point of view this, since we want to be able to handle input that is a generator - so we can't scan through the input to validate that every dictionary has the same exact keys without consuming the entire iterator. The alternative would be to raise an error the first time we spot a dictionary with keys that differ... but that's weird because we commit changes in batches, so we may end up only applying half of the changes before exiting with the error. On that basis, I'm going to leave this as-is and mark this as wontfix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",598640234,.upsert_all() should maybe error if dictionaries passed to it do not have the same keys, https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612173156,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612173156,MDEyOklzc3VlQ29tbWVudDYxMjE3MzE1Ng==,9599,simonw,2020-04-10T19:03:32Z,2020-04-10T23:08:28Z,OWNER,"Investigate this traceback: ``` Traceback (most recent call last): File ""fetch_projects.py"", line 60, in fetch_projects(db, token) File ""fetch_projects.py"", line 41, in fetch_projects db[""projects""].upsert(project, pk=""id"") File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1139, in upsert conversions=conversions, File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1168, in upsert_all upsert=True, File ""/Users/simonw/.local/share/virtualenvs/big-local-datasette-2jT6nJCT/lib/python3.7/site-packages/sqlite_utils/db.py"", line 1107, in insert_all row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0] IndexError: list index out of range ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612258687,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612258687,MDEyOklzc3VlQ29tbWVudDYxMjI1ODY4Nw==,9599,simonw,2020-04-10T23:08:48Z,2020-04-10T23:08:48Z,OWNER,I need a test that reproduces this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707293,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612707293,MDEyOklzc3VlQ29tbWVudDYxMjcwNzI5Mw==,9599,simonw,2020-04-13T01:21:22Z,2020-04-13T01:21:22Z,OWNER,"I have a hunch that the root of the problem here is that accessing `result.lastrowid` during my version of an `.upsert()` doesn't actually make sense: https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1102-L1113 In the bug I'm seeing (which I still haven't reduced to a reproducible test) the debugger shows me this at that point: ``` (Pdb) query 'UPDATE [files] SET [createdAt] = ?, [ext] = ?, [updatedAt] = ?, [uri] = ?, [uriType] = ? WHERE [project] = ? AND [name] = ?' (Pdb) params ['2020-03-04T04:04:40.152000+00:00', 'csv', '2020-03-04T04:04:40.152000+00:00', 'https://storage.googleapis.com/bln_prod/...', 'download', 'UHJvamVjdDo4MTgyMjU2Ny01ZjI0LTQxM2ItYWZmNi05NTlmNGY3MjExMjI=', 'loans_to_documentation.csv'] (Pdb) result.lastrowid 100 ``` But here's the weird thing... there's no row in the table with a rowid of 100! ``` (Pdb) [r['rowid'] for r in self.db.execute_returning_dicts('select rowid, * from files')] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99] ``` So what the heck is going on? The last SQL statement I executed here was an `UPDATE`. The `lastrowid` docs say: https://kite.com/python/docs/sqlite3.Cursor.lastrowid > This read-only attribute provides the rowid of the last modified row. It is only set if you issued a INSERT statement using the execute() method. For operations other than INSERT or when executemany() is called, lastrowid is set to None. So where did that `100` come from? It should be `None`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612707828,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612707828,MDEyOklzc3VlQ29tbWVudDYxMjcwNzgyOA==,9599,simonw,2020-04-13T01:24:05Z,2020-04-13T01:24:16Z,OWNER,"Why do I even care about `lastrowid` here? I'm trying to ensure that after you insert or upsert a row you can use `table.last_pk` to start doing things like building additional foreign key relationships. So maybe it doesn't make sense to make `.last_pk` available _at all_ for cases where you called `.upsert_all()` or `.insert_all()` - it should just be populated for `.upsert()` and `.insert()`. The documentation doesn't say it should work for `.upsert_all()` - it's only documented for the single actions. https://github.com/simonw/sqlite-utils/blob/6161ebf4de44411b3f33feeacaf4501e803d1116/sqlite_utils/db.py#L1113-L1124","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612708274,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612708274,MDEyOklzc3VlQ29tbWVudDYxMjcwODI3NA==,9599,simonw,2020-04-13T01:25:59Z,2020-04-13T01:26:11Z,OWNER,"In mucking around with `sqlite3` it looks like `result.lastrowid` is indeed populated for `UPDATE` - in this case with the last inserted rowid in the table. This differs from the documented behaviour I linked to above. ``` In [1]: import sqlite3 In [2]: c = sqlite3.connect("":memory:"") In [3]: c Out[3]: In [4]: c.execute('create table foo (bar integer);') Out[4]: In [5]: c.execute('insert into foo (bar) values (1)') Out[5]: In [6]: c.execute('select * from foo').fetchall() Out[6]: [(1,)] In [7]: c.execute('insert into foo (bar) values (1)') Out[7]: In [8]: c.execute('select * from foo').fetchall() Out[8]: [(1,), (1,)] In [9]: c.execute('insert into foo (bar) values (1)').lastrowid Out[9]: 3 In [10]: c.execute('select * from foo').fetchall() Out[10]: [(1,), (1,), (1,)] In [11]: c.execute('select rowid, bar from foo').fetchall() Out[11]: [(1, 1), (2, 1), (3, 1)] In [12]: c.execute('insert into foo (bar) values (1)').lastrowid Out[12]: 4 In [13]: c.execute('select rowid, bar from foo').fetchall() Out[13]: [(1, 1), (2, 1), (3, 1), (4, 1)] In [14]: r = c.execute('update foo set bar =2 where rowid = 1') In [15]: r.lastrowid Out[15]: 4 In [16]: c.execute('select rowid, bar from foo').fetchall() Out[16]: [(1, 2), (2, 1), (3, 1), (4, 1)] In [17]: r = c.execute('select rowid, bar from foo') In [18]: r.fetchall() Out[18]: [(1, 2), (2, 1), (3, 1), (4, 1)] In [19]: r.lastrowid Out[19]: 4 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/98#issuecomment-612728047,https://api.github.com/repos/simonw/sqlite-utils/issues/98,612728047,MDEyOklzc3VlQ29tbWVudDYxMjcyODA0Nw==,9599,simonw,2020-04-13T03:06:10Z,2020-04-13T03:06:10Z,OWNER,Implementation plan: `.insert_all()` and `.upsert_all()` should only set `.last_rowid` and `last_pk` if they were called with a single item.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records", https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732129,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612732129,MDEyOklzc3VlQ29tbWVudDYxMjczMjEyOQ==,9599,simonw,2020-04-13T03:25:29Z,2020-04-13T03:25:29Z,OWNER,"Interesting thought. I've run into this myself a lot - many of my scripts intend to create the database from scratch, so I end up running `!rm /tmp/blah.db` in Jupyter and occasionally getting errors if the file doesn't exist. I think adding `recreate=True` could make sense. It could throw an error if you attempt to use it after passing in something other than a path to a file on disk.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293,"Adding a ""recreate"" flag to the `Database` constructor", https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612732453,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612732453,MDEyOklzc3VlQ29tbWVudDYxMjczMjQ1Mw==,9599,simonw,2020-04-13T03:26:46Z,2020-04-13T03:26:46Z,OWNER,"I wonder if it should delete an recreate the file or if it would be safer to drop every table instead? Dropping tables gets messy: then you need to drop triggers and views, and you need to run `vacuum` to clean up the space. My worry with deleting and recreating the file is that it could trigger errors in other processes that are currently attached to that database file. But... if you know that's going to be likely, maybe you shouldn't use the `recreate=True` feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293,"Adding a ""recreate"" flag to the `Database` constructor", https://github.com/simonw/sqlite-utils/issues/97#issuecomment-612738311,https://api.github.com/repos/simonw/sqlite-utils/issues/97,612738311,MDEyOklzc3VlQ29tbWVudDYxMjczODMxMQ==,9599,simonw,2020-04-13T03:55:11Z,2020-04-13T03:55:11Z,OWNER,Shipped in 2.5 - documentation is here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#connecting-to-or-creating-a-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",593751293,"Adding a ""recreate"" flag to the `Database` constructor", https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394349,https://api.github.com/repos/simonw/sqlite-utils/issues/96,606394349,MDEyOklzc3VlQ29tbWVudDYwNjM5NDM0OQ==,9599,simonw,2020-03-31T04:37:16Z,2020-03-31T04:37:16Z,OWNER,Test failure was just a Black formatting issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",589801352,Add type conversion for Panda's Timestamp, https://github.com/simonw/sqlite-utils/pull/96#issuecomment-606394619,https://api.github.com/repos/simonw/sqlite-utils/issues/96,606394619,MDEyOklzc3VlQ29tbWVudDYwNjM5NDYxOQ==,9599,simonw,2020-03-31T04:38:17Z,2020-03-31T04:40:23Z,OWNER,I wonder if there are any other Pandas conversions we should be doing? https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#dtypes ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",589801352,Add type conversion for Panda's Timestamp, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125455,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599125455,MDEyOklzc3VlQ29tbWVudDU5OTEyNTQ1NQ==,9599,simonw,2020-03-14T19:34:35Z,2020-03-14T19:34:35Z,OWNER,"From https://www.sqlite.org/datatype3.html it looks like `FLOAT` is a supported keyword for creating tables but `REAL` is the correct keyword. So actually `sqlite-utils` gets this wrong, because when we create a table we turn Python `float` values into a `FLOAT` column. Looks like the correct behaviour would be to turn them into a `REAL` column. https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L28-L48","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599125557,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599125557,MDEyOklzc3VlQ29tbWVudDU5OTEyNTU1Nw==,9599,simonw,2020-03-14T19:35:29Z,2020-03-14T19:35:29Z,OWNER,"Fixing that would technically constitute a breaking change for library consumers, so it should be a major version release. I'm not inclined to release `3.0` just for this one issue, so I'm going to hold back on fixing that and address the smaller issue in this bug as a dot release instead for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599126831,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599126831,MDEyOklzc3VlQ29tbWVudDU5OTEyNjgzMQ==,9599,simonw,2020-03-14T19:45:28Z,2020-03-14T19:45:28Z,OWNER,"Turns out there are a TON of valid column definitions that aren't being considered yet - https://www.sqlite.org/datatype3.html#affinity_name_examples - stuff like `VARYING CHARACTER(255)` and `DECIMAL(10,5)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127197,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599127197,MDEyOklzc3VlQ29tbWVudDU5OTEyNzE5Nw==,9599,simonw,2020-03-14T19:48:06Z,2020-03-14T19:48:06Z,OWNER,"Actually it looks like I should implement the exact rules described in https://www.sqlite.org/datatype3.html#determination_of_column_affinity > The affinity of a column is determined by the declared type of the column, according to the following rules in the order shown: > > 1. If the declared type contains the string ""INT"" then it is assigned INTEGER affinity. > 2. If the declared type of the column contains any of the strings ""CHAR"", ""CLOB"", or ""TEXT"" then that column has TEXT affinity. Notice that the type VARCHAR contains the string ""CHAR"" and is thus assigned TEXT affinity. > 3. If the declared type for a column contains the string ""BLOB"" or if no type is specified then the column has affinity BLOB. > 4. If the declared type for a column contains any of the strings ""REAL"", ""FLOA"", or ""DOUB"" then the column has REAL affinity. > 5. Otherwise, the affinity is NUMERIC. > > Note that the order of the rules for determining column affinity is important. A column whose declared type is ""CHARINT"" will match both rules 1 and 2 but the first rule takes precedence and so the column affinity will be INTEGER.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599127453,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599127453,MDEyOklzc3VlQ29tbWVudDU5OTEyNzQ1Mw==,9599,simonw,2020-03-14T19:50:08Z,2020-03-14T19:50:08Z,OWNER,"> If the declared type for a column contains the string ""BLOB"" or if no type is specified then the column has affinity BLOB I currently treat those as `str` - it sounds like I should treat them as `bytes`: https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L69 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599128891,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599128891,MDEyOklzc3VlQ29tbWVudDU5OTEyODg5MQ==,9599,simonw,2020-03-14T20:03:45Z,2020-03-14T20:03:45Z,OWNER,I'm going to keep treating them as `str`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/92#issuecomment-599247833,https://api.github.com/repos/simonw/sqlite-utils/issues/92,599247833,MDEyOklzc3VlQ29tbWVudDU5OTI0NzgzMw==,9599,simonw,2020-03-15T18:37:28Z,2020-03-15T18:37:43Z,OWNER,Released in 2.4.2.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",581339961,.columns_dict doesn't work for all possible column types, https://github.com/simonw/sqlite-utils/issues/91#issuecomment-723350956,https://api.github.com/repos/simonw/sqlite-utils/issues/91,723350956,MDEyOklzc3VlQ29tbWVudDcyMzM1MDk1Ng==,9599,simonw,2020-11-06T23:53:25Z,2020-11-06T23:53:25Z,OWNER,"This is now possible, for both FTS4 and FTS5 - see #197.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",577302229,Enable ordering FTS results by rank, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-615515867,https://api.github.com/repos/simonw/sqlite-utils/issues/89,615515867,MDEyOklzc3VlQ29tbWVudDYxNTUxNTg2Nw==,9599,simonw,2020-04-18T00:00:41Z,2020-04-18T00:00:41Z,OWNER,"Yes pleas, I'd love to see that pull request!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242,https://api.github.com/repos/simonw/sqlite-utils/issues/89,710460242,MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==,9599,simonw,2020-10-16T19:17:27Z,2020-10-16T19:17:50Z,OWNER,"I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works: ```python fresh_db.table(""tree"", extracts=[Extract( columns=(""CommonName"", ""LatinName""), table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} )]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature, https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591768604,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591768604,MDEyOklzc3VlQ29tbWVudDU5MTc2ODYwNA==,9599,simonw,2020-02-27T04:03:03Z,2020-02-27T04:03:03Z,OWNER,"`drop table resources_fts` drops the FTS table and the other ones that it created (resources_fts_data, resources_fts_idx, resources_fts_docsize, resources_fts_config) - but keeps the triggers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769046,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769046,MDEyOklzc3VlQ29tbWVudDU5MTc2OTA0Ng==,9599,simonw,2020-02-27T04:05:15Z,2020-02-27T04:05:15Z,OWNER,I can reliably get the list of triggers to delete from `select name from sqlite_master where type = 'trigger' and tbl_name = 'resources';`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769171,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769171,MDEyOklzc3VlQ29tbWVudDU5MTc2OTE3MQ==,9599,simonw,2020-02-27T04:05:58Z,2020-02-27T04:26:31Z,OWNER,"Strange - https://www.sqlite.org/lang_droptrigger.html says ""Note that triggers are automatically dropped when the associated table is dropped"" but that doesn't seem to be true in my experimenting. UPDATE: no that makes sense - the triggers are on `resources` which still exists, it was `resources_fts` that was dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769373,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769373,MDEyOklzc3VlQ29tbWVudDU5MTc2OTM3Mw==,9599,simonw,2020-02-27T04:06:47Z,2020-02-27T04:06:47Z,OWNER,Looks like safest option is to loop through those trigger names and run `DROP TRIGGER IF EXISTS foo` on each one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/88#issuecomment-591769759,https://api.github.com/repos/simonw/sqlite-utils/issues/88,591769759,MDEyOklzc3VlQ29tbWVudDU5MTc2OTc1OQ==,9599,simonw,2020-02-27T04:08:29Z,2020-02-27T04:08:29Z,OWNER,"I think the method should be called `table.disable_fts()` - the opposite of `table.enable_fts(...)`. There should be a `sqlite-utils disable-fts database.db tablename` command to match it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",571805300,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command", https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661250,https://api.github.com/repos/simonw/sqlite-utils/issues/87,586661250,MDEyOklzc3VlQ29tbWVudDU4NjY2MTI1MA==,9599,simonw,2020-02-16T02:19:33Z,2020-02-16T02:19:33Z,OWNER,"Here's the code: https://github.com/simonw/sqlite-utils/blob/5e0000609f9be6efafea1b96f610988eb18d6d89/sqlite_utils/utils.py#L18-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",565837965,Should detect collections.OrderedDict as a regular dictionary, https://github.com/simonw/sqlite-utils/issues/87#issuecomment-586661276,https://api.github.com/repos/simonw/sqlite-utils/issues/87,586661276,MDEyOklzc3VlQ29tbWVudDU4NjY2MTI3Ng==,9599,simonw,2020-02-16T02:20:14Z,2020-02-16T02:20:14Z,OWNER,"`OrderedDict` is actually a subclass of `dict` - so a smart fix would be for this logic to check and see if the type `t` is a subclass of one of `list`, `tuple` or `dict`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",565837965,Should detect collections.OrderedDict as a regular dictionary, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586661934,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586661934,MDEyOklzc3VlQ29tbWVudDU4NjY2MTkzNA==,9599,simonw,2020-02-16T02:33:07Z,2020-02-16T02:33:07Z,OWNER,"Thanks for the example file - looks like it can be trimmed down to just these two lines to replicate the bug: ```csv ""MTU (CET)"",""Day-ahead Price [EUR/MWh]"" ""01.01.2016 00:00 - 01.01.2016 01:00"",""23.86"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586662404,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586662404,MDEyOklzc3VlQ29tbWVudDU4NjY2MjQwNA==,9599,simonw,2020-02-16T02:43:12Z,2020-02-16T02:43:12Z,OWNER,"https://stackoverflow.com/a/22694438 looks like the answer: > When using square brackets, it is not possible to have these characters in the identifier. > > When using double quotes, you can escape them in the name by doubling them: > > `CREATE TABLE ""hello """"world""""""(key INTEGER PRIMARY KEY);`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676640,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586676640,MDEyOklzc3VlQ29tbWVudDU4NjY3NjY0MA==,9599,simonw,2020-02-16T07:16:31Z,2020-02-16T07:16:31Z,OWNER,"There's something weird about this. I created a test database file like so: ``` sqlite3 /tmp/demo.db < .schema CREATE TABLE IF NOT EXISTS ""data"" ( ""MTU (CET)"" TEXT, ""Day-ahead Price [EUR/MWh]"" TEXT ); sqlite> .headers on sqlite> select * from data; MTU (CET)|Day-ahead Price [EUR/MWh] 01.01.2016 00:00 - 01.01.2016 01:00|23.86 sqlite> ``` BUT... if I open the same database in Python, something weird happens: ``` In [1]: import sqlite3 In [2]: conn = sqlite3.connect(""/tmp/demo.db"") In [3]: cursor = conn.cursor() In [4]: cursor.execute(""select * from data"") Out[4]: In [5]: cursor.fetchall() Out[5]: [('01.01.2016 00:00 - 01.01.2016 01:00', '23.86')] In [6]: cursor.description Out[6]: (('MTU (CET)', None, None, None, None, None, None), ('Day-ahead Price', None, None, None, None, None, None)) In [7]: conn.row_factory = sqlite3.Row In [8]: cursor = conn.cursor() In [9]: cursor.execute(""select * from data"") Out[9]: In [10]: row = cursor.fetchall() In [12]: row Out[12]: In [15]: row.keys() Out[15]: ['MTU (CET)', 'Day-ahead Price'] ``` Note that in `cursor.description` AND in `row.keys()` above the second column is displayed as `'Day-ahead Price'` - when we would expect it to be displayed as `Day-ahead Price [EUR/MWh]` So.... it looks like there may be a bug in Python's `sqlite3` module where columns with square braces in them have that portion of the name stripped out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586676856,MDEyOklzc3VlQ29tbWVudDU4NjY3Njg1Ng==,9599,simonw,2020-02-16T07:20:34Z,2020-02-16T07:20:34Z,OWNER,"I'm not sure what to do about this one. I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again. So... I could do one of the following: - Throw an error if a column name includes those characters. That's my preferred option I think. - Automatically replace `[` in column names with `(` and `]` with `)` - Do the automatic replacement but show a user-visible warning when I do it - Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix. Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586729798,https://api.github.com/repos/simonw/sqlite-utils/issues/86,586729798,MDEyOklzc3VlQ29tbWVudDU4NjcyOTc5OA==,9599,simonw,2020-02-16T17:11:02Z,2020-02-16T17:11:02Z,OWNER,I filed a bug in the Python issue tracker here: https://bugs.python.org/issue39652,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/86#issuecomment-591770623,https://api.github.com/repos/simonw/sqlite-utils/issues/86,591770623,MDEyOklzc3VlQ29tbWVudDU5MTc3MDYyMw==,9599,simonw,2020-02-27T04:12:39Z,2020-02-27T04:12:39Z,OWNER,"I pushed a branch with my experiment in it, but I'm going to fix this by throwing an error on `[` or `]` in a column name instead - I won't implement the changes from that branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564579430,Problem with square bracket in CSV column name, https://github.com/simonw/sqlite-utils/issues/85#issuecomment-584426938,https://api.github.com/repos/simonw/sqlite-utils/issues/85,584426938,MDEyOklzc3VlQ29tbWVudDU4NDQyNjkzOA==,9599,simonw,2020-02-11T00:35:09Z,2020-02-11T00:35:09Z,OWNER,"Here's why: https://github.com/simonw/sqlite-utils/blob/0c2451e0690c5f4e6463a2f339b0a280e30ed806/sqlite_utils/db.py#L627-L636","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",562911863,Create index doesn't work for columns containing spaces, https://github.com/simonw/sqlite-utils/issues/83#issuecomment-583789015,https://api.github.com/repos/simonw/sqlite-utils/issues/83,583789015,MDEyOklzc3VlQ29tbWVudDU4Mzc4OTAxNQ==,9599,simonw,2020-02-08T23:58:35Z,2020-02-08T23:58:35Z,OWNER,Shipped as 2.3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559374410,"Make db[""table""].exists a documented API", https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581651409,https://api.github.com/repos/simonw/sqlite-utils/issues/82,581651409,MDEyOklzc3VlQ29tbWVudDU4MTY1MTQwOQ==,9599,simonw,2020-02-03T22:32:41Z,2020-02-03T22:32:41Z,OWNER,This should work - the data should be chunked automatically. It looks like this is a bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745,Tutorial command no longer works, https://github.com/simonw/sqlite-utils/issues/82#issuecomment-581652388,https://api.github.com/repos/simonw/sqlite-utils/issues/82,581652388,MDEyOklzc3VlQ29tbWVudDU4MTY1MjM4OA==,9599,simonw,2020-02-03T22:35:44Z,2020-02-03T22:35:44Z,OWNER,"I can't replicate this problem: ``` /tmp $ sqlite-utils --version sqlite-utils, version 2.2 /tmp $ curl ""https://data.nasa.gov/resource/y77d-th95.json"" | sqlite-utils insert meteorites.db meteorites - --pk=id % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 240k 0 240k 0 0 185k 0 --:--:-- 0:00:01 --:--:-- 185k ``` Could you run `sqlite-utils --version` and tell me what you get?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745,Tutorial command no longer works, https://github.com/simonw/sqlite-utils/issues/82#issuecomment-591771532,https://api.github.com/repos/simonw/sqlite-utils/issues/82,591771532,MDEyOklzc3VlQ29tbWVudDU5MTc3MTUzMg==,9599,simonw,2020-02-27T04:16:30Z,2020-02-27T04:16:30Z,OWNER,Closing as can't reproduce.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",559197745,Tutorial command no longer works, https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071010,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071010,MDEyOklzc3VlQ29tbWVudDU4MTA3MTAxMA==,9599,simonw,2020-02-01T21:27:00Z,2020-02-01T21:27:00Z,OWNER,"Here's the current method: https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/sqlite_utils/db.py#L823-L845 If I make it a utility function instead of a class method I could ensure it is directly importable like so: ```python from sqlite_utils import detect_column_types ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274,"Remove .detect_column_types() from table, make it a documented API", https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071116,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071116,MDEyOklzc3VlQ29tbWVudDU4MTA3MTExNg==,9599,simonw,2020-02-01T21:28:35Z,2020-02-01T21:28:53Z,OWNER,"Should I keep `table.detect_column_types()` working so as not to break existing code? If it was part of the documented API then I wouldn't break that without bumping to 3.x. Since it's undocumented I'm going to make it as a breaking change instead (and bump the `geojson-to-sqlite` dependency version).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274,"Remove .detect_column_types() from table, make it a documented API", https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071235,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071235,MDEyOklzc3VlQ29tbWVudDU4MTA3MTIzNQ==,9599,simonw,2020-02-01T21:30:09Z,2020-02-01T21:30:09Z,OWNER,Actually I'll put it in the `utils.py` module.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274,"Remove .detect_column_types() from table, make it a documented API", https://github.com/simonw/sqlite-utils/issues/81#issuecomment-581071434,https://api.github.com/repos/simonw/sqlite-utils/issues/81,581071434,MDEyOklzc3VlQ29tbWVudDU4MTA3MTQzNA==,9599,simonw,2020-02-01T21:32:34Z,2020-02-01T21:32:34Z,OWNER,While I'm at it I think I'll rename it to `suggest_column_types` - it's not really detecting them since the input is just a list of dictionaries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",558600274,"Remove .detect_column_types() from table, make it a documented API", https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567505,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567505,MDEyOklzc3VlQ29tbWVudDU4MDU2NzUwNQ==,9599,simonw,2020-01-31T03:39:19Z,2020-01-31T03:39:19Z,OWNER,"Still needs documentation and tests. Also I'm not certain that this should be an argument you can pass to the `.table()` constructor, need to think that over.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567604,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567604,MDEyOklzc3VlQ29tbWVudDU4MDU2NzYwNA==,9599,simonw,2020-01-31T03:39:58Z,2020-01-31T03:39:58Z,OWNER,Perhaps this should be called `after_create` instead of `on_create`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580567886,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580567886,MDEyOklzc3VlQ29tbWVudDU4MDU2Nzg4Ng==,9599,simonw,2020-01-31T03:41:31Z,2020-01-31T03:41:31Z,OWNER,I think it does make sense to be able to pass it to the `.table()` constructor.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580569059,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580569059,MDEyOklzc3VlQ29tbWVudDU4MDU2OTA1OQ==,9599,simonw,2020-01-31T03:48:41Z,2020-01-31T03:48:41Z,OWNER,"This may not be the right feature after all, see https://github.com/simonw/geojson-to-sqlite/issues/6#issuecomment-580569002","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/pull/80#issuecomment-580584269,https://api.github.com/repos/simonw/sqlite-utils/issues/80,580584269,MDEyOklzc3VlQ29tbWVudDU4MDU4NDI2OQ==,9599,simonw,2020-01-31T05:08:04Z,2020-01-31T05:08:04Z,OWNER,Ditching this since it won't actually solve my problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557892819,on_create mechanism for after table creation, https://github.com/simonw/sqlite-utils/issues/8#issuecomment-466695500,https://api.github.com/repos/simonw/sqlite-utils/issues/8,466695500,MDEyOklzc3VlQ29tbWVudDQ2NjY5NTUwMA==,9599,simonw,2019-02-23T21:09:03Z,2019-02-23T21:09:03Z,OWNER,"Fixed in https://github.com/simonw/sqlite-utils/commit/228d595f7d10994f34e948888093c2cd290267c4 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403922644,Problems handling column names containing spaces or - , https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029683977,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029683977,IC_kwDOCGYnMM49X7sJ,9599,simonw,2022-02-04T05:58:15Z,2022-02-04T05:58:15Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#spatialite-helpers,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029703503,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029703503,IC_kwDOCGYnMM49YAdP,9599,simonw,2022-02-04T06:46:32Z,2022-02-04T06:46:32Z,OWNER,Shipped in 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245,Helper methods for working with SpatiaLite, https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580515506,https://api.github.com/repos/simonw/sqlite-utils/issues/77,580515506,MDEyOklzc3VlQ29tbWVudDU4MDUxNTUwNg==,9599,simonw,2020-01-30T23:48:41Z,2020-01-30T23:48:41Z,OWNER,"Potential design: a `conversions={}` option. Used like this: ```python db[table].insert(record, conversions={""geom"": ""GeomFromText(?, 4326)""}) ``` The `conversions=` key would be supported on `.insert()`, `.insert_all()`, `.upsert()` etc. It could also be passed to the `db.table()` constructor function: ```python table = db.table( ""features"", pk=""id"", conversions={ ""geom"": ""GeomFromText(?, 4326)"" } ) # Then used like this: table.insert(record) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557825032,Ability to insert data that is transformed by a SQL function, https://github.com/simonw/sqlite-utils/issues/77#issuecomment-580527238,https://api.github.com/repos/simonw/sqlite-utils/issues/77,580527238,MDEyOklzc3VlQ29tbWVudDU4MDUyNzIzOA==,9599,simonw,2020-01-31T00:34:02Z,2020-01-31T00:34:02Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#python-api-conversions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557825032,Ability to insert data that is transformed by a SQL function, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614354219,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614354219,MDEyOklzc3VlQ29tbWVudDYxNDM1NDIxOQ==,9599,simonw,2020-04-16T01:01:34Z,2020-04-16T01:01:34Z,OWNER,"I think a neat way to do this would be with an optional argument for `.rows_where()`: ```python rows = db[""table""].rows_where(""age > 10"", order_by=""age desc"") ``` If you want everything you can use this: ```python rows = db[""table""].rows_where(order_by=""age desc"") ``` It's a tiny bit weird calling `.rows_where()` without a where clause, but I think it makes sense here - especially since `.rows` is a property that can't take any arguments - though under the hood it actually does this: https://github.com/simonw/sqlite-utils/blob/ad6ac19470a67867b96cb4c086450b8e4e46bf02/sqlite_utils/db.py#L436-L443","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310,order_by mechanism, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400454,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614400454,MDEyOklzc3VlQ29tbWVudDYxNDQwMDQ1NA==,9599,simonw,2020-04-16T03:51:01Z,2020-04-16T03:51:01Z,OWNER,Released in 2.6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310,order_by mechanism, https://github.com/simonw/sqlite-utils/issues/76#issuecomment-614400533,https://api.github.com/repos/simonw/sqlite-utils/issues/76,614400533,MDEyOklzc3VlQ29tbWVudDYxNDQwMDUzMw==,9599,simonw,2020-04-16T03:51:26Z,2020-04-16T03:51:26Z,OWNER,Documentation here: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#listing-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",549287310,order_by mechanism, https://github.com/simonw/sqlite-utils/pull/75#issuecomment-580523995,https://api.github.com/repos/simonw/sqlite-utils/issues/75,580523995,MDEyOklzc3VlQ29tbWVudDU4MDUyMzk5NQ==,9599,simonw,2020-01-31T00:21:11Z,2020-01-31T00:21:11Z,OWNER,"This makes sense, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546078359,Explicitly include tests and docs in sdist, https://github.com/simonw/sqlite-utils/issues/74#issuecomment-572871797,https://api.github.com/repos/simonw/sqlite-utils/issues/74,572871797,MDEyOklzc3VlQ29tbWVudDU3Mjg3MTc5Nw==,9599,simonw,2020-01-10T04:47:55Z,2020-01-10T04:47:55Z,OWNER,"This is odd. I'd love to see more about that result object. Could you try running `pytest --pdb` and then `result.exit_code, result.exception` in the PDB prompt, something like this? ``` $ pytest --pdb ========================================================= test session starts ========================================================= platform darwin -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 rootdir: /Users/simonw/Dropbox/Development/sqlite-utils plugins: cov-2.8.1 collected 216 items tests/test_black.py s [ 0%] tests/test_cli.py F >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> db_path = '/private/var/folders/bl/5x847xbj2yb7xmp7f2tz7l280000gn/T/pytest-of-simonw/pytest-3/test_tables0/test.db' def test_tables(db_path): result = CliRunner().invoke(cli.cli, [""tables1"", db_path]) > assert '[{""table"": ""Gosh""},\n {""table"": ""Gosh2""}]' == result.output.strip() E assert '[{""table"": ""...e"": ""Gosh2""}]' == '' E - [{""table"": ""Gosh""}, E - {""table"": ""Gosh2""}] tests/test_cli.py:28: AssertionError >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /Users/simonw/Dropbox/Development/sqlite-utils/tests/test_cli.py(28)test_tables() -> assert '[{""table"": ""Gosh""},\n {""table"": ""Gosh2""}]' == result.output.strip() (Pdb) result.exit_code, result.exception (1, OperationalError('near ""/"": syntax error')) ``` That should show the exception that caused the script to fail to run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",546073980,Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570930239,https://api.github.com/repos/simonw/sqlite-utils/issues/73,570930239,MDEyOklzc3VlQ29tbWVudDU3MDkzMDIzOQ==,9599,simonw,2020-01-05T17:15:18Z,2020-01-05T17:15:18Z,OWNER,I think this is because you forgot to include a `pk=` argument. I'll change the code to throw a more useful error in this case.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-570931650,https://api.github.com/repos/simonw/sqlite-utils/issues/73,570931650,MDEyOklzc3VlQ29tbWVudDU3MDkzMTY1MA==,9599,simonw,2020-01-05T17:34:33Z,2020-01-05T17:34:33Z,OWNER,Released as 2.0.1 https://github.com/simonw/sqlite-utils/releases/tag/2.0.1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/73#issuecomment-572870032,https://api.github.com/repos/simonw/sqlite-utils/issues/73,572870032,MDEyOklzc3VlQ29tbWVudDU3Mjg3MDAzMg==,9599,simonw,2020-01-10T04:38:41Z,2020-01-10T04:38:41Z,OWNER,"Odd.. I'm not able to replicate that error. Here's what I got: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",545407916,upsert_all() throws issue when upserting to empty table, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569233996,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569233996,MDEyOklzc3VlQ29tbWVudDU2OTIzMzk5Ng==,9599,simonw,2019-12-27T09:45:17Z,2019-12-27T09:45:17Z,OWNER,"It looks like those backports no longer include sqlite3 - Google Searches still find it but when you click through to launchpad you get 404s: https://launchpad.net/~jonathonf/+archive/ubuntu/codelite/+build/10511920 Maybe Travis have a newer Ubuntu I can use that ships with FTS5 in its SQLite?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756,Tests are failing due to missing FTS5, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234096,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569234096,MDEyOklzc3VlQ29tbWVudDU2OTIzNDA5Ng==,9599,simonw,2019-12-27T09:45:52Z,2019-12-27T09:45:52Z,OWNER,I'll try `bionic`: https://docs.travis-ci.com/user/reference/bionic/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756,Tests are failing due to missing FTS5, https://github.com/simonw/sqlite-utils/issues/71#issuecomment-569234571,https://api.github.com/repos/simonw/sqlite-utils/issues/71,569234571,MDEyOklzc3VlQ29tbWVudDU2OTIzNDU3MQ==,9599,simonw,2019-12-27T09:48:48Z,2019-12-27T09:48:48Z,OWNER,That fixed it: https://travis-ci.com/simonw/sqlite-utils/builds/142443259,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542814756,Tests are failing due to missing FTS5, https://github.com/simonw/sqlite-utils/issues/70#issuecomment-569130037,https://api.github.com/repos/simonw/sqlite-utils/issues/70,569130037,MDEyOklzc3VlQ29tbWVudDU2OTEzMDAzNw==,9599,simonw,2019-12-26T20:39:04Z,2019-12-26T20:39:04Z,OWNER,"I hadn't thought about those at all. Are you suggesting a utility mechanism in the library for setting it up so that, for a specific foreign key, rows are deleted from other tables if the row they are pointing at is deleted?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",539204432,Implement ON DELETE and ON UPDATE actions for foreign keys, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-457980966,https://api.github.com/repos/simonw/sqlite-utils/issues/7,457980966,MDEyOklzc3VlQ29tbWVudDQ1Nzk4MDk2Ng==,9599,simonw,2019-01-28T02:29:32Z,2019-01-28T02:29:32Z,OWNER,"Remember to remove this TODO (and turn the `[]` into `()` on this line) as part of this task: https://github.com/simonw/sqlite-utils/blob/5309c5c7755818323a0f5353bad0de98ecc866be/sqlite_utils/cli.py#L78-L80","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674,.insert_all() should accept a generator and process it efficiently, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011885,https://api.github.com/repos/simonw/sqlite-utils/issues/7,458011885,MDEyOklzc3VlQ29tbWVudDQ1ODAxMTg4NQ==,9599,simonw,2019-01-28T06:25:48Z,2019-01-28T06:25:48Z,OWNER,Re-opening for the second bit involving the cli tool.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674,.insert_all() should accept a generator and process it efficiently, https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011906,https://api.github.com/repos/simonw/sqlite-utils/issues/7,458011906,MDEyOklzc3VlQ29tbWVudDQ1ODAxMTkwNg==,9599,simonw,2019-01-28T06:25:55Z,2019-01-28T06:25:55Z,OWNER,"I tested this with a script called `churn_em_out.py` ``` i = 0 while True: i += 1 print( '{""id"": I, ""another"": ""row"", ""number"": J}'.replace(""I"", str(i)).replace( ""J"", str(i + 1) ) ) ``` Then I ran this: ``` python churn_em_out.py | \ sqlite-utils insert /tmp/getbig.db stats - \ --nl --batch-size=10000 ``` And used `watch 'ls -lah /tmp/getbig.db'` to watch the file growing as it had 10,000 lines of junk committed in batches. The memory used by the process never grew about around 50MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403625674,.insert_all() should accept a generator and process it efficiently, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-569130196,https://api.github.com/repos/simonw/sqlite-utils/issues/69,569130196,MDEyOklzc3VlQ29tbWVudDU2OTEzMDE5Ng==,9599,simonw,2019-12-26T20:40:21Z,2019-12-26T20:40:21Z,OWNER,"This is a good idea. Datasette has this in the form of the `--load-extension` CLI argument, e.g. for SpatiaLite here: https://datasette.readthedocs.io/en/stable/spatialite.html#installation Having that available for `sqlite-utils` definitely makes sense.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",534507142,Feature request: enable extensions loading, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710405658,MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==,9599,simonw,2020-10-16T18:42:48Z,2020-10-16T18:42:48Z,OWNER,"Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142,Feature request: enable extensions loading, https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776,https://api.github.com/repos/simonw/sqlite-utils/issues/68,695695776,MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng==,9599,simonw,2020-09-20T04:25:47Z,2020-09-20T04:25:47Z,OWNER,This is a dupe of #130 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531583658,Add support for porter stemming in FTS, https://github.com/simonw/sqlite-utils/pull/67#issuecomment-559108591,https://api.github.com/repos/simonw/sqlite-utils/issues/67,559108591,MDEyOklzc3VlQ29tbWVudDU1OTEwODU5MQ==,9599,simonw,2019-11-27T14:24:59Z,2019-11-27T14:24:59Z,OWNER,Failed due to black testing dependency: https://travis-ci.com/simonw/sqlite-utils/jobs/260995814,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",529376481,Run tests against 3.5 too, https://github.com/simonw/sqlite-utils/pull/67#issuecomment-569844320,https://api.github.com/repos/simonw/sqlite-utils/issues/67,569844320,MDEyOklzc3VlQ29tbWVudDU2OTg0NDMyMA==,9599,simonw,2019-12-31T01:29:43Z,2019-12-31T01:29:43Z,OWNER,I don't really care about 3.5 any more.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",529376481,Run tests against 3.5 too, https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553170650,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553170650,MDEyOklzc3VlQ29tbWVudDU1MzE3MDY1MA==,9599,simonw,2019-11-12T23:49:29Z,2019-11-12T23:49:29Z,OWNER,This relates to this bug: https://github.com/dogsheep/github-to-sqlite/pull/8#issuecomment-549233778,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171011,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553171011,MDEyOklzc3VlQ29tbWVudDU1MzE3MTAxMQ==,9599,simonw,2019-11-12T23:50:52Z,2019-11-12T23:50:52Z,OWNER,"Fixing this is going to be a real pain. There's lots of code out there that uses `sqlite-utils` with the expectation that `upsert()` behaves as it currently does. Maybe I need to introduce new terms for both of these different patterns and deprecate the existing `.upsert()` and `.upsert_all()` since their behaviour can't be changed? Or maybe I fix this and ship `sqlite-utils 2.0` with a breaking change?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553171414,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553171414,MDEyOklzc3VlQ29tbWVudDU1MzE3MTQxNA==,9599,simonw,2019-11-12T23:52:35Z,2019-11-12T23:52:35Z,OWNER,"If I do implement the correct definition of `.upsert()` I think I'll use this pattern, since it works in versions of SQLite prior to 3.24: ```sql INSERT OR IGNORE INTO book(id) VALUES(1001); UPDATE book SET name = 'Programming' WHERE id = 1001; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553526685,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553526685,MDEyOklzc3VlQ29tbWVudDU1MzUyNjY4NQ==,9599,simonw,2019-11-13T17:58:59Z,2019-11-13T17:58:59Z,OWNER,"This warrants making a backwards compatible change, which means I'll need to bump the major version number and release 2.0. I'm going to rename the existing `upsert()` and `upsert_all()` methods to `replace()` and `replace_all()` - then write new `upsert()` and `upsert_all()` methods that implement the correct behavior.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553527384,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553527384,MDEyOklzc3VlQ29tbWVudDU1MzUyNzM4NA==,9599,simonw,2019-11-13T18:00:41Z,2019-11-13T18:00:41Z,OWNER,Is `replace()` a good name here? It doesn't really convey the idea that a brand new record will be created if there isn't an existing one to replace.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528386,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553528386,MDEyOklzc3VlQ29tbWVudDU1MzUyODM4Ng==,9599,simonw,2019-11-13T18:03:10Z,2019-11-13T18:03:54Z,OWNER,"Maybe `inplace()` (combining ""insert"" and ""replace"")? It could be an alias for `.insert(..., replace=True)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553528850,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553528850,MDEyOklzc3VlQ29tbWVudDU1MzUyODg1MA==,9599,simonw,2019-11-13T18:04:20Z,2019-11-13T18:04:20Z,OWNER,This is going to affect the design of the CLI subcommands as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553540146,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553540146,MDEyOklzc3VlQ29tbWVudDU1MzU0MDE0Ng==,9599,simonw,2019-11-13T18:33:30Z,2019-11-13T18:33:30Z,OWNER,"Maybe instead of inventing a new term I should tell people to use `.insert(..., replace=True)` directly. That matches `ignore=True`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-553574011,https://api.github.com/repos/simonw/sqlite-utils/issues/66,553574011,MDEyOklzc3VlQ29tbWVudDU1MzU3NDAxMQ==,9599,simonw,2019-11-13T19:53:45Z,2019-11-13T19:53:45Z,OWNER,"First step: add a `replace=True` argument to `insert()` and `insert_all()` that does the same thing as the current `upsert=True` https://github.com/simonw/sqlite-utils/blob/8dab9fd1ccf571e188eec9ccf606a0c50fccf200/sqlite_utils/db.py#L938-L946","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-554565198,https://api.github.com/repos/simonw/sqlite-utils/issues/66,554565198,MDEyOklzc3VlQ29tbWVudDU1NDU2NTE5OA==,9599,simonw,2019-11-15T23:12:28Z,2019-11-15T23:12:28Z,OWNER,"Urgh this is going to be quite a bit of work, especially in the CLI module which shares an implementation for `upsert` and `insert` in a way that looks like it will have to be unwrapped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-555690319,https://api.github.com/repos/simonw/sqlite-utils/issues/66,555690319,MDEyOklzc3VlQ29tbWVudDU1NTY5MDMxOQ==,9599,simonw,2019-11-19T20:10:17Z,2019-11-19T20:10:17Z,OWNER,"Thinking about this further: I believe every time I've personally used `upsert` in the past (either with the Python library or the CLI tool) I've actually wanted the new behaviour, where ""upsert"" means ""update existing record with these changes, or insert a new record if one does not exist"". So I'm happy with `upsert` doing that, and `insert --replace` being added as an option that does what `upsert` does ta the moment. I'll still ship it as version 2.0 since it's technically a breaking change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569131397,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569131397,MDEyOklzc3VlQ29tbWVudDU2OTEzMTM5Nw==,9599,simonw,2019-12-26T20:49:11Z,2019-12-26T20:49:11Z,OWNER,Don't forget to update the documentation. This will be quite an involved task.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569226620,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569226620,MDEyOklzc3VlQ29tbWVudDU2OTIyNjYyMA==,9599,simonw,2019-12-27T09:05:29Z,2019-12-27T09:05:36Z,OWNER,"I'm going to start by ignoring the existing `upsert` entirely and implementing `.insert(..., replace=True)` and `$ sqlite-utils insert --replace`. Including updating the tests. Then I'll figure out how to implement the new `.upsert()` / `$ sqlite-utils upsert`. Then I'll update the documentation, and ship `sqlite-utils` 2.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569588216,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569588216,MDEyOklzc3VlQ29tbWVudDU2OTU4ODIxNg==,9599,simonw,2019-12-30T05:31:45Z,2019-12-30T05:31:45Z,OWNER,Last step: update changelog and ship 2.0. Then I can close this issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/66#issuecomment-569844426,https://api.github.com/repos/simonw/sqlite-utils/issues/66,569844426,MDEyOklzc3VlQ29tbWVudDU2OTg0NDQyNg==,9599,simonw,2019-12-31T01:30:20Z,2019-12-31T01:30:20Z,OWNER,"I shipped 2.0 - release notes here: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v2 I also wrote about it on my blog: https://simonwillison.net/2019/Dec/30/sqlite-utils-2/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",521868864,"The "".upsert()"" method is misnamed", https://github.com/simonw/sqlite-utils/issues/63#issuecomment-549430429,https://api.github.com/repos/simonw/sqlite-utils/issues/63,549430429,MDEyOklzc3VlQ29tbWVudDU0OTQzMDQyOQ==,9599,simonw,2019-11-04T16:20:35Z,2019-11-04T16:20:35Z,OWNER,"I don't think we need this. We already have a `dogs.create_index([""name""], if_not_exists=True)` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517241040,ensure_index() method, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549418134,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549418134,MDEyOklzc3VlQ29tbWVudDU0OTQxODEzNA==,9599,simonw,2019-11-04T15:54:18Z,2019-11-04T15:54:18Z,OWNER,"This is a good idea. The individual row version of it can work like the `.update(pk, ...)` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425012,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549425012,MDEyOklzc3VlQ29tbWVudDU0OTQyNTAxMg==,9599,simonw,2019-11-04T16:09:08Z,2019-11-04T16:09:08Z,OWNER,Documentation for `table.delete()`: https://github.com/simonw/sqlite-utils/blob/19073d6d972fad9d68dd74c28544cd29083f1c12/docs/python-api.rst#deleting-a-specific-record,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549425364,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549425364,MDEyOklzc3VlQ29tbWVudDU0OTQyNTM2NA==,9599,simonw,2019-11-04T16:09:56Z,2019-11-04T16:09:56Z,OWNER,"Since we have `table.rows_where(where, where_args)` it makes sense to me to also support `table.delete_where(where, where_args)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549429512,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549429512,MDEyOklzc3VlQ29tbWVudDU0OTQyOTUxMg==,9599,simonw,2019-11-04T16:18:48Z,2019-11-04T16:18:48Z,OWNER,Documentation for `.delete_where()`: https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/docs/python-api.rst#deleting-multiple-records,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/62#issuecomment-549435364,https://api.github.com/repos/simonw/sqlite-utils/issues/62,549435364,MDEyOklzc3VlQ29tbWVudDU0OTQzNTM2NA==,9599,simonw,2019-11-04T16:30:34Z,2019-11-04T16:30:34Z,OWNER,Released as 1.12.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",500783373,[enhancement] Method to delete a row in python, https://github.com/simonw/sqlite-utils/issues/61#issuecomment-549432592,https://api.github.com/repos/simonw/sqlite-utils/issues/61,549432592,MDEyOklzc3VlQ29tbWVudDU0OTQzMjU5Mg==,9599,simonw,2019-11-04T16:25:01Z,2019-11-04T16:25:01Z,OWNER,"Yeah I've thought about this a bit and I'm OK leaving it out. The core idea of `sqlite-utils` is that if you can create a list, iterator or generator of Python dictionaries you can efficiently insert those into a SQLite table. The `--csv` function is actually implemented as just a few lines of code which turn that incoming CSV into a generator of dictionaries: https://github.com/simonw/sqlite-utils/blob/169ea455fc1f1d5e5b6e44cb339ba7ffa9d49c31/sqlite_utils/cli.py#L364-L368 I could turn this into a reusable helper function but since it wouldn't have anything to do with database inserts (it would just be a helper that turns a CSV into a generator of dictionaries) it doesn't feel like it fits well in the Python library section of this package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",491219910,importing CSV to SQLite as library, https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843465748,https://api.github.com/repos/simonw/sqlite-utils/issues/606,1843465748,IC_kwDOCGYnMM5t4Q4U,9599,simonw,2023-12-06T18:36:51Z,2023-12-06T18:36:51Z,OWNER,I'll add `bytes` too - `float` already works. This makes sense because when you are working with the Python API you use `str` and `float` and `bytes` and `int` to specify column types.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029161033,str and int as aliases for text and integer, https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843579184,https://api.github.com/repos/simonw/sqlite-utils/issues/606,1843579184,IC_kwDOCGYnMM5t4skw,9599,simonw,2023-12-06T19:43:55Z,2023-12-06T19:43:55Z,OWNER,"Updated documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-add-column - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029161033,str and int as aliases for text and integer, https://github.com/simonw/sqlite-utils/issues/605#issuecomment-1846554637,https://api.github.com/repos/simonw/sqlite-utils/issues/605,1846554637,IC_kwDOCGYnMM5uEDAN,9599,simonw,2023-12-08T05:07:54Z,2023-12-08T05:07:54Z,OWNER,"Thanks for opening an issue - this should help future Google searchers figure out what's going on here. Another approach here could be to store large integers as `TEXT` in SQLite (or even as `BLOB`). Both storing as `REAL` and storing as `TEXT/BLOB` feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2007893839,Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here?, https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843586503,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843586503,IC_kwDOCGYnMM5t4uXH,9599,simonw,2023-12-06T19:49:10Z,2023-12-06T19:49:29Z,OWNER,"This looks really great on first glance - design is good, implementation is solid, tests and documentation look great. Looks like a couple of `mypy` failures in the tests at the moment: ``` mypy sqlite_utils tests sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type ""type[Table]"", variable has type ""type[View]"") [assignment] tests/test_lookup.py:156: error: Name ""test_lookup_new_table"" already defined on line 5 [no-redef] Found 2 errors in 2 files (checked 54 source files) Error: Process completed with exit code 1. ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support, https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1846560096,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1846560096,IC_kwDOCGYnMM5uEEVg,9599,simonw,2023-12-08T05:16:44Z,2023-12-08T05:17:20Z,OWNER,"Also tested this manually like so: ```bash sqlite-utils create-table strict.db strictint id integer size integer --strict sqlite-utils create-table strict.db notstrictint id integer size integer sqlite-utils install sqlite-utils-shell sqlite-utils shell strict.db ``` ``` Attached to strict.db Type 'exit' to exit. sqlite-utils> insert into strictint (size) values (4); 1 row affected sqlite-utils> insert into strictint (size) values ('four'); An error occurred: cannot store TEXT value in INTEGER column strictint.size sqlite-utils> insert into notstrictint (size) values ('four'); 1 row affected sqlite-utils> commit; Done ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support, https://github.com/simonw/sqlite-utils/issues/603#issuecomment-1846555822,https://api.github.com/repos/simonw/sqlite-utils/issues/603,1846555822,IC_kwDOCGYnMM5uEDSu,9599,simonw,2023-12-08T05:09:55Z,2023-12-08T05:10:31Z,OWNER,"I'm unable to replicate this issue. This is with a fresh install of `sqlite-utils==3.35.2`: ``` (base) ~ python3.12 Python 3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db[""foo""].insert({""bar"": 1}) >>> import sys >>> sys.version '3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)]' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1988525411,Pyhton 3.12 Bug report, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793263638,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793263638,IC_kwDOCGYnMM5q4wgW,9599,simonw,2023-11-04T00:19:58Z,2023-11-04T00:19:58Z,OWNER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793265952,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793265952,IC_kwDOCGYnMM5q4xEg,9599,simonw,2023-11-04T00:25:34Z,2023-11-04T00:25:34Z,OWNER,The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793268126,IC_kwDOCGYnMM5q4xme,9599,simonw,2023-11-04T00:31:34Z,2023-11-04T00:31:34Z,OWNER,"Testing this manually on macOS using Docker Desk top like this: ```bash docker run -it --rm arm64v8/ubuntu /bin/bash ``` Then inside the container: ```bash uname -m ``` Outputs: `aarch64` Then: ```bash apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y cd /tmp git clone https://github.com/simonw/sqlite-utils cd sqlite-utils python3 -m venv venv source venv/bin/activate pip install -e '.[test]' sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Which output: ``` Traceback (most recent call last): File ""/tmp/sqlite-utils/venv/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1157, in __call__ return self.main(*args, **kwargs) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1078, in main rv = self.invoke(ctx) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 783, in invoke return __callback(*args, **kwargs) File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 1959, in memory _load_extensions(db, load_extension) File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 3232, in _load_extensions if "":"" in ext: TypeError: argument of type 'NoneType' is not iterable ``` Then I ran this: ```bash git checkout -b MikeCoats-spatialite-paths-linux-arm main git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm ``` And now: ```bash sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Outputs: ```json [{""spatialite_version()"": ""5.0.1""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793269219,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793269219,IC_kwDOCGYnMM5q4x3j,9599,simonw,2023-11-04T00:34:33Z,2023-11-04T00:34:33Z,OWNER,"The GIS tests now pass in that container too: ```bash pytest tests/test_gis.py ``` ``` ======================== test session starts ========================= platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0 rootdir: /tmp/sqlite-utils plugins: hypothesis-6.88.1 collected 12 items tests/test_gis.py ............ [100%] ========================= 12 passed in 0.48s ========================= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path, https://github.com/simonw/sqlite-utils/issues/6#issuecomment-457978729,https://api.github.com/repos/simonw/sqlite-utils/issues/6,457978729,MDEyOklzc3VlQ29tbWVudDQ1Nzk3ODcyOQ==,9599,simonw,2019-01-28T02:12:19Z,2019-01-28T02:12:19Z,OWNER,Will need to solve #7 for this to become truly efficient.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403624090,"""sqlite-utils insert"" should support newline-delimited JSON", https://github.com/simonw/sqlite-utils/issues/599#issuecomment-1793268750,https://api.github.com/repos/simonw/sqlite-utils/issues/599,1793268750,IC_kwDOCGYnMM5q4xwO,9599,simonw,2023-11-04T00:33:25Z,2023-11-04T00:33:25Z,OWNER,"See details of how I tested this here: - https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126 Short version: having applied this fix, the following command (on simulated `aarch64`): ```bash sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Outputs: ```json [{""spatialite_version()"": ""5.0.1""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1976986318,Cannot find spatialite on arm64 linux, https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793272429,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793272429,IC_kwDOCGYnMM5q4ypt,9599,simonw,2023-11-04T00:40:34Z,2023-11-04T00:40:34Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132,Fixed issue #433 - CLI eats cursor, https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793274485,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793274485,IC_kwDOCGYnMM5q4zJ1,9599,simonw,2023-11-04T00:46:55Z,2023-11-04T00:46:55Z,OWNER,"Manually tested. Before: ![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/3bdd30ea-1a54-4fec-b42d-793130a17bc1) After: ![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/015d4e4e-b40c-4a93-81f5-1a4adef69b11) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132,Fixed issue #433 - CLI eats cursor, https://github.com/simonw/sqlite-utils/pull/596#issuecomment-1793274869,https://api.github.com/repos/simonw/sqlite-utils/issues/596,1793274869,IC_kwDOCGYnMM5q4zP1,9599,simonw,2023-11-04T00:47:55Z,2023-11-04T00:47:55Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1919296686,"Fixes mapping for time fields related to mysql, closes #522", https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714919806,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714919806,IC_kwDOCGYnMM5mN5l-,9599,simonw,2023-09-12T03:49:41Z,2023-09-12T03:49:41Z,OWNER,"Digging in a bit more: ```pycon >>> pprint(list(db.query('PRAGMA foreign_key_list(courses)'))) [{'from': 'campus_name', 'id': 0, 'match': 'NONE', 'on_delete': 'NO ACTION', 'on_update': 'NO ACTION', 'seq': 0, 'table': 'departments', 'to': 'campus_name'}, {'from': 'dept_code', 'id': 0, 'match': 'NONE', 'on_delete': 'NO ACTION', 'on_update': 'NO ACTION', 'seq': 1, 'table': 'departments', 'to': 'dept_code'}] ``` I think the way you tell it's a compound foreign key is that both of those have the same `id` value - of `0` - but they then have two different `seq` values of `0` and `1`. Right now I ignore those columns entirely: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L1523-L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971,Represent compound foreign keys in table.foreign_keys output, https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714920708,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714920708,IC_kwDOCGYnMM5mN50E,9599,simonw,2023-09-12T03:51:13Z,2023-09-12T03:51:13Z,OWNER,"Changing this without breaking backwards compatibility (and forcing a 4.0 release) will be tricky, because `ForeignKey()` is a `namedtuple`: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L148-L150 I could swap it out for a `dataclass` and add those extra columns, but I need to make sure that code like this still works: ```python for table, column, other_table, other_column in table.foreign_keys: # ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971,Represent compound foreign keys in table.foreign_keys output, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710930934,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710930934,IC_kwDOCGYnMM5l-rv2,9599,simonw,2023-09-08T00:47:57Z,2023-09-08T00:47:57Z,OWNER,"That's odd, I wrote a test for this just now and it passes already: ```python def test_transform_preserves_rowids(fresh_db): # Create a rowid table fresh_db[""places""].insert_all( ( {""name"": ""Paris"", ""country"": ""France""}, {""name"": ""London"", ""country"": ""UK""}, {""name"": ""New York"", ""country"": ""USA""}, ), ) assert fresh_db[""places""].use_rowid previous_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, name from places"") ) # Transform it fresh_db[""places""].transform(column_order=(""country"", ""name"")) # Should be the same next_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, name from places"") ) assert previous_rows == next_rows ``` So maybe I'm wrong about the cause of that bug?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710931605,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710931605,IC_kwDOCGYnMM5l-r6V,9599,simonw,2023-09-08T00:49:02Z,2023-09-08T00:49:02Z,OWNER,"I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710933716,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710933716,IC_kwDOCGYnMM5l-sbU,9599,simonw,2023-09-08T00:52:42Z,2023-09-08T00:52:42Z,OWNER,"I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key. The reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows. But I tried this test and it passed, too: ```python def test_transform_preserves_rowids(fresh_db): fresh_db[""places""].insert_all( [ {""id"": ""1"", ""name"": ""Paris"", ""country"": ""France""}, {""id"": ""2"", ""name"": ""London"", ""country"": ""UK""}, {""id"": ""3"", ""name"": ""New York"", ""country"": ""USA""}, ], pk=""id"", ) previous_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"") ) # Transform it fresh_db[""places""].transform(column_order=(""country"", ""name"")) # Should be the same next_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"") ) assert previous_rows == next_rows ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710934448,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710934448,IC_kwDOCGYnMM5l-smw,9599,simonw,2023-09-08T00:54:03Z,2023-09-08T00:54:03Z,OWNER,"Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3. If I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710935270,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710935270,IC_kwDOCGYnMM5l-szm,9599,simonw,2023-09-08T00:55:30Z,2023-09-08T00:55:30Z,OWNER,"Yes! That recreated the bug: ``` > assert previous_rows == next_rows E AssertionError: assert equals failed E [ [ E (1, '1', 'Paris'), (1, '1', 'Paris'), E (3, '3', 'New York'), (2, '3', 'New York'), E (4, '4', 'London'), (3, '4', 'London'), E ] ... E ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1712895580,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1712895580,IC_kwDOCGYnMM5mGLZc,9599,simonw,2023-09-10T17:46:41Z,2023-09-10T17:46:41Z,OWNER,"In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`: https://www.sqlite.org/lang_vacuum.html#how_vacuum_works > The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid). So this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708695907,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1708695907,IC_kwDOCGYnMM5l2KFj,9599,simonw,2023-09-06T16:15:59Z,2023-09-06T16:19:14Z,OWNER,"The test failure was while installing `numpy`, relating to importing `distutils` - maybe relevant: - https://github.com/pypa/setuptools/issues/3661 ``` 25h Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' ERROR: Exception: Traceback (most recent call last): ... File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py"", line 697, in get_requires_for_build_wheel return super().get_requires_for_build_wheel(config_settings=cs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 166, in get_requires_for_build_wheel return self._call_hook('get_requires_for_build_wheel', { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 321, in _call_hook raise BackendUnavailable(data.get('traceback', '')) pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py"", line 77, in _build_backend obj = import_module(mod_path) ^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py"", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File """", line 1381, in _gcd_import File """", line 1354, in _find_and_load File """", line 1304, in _find_and_load_unlocked File """", line 488, in _call_with_frames_removed File """", line 1381, in _gcd_import File """", line 1354, in _find_and_load File """", line 1325, in _find_and_load_unlocked File """", line 929, in _load_unlocked File """", line 994, in exec_module File """", line 488, in _call_with_frames_removed File ""/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py"", line 10, in import distutils.core ModuleNotFoundError: No module named 'distutils' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,Test against Python 3.12 preview, https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1793278279,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1793278279,IC_kwDOCGYnMM5q40FH,9599,simonw,2023-11-04T00:58:03Z,2023-11-04T00:58:03Z,OWNER,I'm going to abandon this PR and ship the 3.12 testing change directly to `main`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,Test against Python 3.12 preview, https://github.com/simonw/sqlite-utils/issues/590#issuecomment-1704387161,https://api.github.com/repos/simonw/sqlite-utils/issues/590,1704387161,IC_kwDOCGYnMM5lluJZ,9599,simonw,2023-09-03T19:50:36Z,2023-09-03T19:50:36Z,OWNER,"Maybe just populate `db.memory: bool` and `db.memory_name: Optional[str]` for this, then document them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879214365,Ability to tell if a Database is an in-memory one, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704383901,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704383901,IC_kwDOCGYnMM5lltWd,9599,simonw,2023-09-03T19:34:05Z,2023-09-03T19:34:05Z,OWNER,"For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager. I played with this idea a bit: ```python with db.register_functions(md5, md5_random): db.query(...) ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384111,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384111,IC_kwDOCGYnMM5lltZv,9599,simonw,2023-09-03T19:35:03Z,2023-09-03T19:35:03Z,OWNER,"Normally in Python/`sqlite3` you de-register a function by passing `None` to it. You can't do that with `db.register_function()` at the moment because a `fn` of `None` does something else: https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384393,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384393,IC_kwDOCGYnMM5llteJ,9599,simonw,2023-09-03T19:36:34Z,2023-09-03T19:36:34Z,OWNER,"Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b Still needs tests and documentation (and some more thought to make sure it's doing the right thing).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions, https://github.com/simonw/sqlite-utils/issues/588#issuecomment-1694823972,https://api.github.com/repos/simonw/sqlite-utils/issues/588,1694823972,IC_kwDOCGYnMM5lBPYk,9599,simonw,2023-08-28T00:41:54Z,2023-08-28T00:41:54Z,OWNER,Tips on typing `**kwargs`: https://adamj.eu/tech/2021/05/11/python-type-hints-args-and-kwargs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1868713944,`table.get(column=value)` option for retrieving things not by their primary key, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096129,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096129,IC_kwDOCGYnMM5kcIbB,9599,simonw,2023-08-19T20:03:00Z,2023-08-19T20:03:00Z,OWNER,"Simplest possible recreation of the bug: ```bash python -c ' import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute("""""" CREATE TABLE ""logs"" ( [id] INTEGER PRIMARY KEY, [chat_id] INTEGER REFERENCES [log]([id]), [reply_to_id] INTEGER ); """""") db[""logs""].add_foreign_key(""reply_to_id"", ""logs"", ""id"") ' ``` That `chat_id` line is the line that causes the problem - because it is defining a reference to a table that no longer exists!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096284,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096284,IC_kwDOCGYnMM5kcIdc,9599,simonw,2023-08-19T20:03:59Z,2023-08-19T20:03:59Z,OWNER,"Although this is revealing a problem in the underlying code (that schema is invalid), it also represents a regression: `sqlite-utils 3.34` ran this just fine, but it fails on `sqlite-utils 3.35` due to the change made in: - #577","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096381,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096381,IC_kwDOCGYnMM5kcIe9,9599,simonw,2023-08-19T20:04:32Z,2023-08-19T20:04:32Z,OWNER,I'm inclined to say this isn't a bug in `sqlite-utils` though - it's a bug in the code that calls it. So I'm not going to fix it here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683396150,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683396150,IC_kwDOCGYnMM5kVpY2,9599,simonw,2023-08-18T06:02:18Z,2023-08-18T06:06:31Z,OWNER,"More notes in here: - https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683392873 Not all Python/SQLite installations exhibit this problem by default! It turns out this is controlled by the `legacy_alter_table` pragma: https://sqlite.org/pragma.html#pragma_legacy_alter_table If that PRAGMA is turned on (default in newer SQLites) then `alter table` will error if you try to rename a table that is referenced in a view. Here's a one-liner to test if it is on or not: ```bash python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""PRAGMA legacy_alter_table"").fetchall())' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683398866,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683398866,IC_kwDOCGYnMM5kVqDS,9599,simonw,2023-08-18T06:05:50Z,2023-08-18T06:06:42Z,OWNER,"Options: - Provide a `recreate_views: bool` parameter to `table.transform()` controlling if views that might reference this table are stashed and dropped and recreated within a transaction as part of the operation. But should that be `True` or `False` by default? - Read that `PRAGMA` and automatically do that view workaround if it's turned on - Toggle that `PRAGMA` off for the duration of the `.transform()` operation and on again at the end. Does it only affect the current connection? - Try the `transform()` in a transaction, detect the `""error in view""`, `""no such table""`error, if spotted then do the VIEW workaround and try again I'm on the fence as to which of these I like the most. I'm tempted to go with the one which just drops VIEWS and recreates them all the time, because it feels simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683404978,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683404978,IC_kwDOCGYnMM5kVriy,9599,simonw,2023-08-18T06:13:46Z,2023-08-18T06:13:46Z,OWNER,"I shipped the view recreating fix in `datasette-edit-schema`, so at least I can start exercising that fix and see if it has any weird issues.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683195669,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683195669,IC_kwDOCGYnMM5kU4cV,9599,simonw,2023-08-18T01:24:57Z,2023-08-18T01:24:57Z,OWNER,"Currently: ```bash sqlite-utils transform --help ``` ``` Usage: sqlite-utils transform [OPTIONS] PATH TABLE Transform a table beyond the capabilities of ALTER TABLE Example: sqlite-utils transform mydb.db mytable \ --drop column1 \ --rename column2 column_renamed Options: --type ... Change column type to INTEGER, TEXT, FLOAT or BLOB --drop TEXT Drop this column --rename ... Rename this column to X -o, --column-order TEXT Reorder columns --not-null TEXT Set this column to NOT NULL --not-null-false TEXT Remove NOT NULL from this column --pk TEXT Make this column the primary key --pk-none Remove primary key (convert to rowid table) --default ... Set default value for this column --default-none TEXT Remove default from this column --drop-foreign-key TEXT Drop foreign key constraint for this column --sql Output SQL without executing it --load-extension TEXT Path to SQLite extension, with optional :entrypoint -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683197882,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683197882,IC_kwDOCGYnMM5kU4-6,9599,simonw,2023-08-18T01:25:53Z,2023-08-18T01:25:53Z,OWNER,"Probably most relevant here is this snippet from: ```bash sqlite-utils create-table --help ``` ``` --default ... Default value that should be set for a column --fk ... Column, other table, other column to set as a foreign key ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683198740,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683198740,IC_kwDOCGYnMM5kU5MU,9599,simonw,2023-08-18T01:26:47Z,2023-08-18T01:26:47Z,OWNER,"The only CLI feature that supports providing just the column name appears to be this: ```bash sqlite-utils add-foreign-key --help ``` ``` Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE] [OTHER_COLUMN] Add a new foreign key constraint to an existing table Example: sqlite-utils add-foreign-key my.db books author_id authors id WARNING: Could corrupt your database! Back up your database file first. ``` I can drop that WARNING now since I'm not writing to `sqlite_master` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683200128,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683200128,IC_kwDOCGYnMM5kU5iA,9599,simonw,2023-08-18T01:29:00Z,2023-08-18T01:29:00Z,OWNER,I'm not going to implement the `foreign_keys=` option that entirely replaces existing foreign keys - I'll just do a `--add-foreign-key` multi-option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683201239,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683201239,IC_kwDOCGYnMM5kU5zX,9599,simonw,2023-08-18T01:30:46Z,2023-08-18T01:30:46Z,OWNER,"Help can now look like this: ``` --drop-foreign-key TEXT Drop foreign key constraint for this column --add-foreign-key ... Add a foreign key constraint from a column to another table with another column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683212074,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683212074,IC_kwDOCGYnMM5kU8cq,9599,simonw,2023-08-18T01:43:54Z,2023-08-18T01:43:54Z,OWNER,"Some manual testing: ```bash sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [places] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER ); ``` ```bash sqlite-utils create-table /tmp/t.db country id integer name text sqlite-utils create-table /tmp/t.db city id integer name text sqlite-utils create-table /tmp/t.db continent id integer name text sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [places] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER ); CREATE TABLE [country] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [city] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [continent] ( [id] INTEGER, [name] TEXT ); ``` ```bash sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [country] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [city] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [continent] ( [id] INTEGER, [name] TEXT ); CREATE TABLE ""places"" ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER REFERENCES [country]([id]), [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ); ``` ```bash sqlite-utils transform /tmp/t.db places --drop-foreign-key country sqlite-utils schema /tmp/t.db places ``` ```sql CREATE TABLE ""places"" ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683217284,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683217284,IC_kwDOCGYnMM5kU9uE,9599,simonw,2023-08-18T01:50:21Z,2023-08-18T01:50:21Z,OWNER,"And a test of the `--sql` option: ```bash sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id sqlite-utils create-table /tmp/t.db country id integer name text sqlite-utils create-table /tmp/t.db city id integer name text sqlite-utils create-table /tmp/t.db continent id integer name text sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id --sql ``` Outputs: ```sql CREATE TABLE [places_new_6a705d2f5a13] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER REFERENCES [country]([id]), [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ); INSERT INTO [places_new_6a705d2f5a13] ([id], [name], [country], [city], [continent]) SELECT [id], [name], [country], [city], [continent] FROM [places]; DROP TABLE [places]; ALTER TABLE [places_new_6a705d2f5a13] RENAME TO [places]; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,simonw,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left: ``` # Soundness check foreign_keys point to existing tables for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): > raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) E sqlite_utils.db.AlterError: No such column: breeds.rowid sqlite_utils/db.py:882: AlterError ==== short test summary info ==== FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid ==== 1 failed, 378 deselected in 0.49s ==== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,simonw,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,simonw,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem: ```python from sqlite_utils import Database fresh_db = Database(memory=True) fresh_db.create_table(""dogs"", {""name"": str}) fresh_db.create_table(""breeds"", {""name"": str}) fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"") ``` Traceback: ``` Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column self.add_foreign_key(col_name, fk, fk_col) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys self[table].transform(add_foreign_keys=fks) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform sqls = self.transform_sql( ^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql self.db.create_table_sql( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql raise AlterError( sqlite_utils.db.AlterError: No such column: breeds.rowid ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,simonw,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line: ```python fresh_db.create_table(""breeds"", {""name"": str}) ``` Has this schema: ```sql CREATE TABLE [breeds] ( [name] TEXT ); ``` SQLite creates an invisible `rowid` column for it automatically. On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added: ```sql (Pdb) print(fresh_db.schema) CREATE TABLE [dogs] ( [name] TEXT , [breed_id] INTEGER, FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid]) ); CREATE TABLE [breeds] ( [name] TEXT ); ``` But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884 Here's what the debugger reveals about this code: ```python for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) ``` ``` (Pdb) fk ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid') (Pdb) self[fk.other_table].columns [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,simonw,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures: ``` sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"") [assignment] sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[, ] = ..."") [var-annotated] sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform"" [union-attr] sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] Found 5 errors in 1 file (checked 56 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683137259,IC_kwDOCGYnMM5kUqLr,9599,simonw,2023-08-18T00:06:59Z,2023-08-18T00:06:59Z,OWNER,"The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683138953,IC_kwDOCGYnMM5kUqmJ,9599,simonw,2023-08-18T00:09:20Z,2023-08-18T00:09:20Z,OWNER,"Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop: ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683139304,IC_kwDOCGYnMM5kUqro,9599,simonw,2023-08-18T00:09:56Z,2023-08-18T00:09:56Z,OWNER,"Upgrading `flake8` locally replicated the error: ``` pip install -U flake8 flake8 ``` ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683143723,IC_kwDOCGYnMM5kUrwr,9599,simonw,2023-08-18T00:14:52Z,2023-08-18T00:14:52Z,OWNER,"Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times: > The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries. > > `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up. > > Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`. That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145110,IC_kwDOCGYnMM5kUsGW,9599,simonw,2023-08-18T00:16:28Z,2023-08-18T00:16:48Z,OWNER,"One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`: https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708 I should write tests for them too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145819,IC_kwDOCGYnMM5kUsRb,9599,simonw,2023-08-18T00:17:26Z,2023-08-18T00:17:26Z,OWNER,Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683164661,IC_kwDOCGYnMM5kUw31,9599,simonw,2023-08-18T00:45:53Z,2023-08-18T00:45:53Z,OWNER,"More updated documentation: - https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform - https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636,https://api.github.com/repos/simonw/sqlite-utils/issues/583,1683110636,IC_kwDOCGYnMM5kUjrs,9599,simonw,2023-08-17T23:31:27Z,2023-08-17T23:31:27Z,OWNER,"Spotted this while working on: - #577 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855836914,Get rid of test.utils.collapse_whitespace, https://github.com/simonw/sqlite-utils/issues/581#issuecomment-1652496702,https://api.github.com/repos/simonw/sqlite-utils/issues/581,1652496702,IC_kwDOCGYnMM5ifxk-,9599,simonw,2023-07-26T21:07:45Z,2023-07-26T21:07:45Z,OWNER,Docs: https://sqlite-utils.datasette.io/en/latest/cli.html#using-the-debugger,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823160748,`sqlite-utils convert --pdb` option, https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593,https://api.github.com/repos/simonw/sqlite-utils/issues/58,710399593,MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==,9599,simonw,2020-10-16T18:39:31Z,2020-10-16T18:39:31Z,OWNER,I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488293926,Support enabling FTS on views, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648323482,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648323482,IC_kwDOCGYnMM5iP2ua,9599,simonw,2023-07-24T17:31:56Z,2023-07-24T17:31:56Z,OWNER,"The main blocker here is coming up with a design. The challenge is cleanly integrating it with the existing format options: https://github.com/simonw/sqlite-utils/blob/8bee14588687b66c54c7a3dfae5de2f9cc7cac3f/docs/cli-reference.rst#L114-L132 I'd like to avoid breaking backwards compatibility. The most obvious solution is to add a `--format x` option. This is slightly confusing as `--fmt` already exists. Or... `--fmt` could be enhanced to handle plugins too - and the existing set of formats (from the `tabulate` integration) could be refactored to use the new hook. If I ever do release `sqlite-utils 4` I'm tempted to move that tabulate stuff into a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648324312,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648324312,IC_kwDOCGYnMM5iP27Y,9599,simonw,2023-07-24T17:32:21Z,2023-07-24T17:32:21Z,OWNER,I'm currently leaning very slightly towards `--format geojson` - and explaining in the docs that `--format` is for formats added by plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648325682,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648325682,IC_kwDOCGYnMM5iP3Qy,9599,simonw,2023-07-24T17:33:10Z,2023-07-24T17:33:10Z,OWNER,"A related feature would be support for plugins to add new ways of _ingesting_ data - currently `sqlite-utils insert` works against JSON, newline-JSON, CSV and TSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683066934,IC_kwDOCGYnMM5kUZA2,9599,simonw,2023-08-17T22:37:18Z,2023-08-17T22:37:18Z,OWNER,"I'm certain this could work. It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872 Improving this code to support adding foreign keys as well would be pretty simple. And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683068505,IC_kwDOCGYnMM5kUZZZ,9599,simonw,2023-08-17T22:39:17Z,2023-08-17T22:39:38Z,OWNER,"This would help address these issues, among potentially many others: - https://github.com/simonw/llm/issues/60 - https://github.com/simonw/llm/issues/116 - https://github.com/simonw/llm/issues/123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683071519,IC_kwDOCGYnMM5kUaIf,9599,simonw,2023-08-17T22:42:28Z,2023-08-17T22:42:28Z,OWNER,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149 At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`. Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074009,IC_kwDOCGYnMM5kUavZ,9599,simonw,2023-08-17T22:45:29Z,2023-08-17T22:47:08Z,OWNER,"Actually I think `table.transform()` might get the following optional arguments: ```python def transform( self, *, # ... # This one exists already: drop_foreign_keys: Optional[Iterable] = None, # These two are new. This one specifies keys to add: add_foreign_keys: Optional[ForeignKeysType] = None, # Or this one causes them all to be replaced with the new definitions: foreign_keys: Optional[ForeignKeysType] = None, ``` There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074546,IC_kwDOCGYnMM5kUa3y,9599,simonw,2023-08-17T22:46:18Z,2023-08-17T22:46:18Z,OWNER,"Maybe this: ```python drop_foreign_keys: Optional[Iterable] = None, ``` Should be this: ```python drop_foreign_keys: Optional[Iterable[str]] = None, ``` Because it takes a list of column names that should have their foreign keys dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074857,IC_kwDOCGYnMM5kUa8p,9599,simonw,2023-08-17T22:46:40Z,2023-08-17T22:46:40Z,OWNER,"As a reminder: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683076325,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683076325,IC_kwDOCGYnMM5kUbTl,9599,simonw,2023-08-17T22:48:36Z,2023-08-17T22:48:36Z,OWNER,"I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism. It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683098094,IC_kwDOCGYnMM5kUgnu,9599,simonw,2023-08-17T23:15:36Z,2023-08-17T23:15:36Z,OWNER,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133 Needs updating like so: ```diff diff --git a/tests/test_extract.py b/tests/test_extract.py index 70ad0cf..fd52534 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db): assert fresh_db[""tree""].schema == ( 'CREATE TABLE ""tree"" (\n' "" [name] TEXT,\n"" - "" [common_name_latin_name_id] INTEGER,\n"" - "" FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n"" + "" [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n"" "")"" ) assert ( ``` Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in. I don't think this should count as a breaking change release though, but it's still worth noting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1684235760,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1684235760,IC_kwDOCGYnMM5kY2Xw,9599,simonw,2023-08-18T17:43:11Z,2023-08-18T17:43:11Z,OWNER,"Here's a new plugin that brings back the `sqlite_master` modifying version, for those that can use it: https://github.com/simonw/sqlite-utils-fast-fks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646752814,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646752814,IC_kwDOCGYnMM5iJ3Qu,9599,simonw,2023-07-23T05:43:13Z,2023-07-23T05:43:13Z,OWNER,"Using `llm` in my `sqlite-utils` local checkout: ```bash git diff 0.2 0.3 | llm -s 'release notes as markdown bullet points' ``` > - Added a new `Table` class in the `db` module which represents a table in a SQLite database > - The `Table` class has properties that provide information about the table, such as column names, foreign keys, and schema > - The `Table` class also has a `count` property that returns the number of rows in the table > - The `Table` class has a `create` method that can be used to create a new table with specified columns, primary key, and foreign keys > - Updated the `Database` class in the `__init__` module to include the `Table` class and expose it via the `__all__` variable > - Added new tests for creating tables and introspecting tables ```bash git diff 0.3 0.3.1 | llm -s 'release notes as markdown bullet points' ``` > - Updated PyPI version badge to display the latest version > - Updated Travis CI badge to link to the master branch > - Added a badge for documentation status > - Added a license badge > - Added a description of the package > - Added installation instructions > - Added a link to the documentation > - Added a ""Related projects"" section and linked to Datasette and csvs-to-sqlite > - Updated the version in setup.py to 0.3.1 ```bash git diff 0.3.1 0.4 | llm -s 'release notes as markdown bullet points' ``` > - Added an `enable_fts` method to the `Table` class that enables full-text search on specified columns > - Added a `populate_fts` method to the `Table` class that refreshes the search index for full-text search > - Added a `search` method to the `Table` class that allows searching the table using full-text search > - Updated the `setup.py` script to version 0.4 > - Updated the `db.py` file to include the `enable_fts`, `populate_fts`, and `search` methods > - Added test cases for the `enable_fts`, `populate_fts`, and `search` methods in the `test_enable_fts.py` file > - Added test cases for the `count`, `columns`, and `schema` properties in the `test_introspect.py` file > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390,Backfill the release notes prior to 0.4, https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646753678,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646753678,IC_kwDOCGYnMM5iJ3eO,9599,simonw,2023-07-23T05:49:51Z,2023-07-23T05:49:51Z,OWNER,Done - bottom of https://sqlite-utils.datasette.io/en/latest/changelog.html#id116,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390,Backfill the release notes prior to 0.4, https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646687461,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646687461,IC_kwDOCGYnMM5iJnTl,9599,simonw,2023-07-22T23:01:44Z,2023-07-22T23:01:44Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/3f80a026983d3e634f05a46f2a6da162b5139dd9/sqlite_utils/db.py#L346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568,Python API ability to opt-out of connection plugins, https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646688288,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646688288,IC_kwDOCGYnMM5iJngg,9599,simonw,2023-07-22T23:08:22Z,2023-07-22T23:08:22Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568,Python API ability to opt-out of connection plugins, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646686477,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646686477,IC_kwDOCGYnMM5iJnEN,9599,simonw,2023-07-22T22:52:56Z,2023-07-22T22:52:56Z,OWNER,"Alex built this in: - #573","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688339,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646688339,IC_kwDOCGYnMM5iJnhT,9599,simonw,2023-07-22T23:08:49Z,2023-07-22T23:08:49Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#prepare-connection-conn,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688748,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646688748,IC_kwDOCGYnMM5iJnns,9599,simonw,2023-07-22T23:12:03Z,2023-07-22T23:13:14Z,OWNER,"Just tried this out by adding the example from the docs to my `sqlite-utils-hello-world` plugin and running: ```bash sqlite-utils install -e ../sqlite-utils-hello-world ``` ``` Obtaining file:///Users/simon/Dropbox/Development/sqlite-utils-hello-world ... ``` Then: ```bash sqlite-utils memory ""select hello('simon')"" ``` ```json [{""hello('simon')"": ""Hello, simon!""}] ``` Also: ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> list(db.query(""select hello('simon')"")) [{""hello('simon')"": 'Hello, simon!'}] >>> db2 = sqlite_utils.Database(memory=True, execute_plugins=False) >>> list(db2.query(""select hello('simon')"")) Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 494, in query cursor = self.execute(sql, params or tuple()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 512, in execute return self.conn.execute(sql, parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sqlean.dbapi2.OperationalError: no such function: hello ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686332,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686332,IC_kwDOCGYnMM5iJnB8,9599,simonw,2023-07-22T22:52:01Z,2023-07-22T22:52:01Z,OWNER,I was literally seconds away from shipping version 3.34 but I this looks good so I'm going to try and get it in there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686675,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686675,IC_kwDOCGYnMM5iJnHT,9599,simonw,2023-07-22T22:54:38Z,2023-07-22T22:54:38Z,OWNER," Glitch in the rendered documentation from https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646687103,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646687103,IC_kwDOCGYnMM5iJnN_,9599,simonw,2023-07-22T22:58:35Z,2023-07-22T22:58:35Z,OWNER,"https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook, https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646660777,https://api.github.com/repos/simonw/sqlite-utils/issues/572,1646660777,IC_kwDOCGYnMM5iJgyp,9599,simonw,2023-07-22T19:59:13Z,2023-07-22T19:59:13Z,OWNER,I don't know where that is coming from! I can't see `textual` as a dependency we are pulling in anywhere.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816877910,Don't test Python 3.7 against textual, https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646681192,https://api.github.com/repos/simonw/sqlite-utils/issues/572,1646681192,IC_kwDOCGYnMM5iJlxo,9599,simonw,2023-07-22T22:12:08Z,2023-07-22T22:12:08Z,OWNER,Found it: https://github.com/simonw/sqlite-utils/blob/18f190e28334d821be78a1dbbf31d7610fc1f9c1/.github/workflows/test.yml#L29-L31,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816877910,Don't test Python 3.7 against textual, https://github.com/simonw/sqlite-utils/issues/571#issuecomment-1646682686,https://api.github.com/repos/simonw/sqlite-utils/issues/571,1646682686,IC_kwDOCGYnMM5iJmI-,9599,simonw,2023-07-22T22:23:47Z,2023-07-22T22:23:47Z,OWNER,"Demo: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].insert({""name"": ""Cleo""})
>>> print(db[""foo""].schema) CREATE TABLE [foo] ( [name] TEXT ) >>> db[""foo""].transform(not_null={""name""})
>>> print(db[""foo""].schema) CREATE TABLE ""foo"" ( [name] TEXT NOT NULL ) >>> db[""foo""].transform(types={""name"": int}, keep_table=""kept"")
>>> print(db.schema) CREATE TABLE ""kept"" ( [name] TEXT NOT NULL ); CREATE TABLE ""foo"" ( [name] INTEGER NOT NULL ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816876211,`.transform(keep_table=...)` option, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527091278,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527091278,MDEyOklzc3VlQ29tbWVudDUyNzA5MTI3OA==,9599,simonw,2019-09-02T10:13:32Z,2019-09-02T10:13:32Z,OWNER,"This is brilliant! One tiny suggestion: I like `--create-triggers` and `create_triggers=True` better for this as they are shorter but still capture what it dies - especially since the underlying SQL uses `CREATE TRIGGER` statements.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958,Add triggers while enabling FTS, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527258212,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527258212,MDEyOklzc3VlQ29tbWVudDUyNzI1ODIxMg==,9599,simonw,2019-09-02T23:40:56Z,2019-09-02T23:40:56Z,OWNER,"This is fantastic, thanks so much. I spotted a Datasette bug triggered by this: Datasette has its own version of the `detect_fts` function - at https://github.com/simonw/datasette/blob/d224ee2c98ac39c2c6e21a0ac0c62e5c3e1ccd11/datasette/utils/__init__.py#L466-L479 - which fails to pick up FTS tables created using the new escaping pattern. It's a bug in Datasette, not sqlite-utils - so I'll open an issue there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958,Add triggers while enabling FTS, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527266798,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527266798,MDEyOklzc3VlQ29tbWVudDUyNzI2Njc5OA==,9599,simonw,2019-09-03T01:03:59Z,2019-09-03T01:03:59Z,OWNER,Released in 1.11 - thanks again! https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-11,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958,Add triggers while enabling FTS, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646643676,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646643676,IC_kwDOCGYnMM5iJcnc,9599,simonw,2023-07-22T18:18:24Z,2023-07-22T18:18:24Z,OWNER,"Here's where I added that to LLM: https://github.com/simonw/llm/commit/a396950f7934e82a9968703bb3ce9ab7ab62f7f8 - https://github.com/simonw/llm/issues/49","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646645990,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646645990,IC_kwDOCGYnMM5iJdLm,9599,simonw,2023-07-22T18:29:40Z,2023-07-22T18:30:16Z,OWNER,"Example plugin: https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf ```bash sqlite-utils install https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf/archive/cd7960e476c441a3c5f619e2a44a641a39b91467.zip ``` `pyproject.toml`: ```toml [project] name = ""sqlite-utils-hello-world"" version = ""0.1"" [project.entry-points.sqlite_utils] hello_world = ""sqlite_utils_hello_world"" ``` `sqlite_utils_hello_world.py`: ```python import click import sqlite_utils @sqlite_utils.hookimpl def register_commands(cli): @cli.command() def hello_world(): ""Say hello world"" click.echo(""Hello world!"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646654275,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646654275,IC_kwDOCGYnMM5iJfND,9599,simonw,2023-07-22T19:19:35Z,2023-07-22T19:19:35Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#register-commands-cli,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642666,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646642666,IC_kwDOCGYnMM5iJcXq,9599,simonw,2023-07-22T18:13:19Z,2023-07-22T18:13:19Z,OWNER,"https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table ```bash sqlite-utils create-table ... --replace ``` That also has `--ignore`: ``` --ignore If table already exists, do nothing --replace If table already exists, replace it --transform If table already exists, try to transform the schema ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642959,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646642959,IC_kwDOCGYnMM5iJccP,9599,simonw,2023-07-22T18:14:49Z,2023-07-22T18:14:49Z,OWNER,Here's where those are implemented for the `create-table` CLI command: https://github.com/simonw/sqlite-utils/blob/f7af23837deab5c98dae9441d1f68318065d7d8c/sqlite_utils/cli.py#L1543-L1564,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646652105,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646652105,IC_kwDOCGYnMM5iJerJ,9599,simonw,2023-07-22T19:05:13Z,2023-07-22T19:05:13Z,OWNER,I think this is `replace=True` and `ignore=True` to match the CLI. And refactoring the CLI to use them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653382,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646653382,IC_kwDOCGYnMM5iJe_G,9599,simonw,2023-07-22T19:13:20Z,2023-07-22T19:13:20Z,OWNER,"Demo: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].create({""id"": int})
>>> db[""foo""].create({""id"": int}) Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1647, in create self.db.create_table( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1030, in create_table self.execute(sql) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 510, in execute return self.conn.execute(sql) ^^^^^^^^^^^^^^^^^^^^^^ sqlean.dbapi2.OperationalError: table [foo] already exists >>> db[""foo""].create({""id"": int}, ignore=True)
>>> db[""foo""].create({""id"": int, ""name"": str}, replace=True)
>>> db[""foo""].create({""id"": int, ""name"": str, ""age"": int}, transform=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653610,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646653610,IC_kwDOCGYnMM5iJfCq,9599,simonw,2023-07-22T19:14:56Z,2023-07-22T19:14:56Z,OWNER,"Manual testing of CLI command as well: ``` $ sqlite-utils create-table /tmp/f.db foo id integer $ sqlite-utils create-table /tmp/f.db foo id integer Error: Table ""foo"" already exists. Use --replace to delete and replace it. $ sqlite-utils create-table /tmp/f.db foo id integer --replace $ sqlite-utils create-table /tmp/f.db foo id $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER ); $ sqlite-utils create-table /tmp/f.db foo id integer name str --transform Error: column types must be one of ('INTEGER', 'TEXT', 'FLOAT', 'BLOB') $ sqlite-utils create-table /tmp/f.db foo id integer name text --transform $ sqlite-utils schema /tmp/f.db CREATE TABLE ""foo"" ( [id] INTEGER, [name] TEXT ); $ sqlite-utils create-table /tmp/f.db foo id integer name text --ignore $ sqlite-utils create-table /tmp/f.db foo id integer name text --replace $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646654818,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646654818,IC_kwDOCGYnMM5iJfVi,9599,simonw,2023-07-22T19:22:40Z,2023-07-22T19:22:40Z,OWNER,"I think this broke a test: ``` $ pytest tests/test_tracer.py =============================================== test session starts ================================================ platform darwin -- Python 3.11.4, pytest-7.2.2, pluggy-1.0.0 rootdir: /Users/simon/Dropbox/Development/sqlite-utils plugins: icdiff-0.6, hypothesis-6.68.2 collected 2 items tests/test_tracer.py F. [100%] ===================================================== FAILURES ===================================================== ___________________________________________________ test_tracer ____________________________________________________ def test_tracer(): collected = [] db = Database( memory=True, tracer=lambda sql, params: collected.append((sql, params)) ) db[""dogs""].insert({""name"": ""Cleopaws""}) db[""dogs""].enable_fts([""name""]) db[""dogs""].search(""Cleopaws"") > assert collected == [ (""PRAGMA recursive_triggers=on;"", None), (""select name from sqlite_master where type = 'view'"", None), (""select name from sqlite_master where type = 'table'"", None), (""select name from sqlite_master where type = 'view'"", None), (""CREATE TABLE [dogs] (\n [name] TEXT\n);\n "", None), (""select name from sqlite_master where type = 'view'"", None), (""INSERT INTO [dogs] ([name]) VALUES (?);"", [""Cleopaws""]), (""select name from sqlite_master where type = 'view'"", None), ( ""CREATE VIRTUAL TABLE [dogs_fts] USING FTS5 (\n [name],\n content=[dogs]\n)"", None, ), ( ""INSERT INTO [dogs_fts] (rowid, [name])\n SELECT rowid, [name] FROM [dogs];"", None, ), (""select name from sqlite_master where type = 'view'"", None), ] E assert equals failed E [ [ E ('PRAGMA recursive_triggers=on;', None), ('PRAGMA recursive_triggers=on;', None), E ( E ""select name from sqlite_master where type = E 'view'"", E None, ... E E ...Full output truncated (13 lines hidden), use '-vv' to show tests/test_tracer.py:12: AssertionError ============================================= short test summary info ============================================== FAILED tests/test_tracer.py::test_tracer - assert equals failed =========================================== 1 failed, 1 passed in 0.05s ============================================ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646655272,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646655272,IC_kwDOCGYnMM5iJfco,9599,simonw,2023-07-22T19:25:35Z,2023-07-22T19:25:35Z,OWNER,"Here's why that test broke: https://github.com/simonw/sqlite-utils/blob/58b577279fcd5ef6ce88f88b28668dffebfe7f44/sqlite_utils/db.py#L960-L964 I added an extra `if self[name].exists()` check to the `db.create_table()` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638926655,IC_kwDOCGYnMM5hsAk_,9599,simonw,2023-07-17T21:42:37Z,2023-07-17T21:42:37Z,OWNER,"I really like this. I'm also interested in: - Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette - Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes The upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643379,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646643379,IC_kwDOCGYnMM5iJciz,9599,simonw,2023-07-22T18:16:54Z,2023-07-22T18:16:54Z,OWNER,"> Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins? Yeah, it could do! That's not a terrible idea to be honest, those things have really been proliferating. Alternatively, they could each register themselves as plugins in addition - so if you install e.g. `pocket-to-sqlite` you could then optionally also run it as `sqlite-utils pocket-to-sqlite ...` The benefit there is for people who install `sqlite-utils` from Homebrew, where it gets its own virtual environment. They could run: ```bash brew install sqlite-utils sqlite-utils install pocket-to-sqlite sqlite-utils pocket-to-sqlite ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646643450,IC_kwDOCGYnMM5iJcj6,9599,simonw,2023-07-22T18:17:18Z,2023-07-22T18:17:18Z,OWNER,I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646654383,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646654383,IC_kwDOCGYnMM5iJfOv,9599,simonw,2023-07-22T19:20:16Z,2023-07-22T19:20:16Z,OWNER,"Here's documentation for the new plugins mechanism, including a very short tutorial on writing a new plugin (inspired by https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html): https://sqlite-utils.datasette.io/en/latest/plugins.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646686424,IC_kwDOCGYnMM5iJnDY,9599,simonw,2023-07-22T22:52:34Z,2023-07-22T22:52:34Z,OWNER,Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646687219,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646687219,IC_kwDOCGYnMM5iJnPz,9599,simonw,2023-07-22T22:59:36Z,2023-07-22T22:59:36Z,OWNER,Now that we have two plugin hooks I'm closing this issue (we can open other issues for further hooks).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627597872,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627597872,IC_kwDOCGYnMM5hAyww,9599,simonw,2023-07-09T04:09:56Z,2023-07-09T04:09:56Z,OWNER,"Thanks, looks like a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627598570,IC_kwDOCGYnMM5hAy7q,9599,simonw,2023-07-09T04:13:34Z,2023-07-09T04:13:34Z,OWNER,On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1618380888,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1618380888,IC_kwDOCGYnMM5gdohY,9599,simonw,2023-07-03T14:09:11Z,2023-07-03T14:09:31Z,OWNER,"For the CLI: ```bash sqlite-utils rename-table data.db old_table_name new_table_name ``` For the Python code, should it go on Table or on Database? ```python db[""foo""].rename_table(""bar"") db.rename_table(""foo"", ""bar"") ``` I think I like the second better, it's slightly more clear. Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646656246,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646656246,IC_kwDOCGYnMM5iJfr2,9599,simonw,2023-07-22T19:32:07Z,2023-07-22T19:32:07Z,OWNER,"Trying out a simple first implementation: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None >>> db[""foo""].insert({""id"": 1}) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [foo] ( [id] INTEGER ); None select name from sqlite_master where type = 'view' None INSERT INTO [foo] ([id]) VALUES (?); [1] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([foo]) None
>>> db.rename_table(""foo"", ""baz"") ALTER TABLE [foo] RENAME TO [baz] None >>> print(db.schema) select sql from sqlite_master where sql is not null None CREATE TABLE ""baz"" ( [id] INTEGER ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646657324,IC_kwDOCGYnMM5iJf8s,9599,simonw,2023-07-22T19:39:06Z,2023-07-22T19:39:06Z,OWNER,"> Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one. I think `keep_table=""name_of_table""` is good for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657849,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646657849,IC_kwDOCGYnMM5iJgE5,9599,simonw,2023-07-22T19:42:02Z,2023-07-22T19:42:02Z,OWNER,"Manually testing new `rename-table` command: ``` $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT ); $ sqlite-utils rename-table /tmp/f.db bad-table hi Error: Table ""bad-table"" could not be renamed. no such table: bad-table $ sqlite-utils rename-table /tmp/f.db foo foo Error: Table ""foo"" could not be renamed. there is already another table or index with this name: foo $ sqlite-utils rename-table /tmp/f.db foo bar $ sqlite-utils schema /tmp/f.db CREATE TABLE ""bar"" ( [id] INTEGER, [name] TEXT ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646658978,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646658978,IC_kwDOCGYnMM5iJgWi,9599,simonw,2023-07-22T19:48:54Z,2023-07-22T19:48:54Z,OWNER,"Python method documentation: - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.rename_table - https://sqlite-utils.datasette.io/en/latest/python-api.html#renaming-a-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646659809,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646659809,IC_kwDOCGYnMM5iJgjh,9599,simonw,2023-07-22T19:53:56Z,2023-07-22T19:53:56Z,OWNER,"CLI documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#renaming-a-table - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#rename-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/563#issuecomment-1617395444,https://api.github.com/repos/simonw/sqlite-utils/issues/563,1617395444,IC_kwDOCGYnMM5gZ370,9599,simonw,2023-07-03T05:44:43Z,2023-07-03T05:44:43Z,OWNER,Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1785360409,`--empty-null` option when importing CSV, https://github.com/simonw/sqlite-utils/issues/562#issuecomment-1616782404,https://api.github.com/repos/simonw/sqlite-utils/issues/562,1616782404,IC_kwDOCGYnMM5gXiRE,9599,simonw,2023-07-02T19:24:14Z,2023-07-02T19:26:39Z,OWNER,"[Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now. The thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns. Example for https://datasette.io/content/users ```sql CREATE TABLE [users] ( [login] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [avatar_url] TEXT, [gravatar_id] TEXT, [html_url] TEXT, [type] TEXT, [site_admin] INTEGER, [name] TEXT ); ``` And the dataclass: ```python from dataclasses import dataclass @dataclass class User: id: int login: str node_id: str avatar_url: str gravatar_id: str html_url: str type: str site_admin: int name: str ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1784794489,Explore the intersection between sqlite-utils and dataclasses, https://github.com/simonw/sqlite-utils/issues/561#issuecomment-1610040517,https://api.github.com/repos/simonw/sqlite-utils/issues/561,1610040517,IC_kwDOCGYnMM5f90TF,9599,simonw,2023-06-27T18:44:31Z,2023-06-27T18:44:38Z,OWNER,"Got this working: ```bash sqlite-utils insert /tmp/playground.db Playground_Submission_Data \ ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000 ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1777548699,`--stop-after` option for `insert` and `upsert` commands, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270055,IC_kwDOCGYnMM5fvbxn,9599,simonw,2023-06-25T21:31:56Z,2023-06-25T21:31:56Z,OWNER,"Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953 A bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`: ``` @pytest.fixture def db_to_analyze_path(db_to_analyze, tmpdir): path = str(tmpdir / ""test.db"") db = sqlite3.connect(path) > db.executescript(""\n"".join(db_to_analyze.conn.iterdump())) E AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump' ``` Also some of these: ``` def test_analyze_whole_database(db): assert set(db.table_names()) == {""one_index"", ""two_indexes""} db.analyze() > assert set(db.table_names()) == {""one_index"", ""two_indexes"", ""sqlite_stat1""} E AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'} E Extra items in the left set: E 'sqlite_stat4' E Full diff: E - {'two_indexes', 'sqlite_stat1', 'one_index'} E + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'} E ? ++++++++++++++++ ``` Apparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting. Plus some errors that look like this: ``` def test_enable_wal(): runner = CliRunner() dbs = [""test.db"", ""test2.db""] with runner.isolated_filesystem(): for dbname in dbs: db = Database(dbname) db[""t""].create({""pk"": int}, pk=""pk"") assert db.journal_mode == ""delete"" result = runner.invoke(cli.cli, [""enable-wal""] + dbs) > assert 0 == result.exit_code E AssertionError: assert 0 == 1 E + where 1 = .exit_code ``` Test summary: ``` ============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s ============= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270887,IC_kwDOCGYnMM5fvb-n,9599,simonw,2023-06-25T21:37:12Z,2023-06-26T08:21:00Z,OWNER,"On my own laptop I got a crash running the tests - details here: - https://github.com/nalgeon/sqlean.py/issues/3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606273005,IC_kwDOCGYnMM5fvcft,9599,simonw,2023-06-25T21:47:47Z,2023-06-25T21:47:47Z,OWNER,I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606290917,IC_kwDOCGYnMM5fvg3l,9599,simonw,2023-06-25T22:32:28Z,2023-06-25T22:32:28Z,OWNER,"I've fixed most of the test failures, but I still need to fix this one: > cannot change into wal mode from within a transaction","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606293382,IC_kwDOCGYnMM5fvheG,9599,simonw,2023-06-25T22:34:47Z,2023-06-25T22:34:47Z,OWNER,"```pycon >>> import sqlite3 >>> db = sqlite3.connect(""/tmp/1.db"") >>> db.execute('PRAGMA journal_mode=wal;') >>> import sqlean >>> db2 = sqlean.connect(""/tmp/2.db"") >>> db2.execute('PRAGMA journal_mode=wal;') Traceback (most recent call last): File """", line 1, in sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606294627,IC_kwDOCGYnMM5fvhxj,9599,simonw,2023-06-25T22:40:10Z,2023-06-25T22:40:10Z,OWNER,I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606297356,IC_kwDOCGYnMM5fvicM,9599,simonw,2023-06-25T22:42:41Z,2023-06-25T22:42:41Z,OWNER,"Yes that does seem to do the trick: ```pycon >>> import sqlean >>> db = sqlean.connect(""/tmp/4.db"") >>> db.execute('PRAGMA journal_mode;').fetchall() [('delete',)] >>> db.isolation_level '' >>> db.execute('PRAGMA journal_mode=wal;') Traceback (most recent call last): File """", line 1, in sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction >>> db.isolation_level = None >>> db.isolation_level >>> db.execute('PRAGMA journal_mode=wal;') ``` Weird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606310630,IC_kwDOCGYnMM5fvlrm,9599,simonw,2023-06-25T23:06:07Z,2023-06-25T23:06:07Z,OWNER,"Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here: - https://github.com/coleifer/pysqlite3/issues/58","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606315321,IC_kwDOCGYnMM5fvm05,9599,simonw,2023-06-25T23:18:33Z,2023-06-25T23:18:33Z,OWNER,Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527090411,https://api.github.com/repos/simonw/sqlite-utils/issues/56,527090411,MDEyOklzc3VlQ29tbWVudDUyNzA5MDQxMQ==,9599,simonw,2019-09-02T10:10:48Z,2019-09-02T10:10:48Z,OWNER,"Good spot, thanks! Would be useful to add a test as well, derived from your above example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487847945,Escape the table name in populate_fts and search., https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556288300,https://api.github.com/repos/simonw/sqlite-utils/issues/553,1556288300,IC_kwDOCGYnMM5cwxMs,9599,simonw,2023-05-21T20:48:01Z,2023-05-21T20:48:01Z,OWNER,If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718635018,Reformatted CLI examples in docs, https://github.com/simonw/sqlite-utils/issues/552#issuecomment-1556292204,https://api.github.com/repos/simonw/sqlite-utils/issues/552,1556292204,IC_kwDOCGYnMM5cwyJs,9599,simonw,2023-05-21T21:05:15Z,2023-05-21T21:05:15Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/installation.html#setting-up-shell-completion,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718612569,Document how to setup shell auto-completion, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556262574,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556262574,IC_kwDOCGYnMM5cwq6u,9599,simonw,2023-05-21T19:04:59Z,2023-05-21T19:04:59Z,OWNER,"I wrote the docs like this because early examples include both the command and its output: https://sqlite-utils.datasette.io/en/stable/cli.html#returning-json ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556263182,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556263182,IC_kwDOCGYnMM5cwrEO,9599,simonw,2023-05-21T19:06:48Z,2023-05-21T19:06:48Z,OWNER,"I could split them up into two blocks like this: I do miss the visual indication that one of these is the command and one is the output though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556265772,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556265772,IC_kwDOCGYnMM5cwrss,9599,simonw,2023-05-21T19:16:15Z,2023-05-21T19:16:15Z,OWNER,"Another option: That's using this markup: ``` Newline-delimited JSON ~~~~~~~~~~~~~~~~~~~~~~ Use ``--nl`` to get back newline-delimited JSON objects: .. code-block:: bash sqlite-utils dogs.db ""select * from dogs"" --nl .. code-block:: output {""id"": 1, ""age"": 4, ""name"": ""Cleo""} {""id"": 2, ""age"": 2, ""name"": ""Pancakes""} ``` And this extra CSS: ```css .highlight-output .highlight { border-left: 9px solid #30c94f; } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556287599,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556287599,IC_kwDOCGYnMM5cwxBv,9599,simonw,2023-05-21T20:44:55Z,2023-05-21T20:44:55Z,OWNER,"Put this in a PR so I can preview it: - #553 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556288270,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556288270,IC_kwDOCGYnMM5cwxMO,9599,simonw,2023-05-21T20:47:51Z,2023-05-21T20:47:51Z,OWNER,This page has all of the changes: https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556291915,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556291915,IC_kwDOCGYnMM5cwyFL,9599,simonw,2023-05-21T21:04:03Z,2023-05-21T21:04:03Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/cli.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556249984,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556249984,IC_kwDOCGYnMM5cwn2A,9599,simonw,2023-05-21T18:24:48Z,2023-05-21T18:24:48Z,OWNER,"This is blocking: - #549","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556250236,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556250236,IC_kwDOCGYnMM5cwn58,9599,simonw,2023-05-21T18:25:26Z,2023-05-21T18:25:26Z,OWNER,"Relevant issues: - https://github.com/python/importlib_metadata/issues/406 - https://github.com/PyCQA/flake8/issues/1701 It looks to me like this is only a problem for `flake8` on Python 3.7 - 3.8 and higher work OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556255309,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556255309,IC_kwDOCGYnMM5cwpJN,9599,simonw,2023-05-21T18:42:25Z,2023-05-21T18:42:25Z,OWNER,Tests passed here: https://github.com/simonw/sqlite-utils/actions/runs/5039119716,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/pull/55#issuecomment-522238063,https://api.github.com/repos/simonw/sqlite-utils/issues/55,522238063,MDEyOklzc3VlQ29tbWVudDUyMjIzODA2Mw==,9599,simonw,2019-08-17T13:42:40Z,2019-08-17T13:42:40Z,OWNER,"I still need to implement the part of this where certain actions against views (like updating/inserting) are not allowed. I think the better way to do that is to introduce a new class - `View` - rather than reusing the existing `Table` class - since there's a lot of stuff on Table (like primary key / foreign key introspection) which simply doesn't make sense for views. This will get a bit more complicated - maybe I need both Table and View to extend a common subclass here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",481887482,Ability to introspect and run queries against views, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241812,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556241812,IC_kwDOCGYnMM5cwl2U,9599,simonw,2023-05-21T17:58:25Z,2023-05-21T17:58:25Z,OWNER,Documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556242262,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556242262,IC_kwDOCGYnMM5cwl9W,9599,simonw,2023-05-21T18:00:05Z,2023-05-21T18:00:05Z,OWNER,"Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465 ``` sqlite_utils/cli.py:37: error: Skipping analyzing ""trogon"": module is installed, but missing library stubs or py.typed marker [import] sqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 52 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/issues/548#issuecomment-1556231832,https://api.github.com/repos/simonw/sqlite-utils/issues/548,1556231832,IC_kwDOCGYnMM5cwjaY,9599,simonw,2023-05-21T17:24:13Z,2023-05-21T17:24:13Z,OWNER,"Oh, I see why that is now: https://github.com/simonw/sqlite-utils/blob/6027f3ea6939a399aeef2578fca17efec0e539df/sqlite_utils/cli.py#L2670-L2679 This is because of the following command: sqlite-utils analyze-tables table1 table2 --column x Since you can pass multiple tables AND multiple columns, the tool currently assumes that the column(s) you specify may be available on a subset of the provided tables. I'm going to change this so if the column is not on ANY of those tables you get an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718576761,analyze-tables should validate provide --column names, https://github.com/simonw/sqlite-utils/issues/547#issuecomment-1556228395,https://api.github.com/repos/simonw/sqlite-utils/issues/547,1556228395,IC_kwDOCGYnMM5cwikr,9599,simonw,2023-05-21T17:11:15Z,2023-05-21T17:11:15Z,OWNER,"This will be a cosmetic change to the CLI output only - the options to save data to the database and the Python API function will continue to return `[(None, 158)]`. I can add an optimization though to avoid running the SQL count query if we know that it's all `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718572201,No need to show common values if everything is null, https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213396,https://api.github.com/repos/simonw/sqlite-utils/issues/546,1556213396,IC_kwDOCGYnMM5cwe6U,9599,simonw,2023-05-21T15:58:12Z,2023-05-21T16:18:46Z,OWNER,"Documentation preview: - https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables - https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables - https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column - https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718550688,"Analyze tables options: --common-limit, --no-most, --no-least", https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556189823,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556189823,IC_kwDOCGYnMM5cwZJ_,9599,simonw,2023-05-21T14:09:59Z,2023-05-21T14:09:59Z,OWNER,"I don't want to add `trogon` as a default dependency because it's a little heavy - it pulls in all of Rich and Textual as well. People who use `sqlite-utils` just for its Python API won't benefit from this - it's a CLI feature only. But I have a `sqlite-utils install ...` command for helping people to install packages into the same virtual environment as `sqlite-utils` no matter how they installed that tool: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install So I can treat Trogon as an optional dependency and add the `sqlite-utils tui` command only if that package is also installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556190531,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556190531,IC_kwDOCGYnMM5cwZVD,9599,simonw,2023-05-21T14:13:43Z,2023-05-21T14:13:43Z,OWNER,"OK, this works! ![trogon](https://github.com/simonw/sqlite-utils/assets/9599/2ae194c5-ec82-471a-9d1b-b01b3f2632f3) To try it out, install that branch from GitHub: pip install https://github.com/simonw/sqlite-utils/archive/refs/heads/trogon.zip Then run this: sqlite-utils install trogon And this: sqlite-utils tui ","{""total_count"": 5, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 3, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556191894,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556191894,IC_kwDOCGYnMM5cwZqW,9599,simonw,2023-05-21T14:20:14Z,2023-05-21T14:20:14Z,OWNER,"Opened a feature request for customizing the help and command name: - https://github.com/Textualize/trogon/issues/2","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556210844,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556210844,IC_kwDOCGYnMM5cweSc,9599,simonw,2023-05-21T15:44:10Z,2023-05-21T15:44:10Z,OWNER,"It looks like `nargs=-1` on a positional argument isn't yet supported - opened an issue here: - https://github.com/Textualize/trogon/issues/4","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556247818,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556247818,IC_kwDOCGYnMM5cwnUK,9599,simonw,2023-05-21T18:17:46Z,2023-05-21T18:17:46Z,OWNER,Draft documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556269616,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556269616,IC_kwDOCGYnMM5cwsow,9599,simonw,2023-05-21T19:33:13Z,2023-05-21T19:33:13Z,OWNER,Now released: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556211643,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556211643,IC_kwDOCGYnMM5cwee7,9599,simonw,2023-05-21T15:48:17Z,2023-05-21T15:48:17Z,OWNER,I generated the commit message in https://github.com/simonw/sqlite-utils/commit/1c1991b447a1ddd3d61d9d4a8a1d6a9da47ced20 using `git diff | llm --system 'describe this change'`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least, https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556225788,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556225788,IC_kwDOCGYnMM5cwh78,9599,simonw,2023-05-21T17:02:05Z,2023-05-21T17:02:05Z,OWNER,"New docs: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-analyze-tables - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#analyze-tables - https://sqlite-utils.datasette.io/en/latest/python-api.html#analyzing-a-column - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.analyze_column New help output: ``` % sqlite-utils analyze-tables --help Usage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]... Analyze the columns in one or more tables Example: sqlite-utils analyze-tables data.db trees Options: -c, --column TEXT Specific columns to analyze --save Save results to _analyze_tables table --common-limit INTEGER How many common values --no-most Skip most common values --no-least Skip least common values --load-extension TEXT Path to SQLite extension, with optional :entrypoint -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least, https://github.com/simonw/sqlite-utils/issues/542#issuecomment-1539052467,https://api.github.com/repos/simonw/sqlite-utils/issues/542,1539052467,IC_kwDOCGYnMM5bvBOz,9599,simonw,2023-05-08T21:07:41Z,2023-05-08T21:07:41Z,OWNER,"Relevant commits (will mostly revert these): - https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd - https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700936245,Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0, https://github.com/simonw/sqlite-utils/issues/541#issuecomment-1538963959,https://api.github.com/repos/simonw/sqlite-utils/issues/541,1538963959,IC_kwDOCGYnMM5burn3,9599,simonw,2023-05-08T19:59:34Z,2023-05-08T19:59:34Z,OWNER,"There are 8 failing tests left: ``` ==== short test summary info ==== FAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bcaca0> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bc9620> ERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> ==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ==== ``` Full traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700840265,Get tests to pass with `pytest -Werror`, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513653,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513653,IC_kwDOCGYnMM5bpJi1,9599,simonw,2023-05-07T18:37:59Z,2023-05-07T18:38:19Z,OWNER,"Useful comment here: - https://github.com/urllib3/urllib3/issues/2168#issuecomment-1537360928 > I faced the same issue. I switched to a different Python kernel (3.11.2) and it worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513912,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513912,IC_kwDOCGYnMM5bpJm4,9599,simonw,2023-05-07T18:39:29Z,2023-05-07T18:39:29Z,OWNER,"https://readthedocs.org/projects/sqlite-utils/builds/20513034/ said: > Problem in your project's configuration. Invalid ""python.version"": expected one of (2, 2.7, 3, 3.5, 3.6, 3.7, 3.8, pypy3.5), got 3.11","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537514069,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537514069,IC_kwDOCGYnMM5bpJpV,9599,simonw,2023-05-07T18:40:18Z,2023-05-07T18:40:18Z,OWNER,"https://docs.readthedocs.io/en/stable/config-file/v2.html suggests: ```yaml build: os: ubuntu-22.04 tools: python: ""3.11"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/54#issuecomment-522236053,https://api.github.com/repos/simonw/sqlite-utils/issues/54,522236053,MDEyOklzc3VlQ29tbWVudDUyMjIzNjA1Mw==,9599,simonw,2019-08-17T13:22:26Z,2019-08-17T13:22:26Z,OWNER,"Good catch: `create_view()` is an earlier utility method from before I started adding methods like `db[""table""].rows` and `.rows_where()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524294383,https://api.github.com/repos/simonw/sqlite-utils/issues/54,524294383,MDEyOklzc3VlQ29tbWVudDUyNDI5NDM4Mw==,9599,simonw,2019-08-23T12:20:15Z,2019-08-23T12:20:15Z,OWNER,Closed by #55,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524294928,https://api.github.com/repos/simonw/sqlite-utils/issues/54,524294928,MDEyOklzc3VlQ29tbWVudDUyNDI5NDkyOA==,9599,simonw,2019-08-23T12:22:11Z,2019-08-23T12:22:11Z,OWNER,Documentation is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#listing-views,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/54#issuecomment-524297256,https://api.github.com/repos/simonw/sqlite-utils/issues/54,524297256,MDEyOklzc3VlQ29tbWVudDUyNDI5NzI1Ng==,9599,simonw,2019-08-23T12:30:20Z,2019-08-23T12:30:20Z,OWNER,Released in 1.10: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v1-10,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",480961330,"Ability to list views, and to access db[""view_name""].rows / rows_where / etc", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507394,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507394,IC_kwDOCGYnMM5bpIBC,9599,simonw,2023-05-07T18:08:44Z,2023-05-07T18:08:44Z,OWNER,"Prototype: ```diff diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 153e5f9..c830518 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -124,6 +124,7 @@ See :ref:`cli_query`. --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row + --raw-lines Raw output, first column of each row -p, --param ... Named :parameters for SQL query --functions TEXT Python code defining one or more custom SQL functions @@ -192,6 +193,7 @@ See :ref:`cli_memory`. --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row + --raw-lines Raw output, first column of each row -p, --param ... Named :parameters for SQL query --encoding TEXT Character encoding for CSV input, defaults to utf-8 diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index d25b1df..da0e4b6 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1653,6 +1653,7 @@ def drop_view(path, view, ignore, load_extension): ) @output_options @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"") +@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"") @click.option( ""-p"", ""--param"", @@ -1677,6 +1678,7 @@ def query( fmt, json_cols, raw, + raw_lines, param, load_extension, functions, @@ -1700,7 +1702,19 @@ def query( _register_functions(db, functions) _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ) @@ -1728,6 +1742,7 @@ def query( ) @output_options @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"") +@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"") @click.option( ""-p"", ""--param"", @@ -1773,6 +1788,7 @@ def memory( fmt, json_cols, raw, + raw_lines, param, encoding, no_detect_types, @@ -1879,12 +1895,36 @@ def memory( _register_functions(db, functions) _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ) def _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ): with db.conn: try: @@ -1903,6 +1943,13 @@ def _execute_query( sys.stdout.buffer.write(data) else: sys.stdout.write(str(data)) + elif raw_lines: + for row in cursor: + data = row[0] + if isinstance(data, bytes): + sys.stdout.buffer.write(data + b""\n"") + else: + sys.stdout.write(str(data) + ""\n"") elif fmt or table: print( tabulate.tabulate( ``` Needs tests and more documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507525,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507525,IC_kwDOCGYnMM5bpIDF,9599,simonw,2023-05-07T18:09:09Z,2023-05-07T18:09:09Z,OWNER,"I'm tempted to upgrade `--raw` to do this instead, but that would be a breaking change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507676,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507676,IC_kwDOCGYnMM5bpIFc,9599,simonw,2023-05-07T18:09:43Z,2023-05-07T18:09:54Z,OWNER,"This worked: ```bash sqlite-utils memory /tmp/books3.json:nl \ 'select name from books3' --raw-lines > titles.txt ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537514610,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537514610,IC_kwDOCGYnMM5bpJxy,9599,simonw,2023-05-07T18:43:24Z,2023-05-07T18:43:24Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#returning-raw-data-such-as-binary-content - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#memory","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538793817,IC_kwDOCGYnMM5buCFZ,9599,simonw,2023-05-08T17:55:10Z,2023-05-08T17:55:10Z,OWNER,"Confirmed - I added this test and it fails: ```python def test_upsert_all_not_null(fresh_db): # https://github.com/simonw/sqlite-utils/issues/538 fresh_db[""comments""].upsert_all( [{""id"": 1, ""name"": ""Cleo""}], pk=""id"", not_null=[""name""], ) assert list(fresh_db[""comments""].rows) == [{""id"": 1, ""name"": ""Cleo""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538801855,IC_kwDOCGYnMM5buEC_,9599,simonw,2023-05-08T18:00:17Z,2023-05-08T18:00:17Z,OWNER,"From time in the debugger, after creating the table it ends up doing this: ``` (Pdb) queries_and_params [ ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538887361,IC_kwDOCGYnMM5buY7B,9599,simonw,2023-05-08T19:01:20Z,2023-05-08T19:01:20Z,OWNER,"Here's the problem: ```python import sqlite3 db = sqlite3.connect("":memory:"") db.execute('create table foo (id integer primary key, name not null)') db.execute('insert into foo (id) values (1)') ``` Produces: ``` IntegrityError: NOT NULL constraint failed: foo.name ``` But this: ```python db.execute('insert or ignore into foo (id) values (1)') ``` Completes without an exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538889482,IC_kwDOCGYnMM5buZcK,9599,simonw,2023-05-08T19:02:38Z,2023-05-08T19:02:38Z,OWNER,"Here's the code at fault: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538893329,IC_kwDOCGYnMM5buaYR,9599,simonw,2023-05-08T19:04:47Z,2023-05-08T19:04:47Z,OWNER,"This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`. One fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html But... > UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04). I still want to support SQLite versions earlier than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538903556,IC_kwDOCGYnMM5buc4E,9599,simonw,2023-05-08T19:11:24Z,2023-05-08T19:13:23Z,OWNER,"I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though. This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538910894,IC_kwDOCGYnMM5buequ,9599,simonw,2023-05-08T19:16:52Z,2023-05-08T19:17:00Z,OWNER,"How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`? Something like this: ```python [ ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538921774,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538921774,IC_kwDOCGYnMM5buhUu,9599,simonw,2023-05-08T19:24:41Z,2023-05-08T19:24:41Z,OWNER,That fix seems to work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539055393,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539055393,IC_kwDOCGYnMM5bvB8h,9599,simonw,2023-05-08T21:10:06Z,2023-05-08T21:10:06Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539157643,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539157643,IC_kwDOCGYnMM5bva6L,9599,simonw,2023-05-08T22:45:09Z,2023-05-08T22:45:21Z,OWNER,"Here's an example from the new tests: https://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/issues/534#issuecomment-1538933540,https://api.github.com/repos/simonw/sqlite-utils/issues/534,1538933540,IC_kwDOCGYnMM5bukMk,9599,simonw,2023-05-08T19:34:37Z,2023-05-08T19:34:37Z,OWNER,"On macOS this shows the same warning: ``` % python -Wdefault $(which sqlite-utils) insert dogs.db dogs dogs.csv --csv [############------------------------] 35% [####################################] 100%/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs.csv' encoding='utf-8-sig'> insert_upsert_implementation( ResourceWarning: Enable tracemalloc to get the object allocation traceback ``` The file itself is a `click.File` which is automatically closed - https://click.palletsprojects.com/en/8.1.x/api/#click.File - but it looks like it's the `_io.TextIOWrapper` which is not being closed: https://github.com/simonw/sqlite-utils/blob/2376c452a56b0c3e75e7ca698273434e32945304/sqlite_utils/cli.py#L949-L956","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1622640374, ResourceWarning: unclosed file, https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465302936,https://api.github.com/repos/simonw/sqlite-utils/issues/533,1465302936,IC_kwDOCGYnMM5XVr-Y,9599,simonw,2023-03-12T21:22:09Z,2023-03-12T21:22:09Z,OWNER,"Could be the same problem as: - https://github.com/simonw/datasette/issues/1972 Which I fixed in https://github.com/simonw/datasette/commit/3af313e165215696af899e772f47bf7c27873ae3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620516340,ReadTheDocs error: not all arguments converted during string formatting, https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465303378,https://api.github.com/repos/simonw/sqlite-utils/issues/533,1465303378,IC_kwDOCGYnMM5XVsFS,9599,simonw,2023-03-12T21:24:04Z,2023-03-12T21:24:04Z,OWNER,"Upgraded to Sphinx 6 locally and got the same error: ``` % just docs Cogging README.md Cogging docs/changelog.rst Cogging docs/cli-reference.rst Cogging docs/cli.rst Cogging docs/contributing.rst Cogging docs/index.rst Cogging docs/installation.rst Cogging docs/python-api.rst Cogging docs/reference.rst sphinx-autobuild -a -b html ""."" ""_build"" --watch ../sqlite_utils [sphinx-autobuild] > sphinx-build -b html -a /Users/simon/Dropbox/Development/sqlite-utils/docs /Users/simon/Dropbox/Development/sqlite-utils/docs/_build Running Sphinx v6.1.3 loading pickled environment... failed failed: Can't get attribute '_stable_repr_object' on building [mo]: all of 0 po files writing output... building [html]: all source files updating environment: [new config] 8 added, 0 changed, 0 removed reading sources... [ 12%] changelog Exception occurred: File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/sphinx/ext/extlinks.py"", line 103, in role title = caption % part ~~~~~~~~^~~~~~ TypeError: not all arguments converted during string formatting The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1ey36c1n.log, if you want to report the issue to the developers. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620516340,ReadTheDocs error: not all arguments converted during string formatting, https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539006509,https://api.github.com/repos/simonw/sqlite-utils/issues/532,1539006509,IC_kwDOCGYnMM5bu2At,9599,simonw,2023-05-08T20:28:56Z,2023-05-08T20:28:56Z,OWNER,Was this a newline-delimited JSON file perhaps?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620254998,Show more information when JSON can't be imported with sqlite-utils insert, https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539009453,https://api.github.com/repos/simonw/sqlite-utils/issues/532,1539009453,IC_kwDOCGYnMM5bu2ut,9599,simonw,2023-05-08T20:30:29Z,2023-05-08T20:30:42Z,OWNER,"Here's an improvement: ``` % sqlite-utils insert /tmp/b.db blah /tmp/blah.txt [####################################] 100% Error: Invalid JSON - use --csv for CSV or --tsv for TSV files JSON error: Expecting value: line 1 column 1 (char 0) ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620254998,Show more information when JSON can't be imported with sqlite-utils insert, https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465302343,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465302343,IC_kwDOCGYnMM5XVr1H,9599,simonw,2023-03-12T21:19:13Z,2023-03-12T21:19:13Z,OWNER,"Aah, I think I see why you wrote it like that. The problem is that `init_spatialite()` does other stuff too: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171 So it needs to be able to load the SpatiaLite extension from the correct place, and THEN run `select InitSpatialMetadata()` to configure the database, if needed. So the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon, https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539015064,https://api.github.com/repos/simonw/sqlite-utils/issues/530,1539015064,IC_kwDOCGYnMM5bu4GY,9599,simonw,2023-05-08T20:35:07Z,2023-05-08T20:35:07Z,OWNER,"Wow, this is a neat feature I didn't know about. Looks like there are a bunch of options: - NO ACTION (default) - RESTRICT: application is prohibited from deleting a parent key when there exists one or more child keys mapped to it - SET NULL: when a parent key is deleted the child key columns of all rows in the child table that mapped to the parent key are set to contain SQL NULL values - SET DEFAULT: set a specific default - CASCADE: propagates the delete or update operation on the parent key to each dependent child key","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1595340692,"add ability to configure ""on delete"" and ""on update"" attributes of foreign keys:", https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539018912,https://api.github.com/repos/simonw/sqlite-utils/issues/530,1539018912,IC_kwDOCGYnMM5bu5Cg,9599,simonw,2023-05-08T20:39:00Z,2023-05-08T20:39:00Z,OWNER,"I think the natural place to add these in the Python library API would be https://sqlite-utils.datasette.io/en/stable/python-api.html#adding-foreign-key-constraints - maybe something like this: ```python db[""books""].add_foreign_key(""author_id"", ""authors"", ""id"", on_delete=RESTRICT) ``` Then for the CLI tool could be added to https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key ``` sqlite-utils add-foreign-key my.db books author_id authors id --on-update SET_NULL ``` I wouldn't support these for the other methods of adding foreign keys - `foreign_keys(...)` for the various `.insert()` etc methods and the `add_foreign_keys(...)` bulk menthod.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1595340692,"add ability to configure ""on delete"" and ""on update"" attributes of foreign keys:", https://github.com/simonw/sqlite-utils/pull/528#issuecomment-1539053230,https://api.github.com/repos/simonw/sqlite-utils/issues/528,1539053230,IC_kwDOCGYnMM5bvBau,9599,simonw,2023-05-08T21:08:23Z,2023-05-08T21:08:23Z,OWNER,"I fixed this in: - #527 Will fully remove this misfeature in: - #542","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578793661,Enable `Table.convert()` on falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1506223848,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1506223848,IC_kwDOCGYnMM5Zxybo,9599,simonw,2023-04-13T02:08:16Z,2023-04-13T02:08:16Z,OWNER,"I agree, this is a design flaw. It's technically a breaking change. As such, I would need to bump to v4 to responsibly release this. I'd rather bundle in a few more breaking changes before shipping that version. One thing we could do here is add an extra argument to `.convert()` - something like this: ```python table.convert(col, lambda x: x+1, skip_false=False) ``` This would trigger the new, improved behaviour without breaking existing code that depends on how it works at the moment. Then in `sqlite-utils` 4 we can change the default of that option. What do you think? (I'm open to suggestions for better names for this parameter too)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539033736,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539033736,IC_kwDOCGYnMM5bu8qI,9599,simonw,2023-05-08T20:52:51Z,2023-05-08T20:52:51Z,OWNER,"OK, I implemented that at the Python API level. I need to decide how it should work for the `sqlite-utils convert` command too: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#convert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539035838,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539035838,IC_kwDOCGYnMM5bu9K-,9599,simonw,2023-05-08T20:55:00Z,2023-05-08T20:55:00Z,OWNER,"I'm going to go with `--no-skip-false` as the CLI option. It's ugly, but this whole thing is ugly. I'm going to make a note to remove this misfeature in `sqlite-utils` 4.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539051724,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539051724,IC_kwDOCGYnMM5bvBDM,9599,simonw,2023-05-08T21:07:04Z,2023-05-08T21:07:04Z,OWNER,"Updated documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#convert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values, https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1539101853,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1539101853,IC_kwDOCGYnMM5bvNSd,9599,simonw,2023-05-08T21:52:44Z,2023-05-08T21:52:44Z,OWNER,I like the `lambda-{uuid}` idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail, https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1539108140,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1539108140,IC_kwDOCGYnMM5bvO0s,9599,simonw,2023-05-08T21:59:41Z,2023-05-08T21:59:41Z,OWNER,That original example passes against `main` now.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail, https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1369333759,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1369333759,IC_kwDOCGYnMM5Rnl__,9599,simonw,2023-01-03T02:23:43Z,2023-01-03T02:23:43Z,OWNER,"The documentation here does at least say the following: https://sqlite-utils.datasette.io/en/3.30/python-api.html#reading-rows-from-a-file > - **fp** (*BinaryIO*) -- a file-like object containing binary data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1539109587,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1539109587,IC_kwDOCGYnMM5bvPLT,9599,simonw,2023-05-08T22:00:46Z,2023-05-08T22:00:46Z,OWNER,"> Hey, isn't this essentially the same issue as #448 ? Yes it is, good catch!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/sqlite-utils/issues/52#issuecomment-550694197,https://api.github.com/repos/simonw/sqlite-utils/issues/52,550694197,MDEyOklzc3VlQ29tbWVudDU1MDY5NDE5Nw==,9599,simonw,2019-11-07T04:11:05Z,2019-11-07T04:11:05Z,OWNER,I just hit this bug while running `healthkit-to-sqlite`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",476413293,Throws error if .insert_all() / .upsert_all() called with empty list, https://github.com/simonw/sqlite-utils/pull/519#issuecomment-1539058795,https://api.github.com/repos/simonw/sqlite-utils/issues/519,1539058795,IC_kwDOCGYnMM5bvCxr,9599,simonw,2023-05-08T21:12:52Z,2023-05-08T21:12:52Z,OWNER,"This is a really neat fix, thank you.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1505568103,Fixes breaking DEFAULT values, https://github.com/simonw/sqlite-utils/issues/517#issuecomment-1344965367,https://api.github.com/repos/simonw/sqlite-utils/issues/517,1344965367,IC_kwDOCGYnMM5QKor3,9599,simonw,2022-12-10T01:26:31Z,2022-12-10T01:26:31Z,OWNER,At some point I should drop it from all of these other projects too: https://cs.github.com/?scopeName=All+repos&scope=&q=user%3Asimonw+%223.6%22+path%3A.github%2Fworkflows%2F*,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1487757143,Drop support for Python 3.6, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339834918,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339834918,IC_kwDOCGYnMM5P3EIm,9599,simonw,2022-12-06T19:00:18Z,2022-12-06T19:00:35Z,OWNER,"Right now the command produces no output at all. Maybe a `--verbose` mode that writes these numbers to standard error (or even standard output since it's an option)? Is there a better name than `--verbose` for this? `--summary` perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599,Feature request: output number of ignored/replaced rows for insert command, https://github.com/simonw/sqlite-utils/pull/515#issuecomment-1539077777,https://api.github.com/repos/simonw/sqlite-utils/issues/515,1539077777,IC_kwDOCGYnMM5bvHaR,9599,simonw,2023-05-08T21:27:10Z,2023-05-08T21:27:10Z,OWNER,I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194930,"upsert new rows with constraints, fixes #514", https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539078429,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539078429,IC_kwDOCGYnMM5bvHkd,9599,simonw,2023-05-08T21:27:40Z,2023-05-08T21:27:40Z,OWNER,"Dupe of: - #538","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539079507,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539079507,IC_kwDOCGYnMM5bvH1T,9599,simonw,2023-05-08T21:28:37Z,2023-05-08T21:28:37Z,OWNER,"> This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted. Huh... on that basis, it's possible my fix in https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304 is incomplete. I only covered the 'not null' case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539094287,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539094287,IC_kwDOCGYnMM5bvLcP,9599,simonw,2023-05-08T21:44:11Z,2023-05-08T21:44:11Z,OWNER,"OK, this fails silently: ```python import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute('''CREATE TABLE employees ( id INTEGER PRIMARY KEY, name TEXT, age INTEGER, salary REAL, CHECK (salary is not null and salary > 0) );''') db[""employees""].upsert({""id"": 1, ""name"": ""Bob""}, pk=""id"") list(db[""employees""].rows) ```` It outputs: ```python [] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539099703,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539099703,IC_kwDOCGYnMM5bvMw3,9599,simonw,2023-05-08T21:50:06Z,2023-05-08T21:50:06Z,OWNER,"Applying the fix from the PR here doesn't fix the above problem either: - https://github.com/simonw/sqlite-utils/pull/515 So it looks like these kinds of `check` constraints currently aren't compatible with how `upsert()` works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539100300,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539100300,IC_kwDOCGYnMM5bvM6M,9599,simonw,2023-05-08T21:50:51Z,2023-05-08T21:50:51Z,OWNER,Seeing as `sqlite-utils` doesn't currently provide mechanisms for adding `check` constraints like this I'm going to leave this - I'm happy with the fix I put in for the `not null` constraints.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316437748,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316437748,IC_kwDOCGYnMM5Odz70,9599,simonw,2022-11-16T06:24:31Z,2022-11-16T06:24:31Z,OWNER,"``` sqlite-utils % pipx run no_implicit_optional . Calculating full-repo metadata... Executing codemod... 11.43s 98% complete, 0.24s estimated for 5 files to go... ``` Then: ``` Finished codemodding 239 files! - Transformed 239 files successfully. - Skipped 0 files. - Failed to codemod 0 files. - 0 warnings were generated. ``` Here's the diff: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..e819d17 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -297,12 +297,12 @@ class Database: def __init__( self, - filename_or_conn: Union[str, pathlib.Path, sqlite3.Connection] = None, + filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None, memory: bool = False, - memory_name: str = None, + memory_name: Optional[str] = None, recreate: bool = False, recursive_triggers: bool = True, - tracer: Callable = None, + tracer: Optional[Callable] = None, use_counts_table: bool = False, ): assert (filename_or_conn is not None and (not memory and not memory_name)) or ( @@ -341,7 +341,7 @@ class Database: self.conn.close() @contextlib.contextmanager - def tracer(self, tracer: Callable = None): + def tracer(self, tracer: Optional[Callable] = None): """""" Context manager to temporarily set a tracer function - all executed SQL queries will be passed to this. @@ -378,7 +378,7 @@ class Database: def register_function( self, - fn: Callable = None, + fn: Optional[Callable] = None, deterministic: bool = False, replace: bool = False, name: Optional[str] = None, @@ -879,7 +879,7 @@ class Database: pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -1129,7 +1129,7 @@ class Database: sql += "" [{}]"".format(name) self.execute(sql) - def init_spatialite(self, path: str = None) -> bool: + def init_spatialite(self, path: Optional[str] = None) -> bool: """""" The ``init_spatialite`` method will load and initialize the SpatiaLite extension. The ``path`` argument should be an absolute path to the compiled extension, which @@ -1182,7 +1182,7 @@ class Queryable: def count_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, ) -> int: """""" @@ -1213,12 +1213,12 @@ class Queryable: def rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, + order_by: Optional[str] = None, select: str = ""*"", - limit: int = None, - offset: int = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[dict, None, None]: """""" Iterate over every row in this table or view that matches the specified where clause. @@ -1251,11 +1251,11 @@ class Queryable: def pks_and_rows_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, - order_by: str = None, - limit: int = None, - offset: int = None, + order_by: Optional[str] = None, + limit: Optional[int] = None, + offset: Optional[int] = None, ) -> Generator[Tuple[Any, Dict], None, None]: """""" Like ``.rows_where()`` but returns ``(pk, row)`` pairs - ``pk`` can be a single value or tuple. @@ -1345,7 +1345,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, batch_size: int = 100, hash_id: Optional[str] = None, @@ -1545,7 +1545,7 @@ class Table(Queryable): pk: Optional[Any] = None, foreign_keys: Optional[ForeignKeysType] = None, column_order: Optional[List[str]] = None, - not_null: Iterable[str] = None, + not_null: Optional[Iterable[str]] = None, defaults: Optional[Dict[str, Any]] = None, hash_id: Optional[str] = None, hash_id_columns: Optional[Iterable[str]] = None, @@ -2464,7 +2464,7 @@ class Table(Queryable): columns: Optional[Iterable[str]] = None, limit: Optional[int] = None, offset: Optional[int] = None, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, quote: bool = False, ) -> Generator[dict, None, None]: @@ -2527,7 +2527,7 @@ class Table(Queryable): def delete_where( self, - where: str = None, + where: Optional[str] = None, where_args: Optional[Union[Iterable, dict]] = None, analyze: bool = False, ) -> ""Table"": ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316447182,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316447182,IC_kwDOCGYnMM5Od2PO,9599,simonw,2022-11-16T06:32:31Z,2022-11-16T06:32:31Z,OWNER,"Test failed again: https://github.com/simonw/sqlite-utils/actions/runs/3476950474/jobs/5812663096 `E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/s/spatialite/libsqlite3-mod-spatialite_4.3.0a-6build1_amd64.deb Unable to connect to azure.archive.ubuntu.com:http:` That looks like an intermittent error. I'll try running it again in the morning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/issues/512#issuecomment-1316530539,https://api.github.com/repos/simonw/sqlite-utils/issues/512,1316530539,IC_kwDOCGYnMM5OeKlr,9599,simonw,2022-11-16T07:49:50Z,2022-11-16T07:49:50Z,OWNER,Tests passed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450952393,mypy failures in CI, https://github.com/simonw/sqlite-utils/pull/51#issuecomment-515756563,https://api.github.com/repos/simonw/sqlite-utils/issues/51,515756563,MDEyOklzc3VlQ29tbWVudDUxNTc1NjU2Mw==,9599,simonw,2019-07-28T11:56:10Z,2019-07-28T11:56:10Z,OWNER,"Interesting. The tests failed presumably because the version of SQLite I am running on Travis CI doesn't have that same 999 limit. I'm going to enforce the 999 limit within the library itself, to discourage people from writing code which will fail if it runs on a host that DOES have that limit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473733752,"Fix for too many SQL variables, closes #50", https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1296358636,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1296358636,IC_kwDOCGYnMM5NRNzs,9599,simonw,2022-10-30T21:52:11Z,2022-10-30T21:52:11Z,OWNER,This could work in a similar way to `db.insert(...).last_rowid`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298877872,IC_kwDOCGYnMM5Na02w,9599,simonw,2022-11-01T17:35:30Z,2022-11-01T17:35:30Z,OWNER,"This may not make sense. First, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object). So I tried this prototype: ```diff diff --git a/docs/python-api.rst b/docs/python-api.rst index 206e5e6..78d3a8d 100644 --- a/docs/python-api.rst +++ b/docs/python-api.rst @@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over # {'name': 'Cleo'} # {'name': 'Pancakes'} +After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query: + +.. code-block:: python + + db = Database(memory=True) + db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) + print(db.rowcount) + # Outputs: 2 + .. _python_api_execute: db.execute(sql, params) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..c19c2dd 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -294,6 +294,8 @@ class Database: _counts_table_name = ""_counts"" use_counts_table = False + # Number of rows inserted, updated or deleted + rowcount: Optional[int] = None def __init__( self, @@ -480,9 +482,11 @@ class Database: if self._tracer: self._tracer(sql, parameters) if parameters is not None: - return self.conn.execute(sql, parameters) + cursor = self.conn.execute(sql, parameters) else: - return self.conn.execute(sql) + cursor = self.conn.execute(sql) + self.rowcount = cursor.rowcount + return cursor def executescript(self, sql: str) -> sqlite3.Cursor: """""" ``` But this happens: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}])
>>> db.rowcount -1 ``` Turning on query tracing demonstrates why: ```pycon >>> db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [dogs] ( [name] TEXT ); None select name from sqlite_master where type = 'view' None INSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes'] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([dogs]) None
>>> ``` The `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298879701,IC_kwDOCGYnMM5Na1TV,9599,simonw,2022-11-01T17:37:13Z,2022-11-01T17:37:13Z,OWNER,"The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`? I don't know that `.rowcount` would ever be useful here, since the ""correct"" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example? So I think if people need `rowcount` they can get it by using a `cursor` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291203911,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291203911,IC_kwDOCGYnMM5M9jVH,9599,simonw,2022-10-25T22:21:02Z,2022-10-25T22:21:02Z,OWNER,"- Now tested against Python 3.11. ([#502](https://github.com/simonw/sqlite-utils/issues/502)) - New `table.search_sql(include_rank=True)` option, which adds a `rank` column to the generated SQL. Thanks, Jacob Chapman. ([#480](https://github.com/simonw/sqlite-utils/pull/480)) - Progress bars now display for newline-delimited JSON files using the `--nl` option. Thanks, Mischa Untaga. ([#485](https://github.com/simonw/sqlite-utils/issues/485)) - New `db.close()` method. ([#504](https://github.com/simonw/sqlite-utils/issues/504)) - Conversion functions passed to [table.convert(...)](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-convert) can now return lists or dictionaries, which will be inserted into the database as JSON strings. ([#495](https://github.com/simonw/sqlite-utils/issues/495)) - `sqlite-utils install` and `sqlite-utils uninstall` commands for installing packages into the same virtual environment as `sqlite-utils`, [described here](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install). ([#483](https://github.com/simonw/sqlite-utils/issues/483)) - New [sqlite_utils.utils.flatten()](https://sqlite-utils.datasette.io/en/stable/reference.html#reference-utils-flatten) utility function. ([#500](https://github.com/simonw/sqlite-utils/issues/500)) - Documentation on [using Just](https://sqlite-utils.datasette.io/en/stable/contributing.html#contributing-just) to run tests, linters and build documentation. - Documentation now covers the [Release process](https://sqlite-utils.datasette.io/en/stable/contributing.html#release-process) for this package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291216193,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291216193,IC_kwDOCGYnMM5M9mVB,9599,simonw,2022-10-25T22:41:16Z,2022-10-25T22:41:16Z,OWNER,Tweeted about it here: https://twitter.com/simonw/status/1585038766678609921,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/504#issuecomment-1291136971,https://api.github.com/repos/simonw/sqlite-utils/issues/504,1291136971,IC_kwDOCGYnMM5M9S_L,9599,simonw,2022-10-25T21:00:29Z,2022-10-25T21:00:29Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423069384,"db.close() method, calling db.conn.close()", https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291071627,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291071627,IC_kwDOCGYnMM5M9DCL,9599,simonw,2022-10-25T20:02:18Z,2022-10-25T20:02:18Z,OWNER,Passes on Windows with other Python versions for some reason.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291076031,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291076031,IC_kwDOCGYnMM5M9EG_,9599,simonw,2022-10-25T20:06:28Z,2022-10-25T20:06:28Z,OWNER,"This is the failing test: https://github.com/simonw/sqlite-utils/blob/7b2d1c0ffd0b874e280292b926f328a61cb31e2c/tests/test_recreate.py#L21-L32 I'm going to try a different way of creating the temporary file: https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291083188,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291083188,IC_kwDOCGYnMM5M9F20,9599,simonw,2022-10-25T20:12:52Z,2022-10-25T20:12:52Z,OWNER,"Failed again, but just noticed this: https://github.com/simonw/sqlite-utils/actions/runs/3323932266/jobs/5494890223 ``` > Database(filepath, recreate=True)[""t2""].insert({""foo"": ""bar""}) tests\test_recreate.py:31: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError(""'Database' object has no attribute 'conn'"") raised in repr()] Database object at 0x29fc125aa90> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291088108,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291088108,IC_kwDOCGYnMM5M9HDs,9599,simonw,2022-10-25T20:17:36Z,2022-10-25T20:17:36Z,OWNER,"Now `mypy` is failing: ``` sqlite_utils/db.py:474: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:476: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:486: error: Item ""None"" of ""Optional[Any]"" has no attribute ""executescript"" sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__"" sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__"" sqlite_utils/db.py:604: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:607: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__"" sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__"" sqlite_utils/db.py:1083: error: Item ""None"" of ""Optional[Any]"" has no attribute ""cursor"" sqlite_utils/db.py:1155: error: Item ""None"" of ""Optional[Any]"" has no attribute ""enable_load_extension"" sqlite_utils/db.py:1156: error: Item ""None"" of ""Optional[Any]"" has no attribute ""load_extension"" Found 12 errors in 1 file (checked 51 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291093581,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291093581,IC_kwDOCGYnMM5M9IZN,9599,simonw,2022-10-25T20:23:00Z,2022-10-25T20:23:00Z,OWNER,"I'm not hugely happy with my fix there: https://github.com/simonw/sqlite-utils/blob/c5d7ec1dd71fa1dce829bc8bb82b639018befd63/sqlite_utils/db.py#L321-L328 The problem here was that in the case where the `os.remove()` failed the `self.conn` property was NOT being set to a valid connection - which caused `__repr__` to fail later on. So now I catch the `os.remove()` error, set `self.conn` to a memory connection, then raise the error again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291103021,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291103021,IC_kwDOCGYnMM5M9Kst,9599,simonw,2022-10-25T20:32:01Z,2022-10-25T20:32:01Z,OWNER,"This test reliably fails on Windows with Python 3.11. I'm going to skip the test for the moment to get back to green CI... but I'll leave this issue open. This is definitely concerning, I just don't have the right local environment to solve this at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291111357,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291111357,IC_kwDOCGYnMM5M9Mu9,9599,simonw,2022-10-25T20:36:06Z,2022-10-25T20:36:06Z,OWNER,... or maybe Windows doesn't like attempts to remove a file that the process has opened?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291115986,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291115986,IC_kwDOCGYnMM5M9N3S,9599,simonw,2022-10-25T20:39:24Z,2022-10-25T20:39:24Z,OWNER,"Used `psutil` to confirm that closing a SQLite connection closes the underlying file: https://til.simonwillison.net/python/too-many-open-files-psutil ```pycon >>> import psutil >>> import sqlite3 >>> for f in psutil.Process().open_files(): print(f) ... >>> sqlite3.connect(""/tmp/blah.db"") >>> conn = _ >>> for f in psutil.Process().open_files(): print(f) ... popenfile(path='/private/tmp/blah.db', fd=3) >>> conn.close() >>> for f in psutil.Process().open_files(): print(f) ... >>> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291122389,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291122389,IC_kwDOCGYnMM5M9PbV,9599,simonw,2022-10-25T20:45:43Z,2022-10-25T20:45:43Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291124413,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291124413,IC_kwDOCGYnMM5M9P69,9599,simonw,2022-10-25T20:47:34Z,2022-10-25T20:47:34Z,OWNER,TIL about this: https://til.simonwillison.net/python/os-remove-windows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/502#issuecomment-1291029761,https://api.github.com/repos/simonw/sqlite-utils/issues/502,1291029761,IC_kwDOCGYnMM5M840B,9599,simonw,2022-10-25T19:21:44Z,2022-10-25T19:21:44Z,OWNER,"Replicated locally using a fresh virtual environment with Python 3.11 and: pytest -k test_query_invalid_function","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422954582,Fix tests for Python 3.11, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282813168,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282813168,IC_kwDOCGYnMM5Mdizw,9599,simonw,2022-10-18T18:12:15Z,2022-10-18T18:12:15Z,OWNER,"Here's the new Tabulate release: - https://github.com/astanin/python-tabulate/releases/tag/v0.9.0 - https://github.com/astanin/python-tabulate/compare/v0.8.10...v0.9.0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282817901,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282817901,IC_kwDOCGYnMM5Mdj9t,9599,simonw,2022-10-18T18:14:35Z,2022-10-18T18:14:35Z,OWNER,"Now the 3.6 tests fail - because the new release of tabulate dropped support for that Python version (so on Python 3.6 you get an older version): https://github.com/simonw/sqlite-utils/actions/runs/3275842849/jobs/5391181675 https://github.com/astanin/python-tabulate/blame/20c6370d5da2dae89b305bfb6c7f12a0f8b7236c/pyproject.toml#L22 shows minimum is 3.7 now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282819035,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282819035,IC_kwDOCGYnMM5MdkPb,9599,simonw,2022-10-18T18:15:05Z,2022-10-18T18:15:05Z,OWNER,I'm going to skip the cog test on Python 3.6 to address this. The documentation on the website will show the available list of options for 3.7 and higher.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282830806,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282830806,IC_kwDOCGYnMM5MdnHW,9599,simonw,2022-10-18T18:23:36Z,2022-10-18T18:23:36Z,OWNER,"Tests pass now. Updated docs: - https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - and many other places on that page","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282778928,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282778928,IC_kwDOCGYnMM5Mdacw,9599,simonw,2022-10-18T17:44:20Z,2022-10-18T17:44:20Z,OWNER,"Here's how it works: https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1847-L1848 https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1082-L1088","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282779755,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282779755,IC_kwDOCGYnMM5Mdapr,9599,simonw,2022-10-18T17:45:10Z,2022-10-18T17:45:10Z,OWNER,It should go in `sqlite_utils.utils` - documented here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282780770,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282780770,IC_kwDOCGYnMM5Mda5i,9599,simonw,2022-10-18T17:45:56Z,2022-10-18T17:46:05Z,OWNER,I think the public interface is a `flatten(row)` function that does `dict(_flatten(row))`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282800547,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282800547,IC_kwDOCGYnMM5Mdfuj,9599,simonw,2022-10-18T18:02:09Z,2022-10-18T18:02:09Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-flatten,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515751719,https://api.github.com/repos/simonw/sqlite-utils/issues/50,515751719,MDEyOklzc3VlQ29tbWVudDUxNTc1MTcxOQ==,9599,simonw,2019-07-28T10:40:11Z,2019-07-28T10:40:11Z,OWNER,"I think the fix here is for me to switch to using `executemany()` - example from the Python docs: https://docs.python.org/3/library/sqlite3.html ```python purchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00), ('2006-04-05', 'BUY', 'MSFT', 1000, 72.00), ('2006-04-06', 'SELL', 'IBM', 500, 53.00), ] c.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts", https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752129,https://api.github.com/repos/simonw/sqlite-utils/issues/50,515752129,MDEyOklzc3VlQ29tbWVudDUxNTc1MjEyOQ==,9599,simonw,2019-07-28T10:46:49Z,2019-07-28T10:46:49Z,OWNER,"The problem with `.executemany()` is it breaks `lastrowid`: > This read-only attribute provides the rowid of the last modified row. It is only set if you issued an INSERT or a REPLACE statement using the execute() method. For operations other than INSERT or REPLACE or when executemany() is called, lastrowid is set to None. So I think I need to continue to use my existing way of executing bulk inserts (with a giant repeated `INSERT INTO ... VALUES` block) but ensure that I calculate the chunk size such that I don't ever try to pass more than 999 values at once.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts", https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752204,https://api.github.com/repos/simonw/sqlite-utils/issues/50,515752204,MDEyOklzc3VlQ29tbWVudDUxNTc1MjIwNA==,9599,simonw,2019-07-28T10:48:14Z,2019-07-28T10:48:14Z,OWNER,"Here's the diff where I tried to use `.executemany()` and ran into the `lastrowid` problem: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index ef55976..7f85759 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -881,13 +881,10 @@ class Table: or_what=or_what, table=self.name, columns="", "".join(""[{}]"".format(c) for c in all_columns), - rows="", "".join( - """""" + rows="""""" ({placeholders}) """""".format( - placeholders="", "".join([""?""] * len(all_columns)) - ) - for record in chunk + placeholders="", "".join([""?""] * len(all_columns)) ), ) values = [] @@ -902,15 +899,15 @@ class Table: extract_table = extracts[key] value = self.db[extract_table].lookup({""value"": value}) record_values.append(value) - values.extend(record_values) + values.append(record_values) with self.db.conn: try: - result = self.db.conn.execute(sql, values) + result = self.db.conn.executemany(sql, values) except sqlite3.OperationalError as e: if alter and ("" has no column "" in e.args[0]): # Attempt to add any missing columns, then try again self.add_missing_columns(chunk) - result = self.db.conn.execute(sql, values) + result = self.db.conn.executemany(sql, values) else: raise self.last_rowid = result.lastrowid ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts", https://github.com/simonw/sqlite-utils/issues/497#issuecomment-1291146850,https://api.github.com/repos/simonw/sqlite-utils/issues/497,1291146850,IC_kwDOCGYnMM5M9VZi,9599,simonw,2022-10-25T21:09:28Z,2022-10-25T21:09:28Z,OWNER,"Yeah, `table.columns` and `table.columns_dict` are meant to handle this: https://sqlite-utils.datasette.io/en/stable/python-api.html#columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393212964,column_names, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291167887,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291167887,IC_kwDOCGYnMM5M9aiP,9599,simonw,2022-10-25T21:33:25Z,2022-10-25T21:33:25Z,OWNER,"I do care about this, but I'm not hugely experienced with types yet so I'm open to suggestions about how to do it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291170072,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291170072,IC_kwDOCGYnMM5M9bEY,9599,simonw,2022-10-25T21:36:12Z,2022-10-25T21:36:12Z,OWNER,"I was going to suggest using `db.table(name)` instead of `db[name]` - but it looks like that method will have the same problem: https://github.com/simonw/sqlite-utils/blob/defa2974c6d3abc19be28d6b319649b8028dc966/sqlite_utils/db.py#L497-L506 I could change `sqlite-utils` so `db.table(name)` always returns a table and you need to call `db.view(name)` if you want to access a view - that would require bumping to 4.0 though. I'm not convinced that's the best approach here either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291149509,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291149509,IC_kwDOCGYnMM5M9WDF,9599,simonw,2022-10-25T21:12:11Z,2022-10-25T21:12:11Z,OWNER,"This makes sense to me. There are other places in the codebase where JSON is automatically stringified: https://github.com/simonw/sqlite-utils/blob/c7e4308e6f49d929704163531632e558f9646e4a/sqlite_utils/db.py#L2759-L2766 I don't see why the return value from a convert function shouldn't do the same thing. Since this will result in previous errors working, I don't think it warrants a major version bump either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291152433,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291152433,IC_kwDOCGYnMM5M9Wwx,9599,simonw,2022-10-25T21:14:54Z,2022-10-25T21:14:54Z,OWNER,"There is a case where the function can return a dictionary at the moment: `multi=True` ```python table.convert( ""title"", lambda v: {""upper"": v.upper(), ""lower"": v.lower()}, multi=True ) ``` But I think this change is still compatible with that. if you don't use `multi=True` then the return value will be stringified. If you DO use `multi=True` then something like this could work: ```python table.convert( ""title"", lambda v: {""upper"": {""str"": v.upper()}, ""lower"": {""str"": v.lower()}}, multi=True ) ``` This would result in a `upper` and `lower` column, each containing the JSON string `{""str"": ""UPPERCASE""}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291159549,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291159549,IC_kwDOCGYnMM5M9Yf9,9599,simonw,2022-10-25T21:23:01Z,2022-10-25T21:23:01Z,OWNER,"I've decided not to explicitly document this, since it's consistent with how other parts of the library work already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258516872,https://api.github.com/repos/simonw/sqlite-utils/issues/494,1258516872,IC_kwDOCGYnMM5LA3GI,9599,simonw,2022-09-26T19:28:36Z,2022-09-26T19:28:36Z,OWNER,New documentation: https://sqlite-utils.datasette.io/en/latest/contributing.html#using-just-and-pipenv,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386593843,Document how to use Just, https://github.com/simonw/sqlite-utils/issues/494#issuecomment-1258521333,https://api.github.com/repos/simonw/sqlite-utils/issues/494,1258521333,IC_kwDOCGYnMM5LA4L1,9599,simonw,2022-09-26T19:32:36Z,2022-09-26T19:32:36Z,OWNER,Tweeted about it too: https://twitter.com/simonw/status/1574481628507668480,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386593843,Document how to use Just, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1258476455,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1258476455,IC_kwDOCGYnMM5LAtOn,9599,simonw,2022-09-26T19:01:49Z,2022-09-26T19:01:49Z,OWNER,"I tried the tips in https://stackoverflow.com/questions/15258831/how-to-handle-two-dashes-in-rest (not the settings change though, because I might want smart quotes elsewhere) and they didn't work. Maybe I should disable smart quotes entirely? I feel like there should be an escaping trick that works here though. I tried `insert -\\-convert` but it didn't help.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1291166273,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1291166273,IC_kwDOCGYnMM5M9aJB,9599,simonw,2022-10-25T21:31:15Z,2022-10-25T21:31:15Z,OWNER,"Based on the docs here I tried the following too: https://docutils.sourceforge.io/docs/user/smartquotes.html#description - `\--` - `\\--` - `\\-\\-` - `\-\-` But none of them had the desired effect in this particular piece of markup: the :ref:`insert \--convert ` I think because this is text inside a `:ref:` block, not regular text. Consider the following: The \--convert and the :ref:`insert \--convert ` and It's rendered like this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/sqlite-utils/issues/492#issuecomment-1258446128,https://api.github.com/repos/simonw/sqlite-utils/issues/492,1258446128,IC_kwDOCGYnMM5LAl0w,9599,simonw,2022-09-26T18:32:14Z,2022-09-26T18:33:19Z,OWNER,"This idea would make more sense if there was a good mechanism to say ""run the conversion script held in this file"" as opposed to passing it as an option. This would also make having to remember bash escaping rules ([see tip](https://til.simonwillison.net/zsh/argument-heredoc)) much easier! `shot-scraper` has that for `--javascript`, using the `--input` option: https://shot-scraper.datasette.io/en/stable/javascript.html#shot-scraper-javascript-help Maybe `--convert-script` would work here? Or `--convert-file`? It should accept `-` for stdin too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386530156,Idea: ability to pass extra variables to `--convert` scripts, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258449887,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258449887,IC_kwDOCGYnMM5LAmvf,9599,simonw,2022-09-26T18:35:50Z,2022-09-26T18:35:50Z,OWNER,"This is a really interesting idea. I'm nervous about needing to set the rules for how duplicate tables should be merged though. This feels like a complex topic - one where there isn't necessarily an obviously ""correct"" way of doing it, but where different problems that people are solving might need different merging approaches. Likewise, merging isn't just a database-to-database thing at that point - I could see a need for merging two tables using similar rules to those used for merging two databases. So I think I'd want to have some good concrete use-cases in mind before trying to design how something like this should work. Will leave this thread open for people to drop those in!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258450447,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258450447,IC_kwDOCGYnMM5LAm4P,9599,simonw,2022-09-26T18:36:23Z,2022-09-26T18:36:23Z,OWNER,This is also the kind of feature that would need to express itself in both the Python library and the CLI utility.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258697384,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1258697384,IC_kwDOCGYnMM5LBjKo,9599,simonw,2022-09-26T22:12:45Z,2022-09-26T22:12:45Z,OWNER,That feels like a slightly different command to me - maybe `sqlite-utils backup data.db data-backup.db`? It doesn't have any of the mechanics for merging tables together. Could be a useful feature separately though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables, https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1256428818,https://api.github.com/repos/simonw/sqlite-utils/issues/490,1256428818,IC_kwDOCGYnMM5K45US,9599,simonw,2022-09-23T16:37:58Z,2022-09-23T16:38:35Z,OWNER,"It should be possible to achieve this with the `--text` option: https://sqlite-utils.datasette.io/en/stable/cli.html?highlight=text#convert-with-text Given an example like this in `multiline.log`: ``` 2022-03-01T12:04:52: Here is a log message that spans multiple lines 2022-03-01T12:04:52: This is a single line 2022-03-01T12:04:52: Here is another message that spans multiple lines ``` You should be able to run something like this: ``` sqlite-utils insert /tmp/log.db log multiline.log --text --convert "" import re r = re.compile(r'^(?P\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}):(?P.*)', re.MULTILINE) def convert(text): return [m.groupdict() for m in r.finditer(text)] "" ``` After running this I get: ``` sqlite-utils rows /tmp/log.db log [{""datetime"": ""2022-03-01T12:04:52"", ""log"": "" Here is a log message""}, {""datetime"": ""2022-03-01T12:04:52"", ""log"": "" This is a single line""}, {""datetime"": ""2022-03-01T12:04:52"", ""log"": "" Here is another message""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1382457780,Ability to insert multi-line files, https://github.com/simonw/sqlite-utils/issues/490#issuecomment-1258437060,https://api.github.com/repos/simonw/sqlite-utils/issues/490,1258437060,IC_kwDOCGYnMM5LAjnE,9599,simonw,2022-09-26T18:24:44Z,2022-09-26T18:24:44Z,OWNER,Just saw your great write-up on this: https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1382457780,Ability to insert multi-line files, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-514509307,https://api.github.com/repos/simonw/sqlite-utils/issues/49,514509307,MDEyOklzc3VlQ29tbWVudDUxNDUwOTMwNw==,9599,simonw,2019-07-24T07:09:43Z,2019-07-24T07:10:21Z,OWNER,"This syntax should be shared with #42 as much as possible. Maybe something based on a namedtuple would work, since those are already used in the library. ```python workouts = db.table(""workouts"", extracts=[Extract( columns=[""source"", ""source_version""], table=""Sources"" )]) ``` Since namedtuples cannot have default values this should probably be a class instead. Actually it looks like there is a trick for defaults here: https://stackoverflow.com/a/18348004","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710346830,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710346830,MDEyOklzc3VlQ29tbWVudDcxMDM0NjgzMA==,9599,simonw,2020-10-16T18:08:52Z,2020-10-16T18:09:21Z,OWNER,"The new `.extract()` method can handle multiple columns: https://github.com/simonw/sqlite-utils/blob/2c541fac352632e23e40b0d21e3f233f7a744a57/tests/test_extract.py#L70-L87","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710359724,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710359724,MDEyOklzc3VlQ29tbWVudDcxMDM1OTcyNA==,9599,simonw,2020-10-16T18:15:31Z,2020-10-16T18:15:31Z,OWNER,"Using a tuple would work: ```python fresh_db.table(""tree"", extracts=[(""common_name"", ""latin_name"")]) ``` Or to define a custom name: ```python fresh_db.table(""tree"", extracts={(""common_name"", ""latin_name""): ""names""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710363789,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710363789,MDEyOklzc3VlQ29tbWVudDcxMDM2Mzc4OQ==,9599,simonw,2020-10-16T18:18:05Z,2020-10-16T18:18:05Z,OWNER,I wonder if there's value in extending the `extracts=` option at all given the existence of `table.extract()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710364942,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710364942,MDEyOklzc3VlQ29tbWVudDcxMDM2NDk0Mg==,9599,simonw,2020-10-16T18:18:48Z,2020-10-16T18:18:48Z,OWNER,"I think there is. It's a nice existing feature, and I don't think adding tuple support to it would be a huge lift.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710390915,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710390915,MDEyOklzc3VlQ29tbWVudDcxMDM5MDkxNQ==,9599,simonw,2020-10-16T18:34:26Z,2020-10-16T18:34:50Z,OWNER,"Here's the most complex example of `.extracts()`: ```python db[""Trees""].extract( [""CommonName"", ""LatinName""], table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} ) ``` Resulting in: ```sql CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [latin] TEXT ) ``` From https://sqlite-utils.readthedocs.io/en/stable/python-api.html#extracting-columns-into-a-separate-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710393550,MDEyOklzc3VlQ29tbWVudDcxMDM5MzU1MA==,9599,simonw,2020-10-16T18:35:57Z,2020-10-16T18:36:39Z,OWNER,"If I want to support that most complicated example, I think the option to pass a `Extracts()` object to `extracts=` is the best way to do it: ```python fresh_db.table(""tree"", extracts=[Extract( columns=(""CommonName"", ""LatinName""), table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} )]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710395444,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710395444,MDEyOklzc3VlQ29tbWVudDcxMDM5NTQ0NA==,9599,simonw,2020-10-16T18:37:10Z,2020-10-16T18:37:10Z,OWNER,"But this begins to feel too complicated, given that `table.extract()` can already be used to achieve the same thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710397574,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710397574,MDEyOklzc3VlQ29tbWVudDcxMDM5NzU3NA==,9599,simonw,2020-10-16T18:38:21Z,2020-10-16T18:38:21Z,OWNER,"I'm not going to implement this. I'll leave `extract=...` as it is right now, suitable for quick simple single-column operations on input, but if users want to do something more complicated involving multiple columns they should use the `table.extract()` method after the initial insert instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710461468,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710461468,MDEyOklzc3VlQ29tbWVudDcxMDQ2MTQ2OA==,9599,simonw,2020-10-16T19:18:19Z,2020-10-16T19:18:19Z,OWNER,"Reconsidering: #89 was a feature request that relates to this, so maybe this is worth implementing after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248474806,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248474806,IC_kwDOCGYnMM5Kaja2,9599,simonw,2022-09-15T18:48:09Z,2022-09-15T18:48:09Z,OWNER,"Built a prototype of this that works really well: ```diff diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index c0b7bf1..f9a482c 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -272,7 +272,19 @@ def rows_from_file( if format == Format.JSON: decoded = json.load(fp) if isinstance(decoded, dict): - decoded = [decoded] + # TODO: Solve for if this isn't what people want + # Does it have just one key that is a list of dicts? + list_keys = [ + k + for k in decoded + if isinstance(decoded[k], list) + and decoded[k] + and all(isinstance(o, dict) for o in decoded[k]) + ] + if len(list_keys) == 1: + decoded = decoded[list_keys[0]] + else: + decoded = [decoded] if not isinstance(decoded, list): raise RowsFromFileBadJSON(""JSON must be a list or a dictionary"") return decoded, Format.JSON ``` I used that to build this: https://gist.github.com/simonw/0e6901974a14ab7d56c2746a04d72c8c One problem though: right now, if you do this `sqlite-utils` treats it as a single object and adds a `tags` column with JSON in it: ``` echo '{""title"": ""Hi"", ""tags"": [{""t"": ""one""}]}` | sqlite-utils insert db.db t - ``` If I implement this new mechanism the above line would behave differently - which would be a backwards incompatible change. So I probably need some kind of opt-in mechanism for this. And I need a good name for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248475718,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248475718,IC_kwDOCGYnMM5KajpG,9599,simonw,2022-09-15T18:49:05Z,2022-09-15T18:49:53Z,OWNER,"Here's how I used my prototype to build [that Gist](https://gist.github.com/simonw/0e6901974a14ab7d56c2746a04d72c8c): sqlite-utils memory ~/Downloads/CVR_Export_20220908084311/*.json --schema > database.sql ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248479485,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248479485,IC_kwDOCGYnMM5Kakj9,9599,simonw,2022-09-15T18:52:52Z,2022-09-15T18:53:45Z,OWNER,"The most similar option I have at the moment is probably `--flatten`. What would good names for this option be? - `--auto-list` - `--auto-key` - `--inner-key` - `--auto-json` - `--find-list` - `--find-key` Those are all bad. Another option: introduce a new explicit format for it. Right now the explicit formats you can use are: https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/docs/cli-reference.rst#L153-L158 So I could add a `:autojson` format.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248481303,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248481303,IC_kwDOCGYnMM5KalAX,9599,simonw,2022-09-15T18:54:30Z,2022-09-15T18:55:14Z,OWNER,"Maybe this would make more sense as a mechanism where you can say ""Use the data in the key called X"" - but there's a special option for ""figure out that key automatically"". The syntax then could be: `--list-key List` Or for automatic detection: `--list-key-auto` Could also go with `--key List` and `--key-auto` - but would that be as obvious as `--list-key`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248484094,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248484094,IC_kwDOCGYnMM5Kalr-,9599,simonw,2022-09-15T18:56:31Z,2022-09-15T18:56:31Z,OWNER,"Actually I quite like `--key X` - it could work for single nested objects too. You could insert a single record like this: ```json { ""record"" { ""id"": 1 } } ``` ``` sqlite-utils insert db.db records record.json --key record ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248501824,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248501824,IC_kwDOCGYnMM5KaqBA,9599,simonw,2022-09-15T19:10:48Z,2022-09-15T19:10:48Z,OWNER,"This feels pretty good: ``` % sqlite-utils memory ~/Downloads/CVR_Export_20220908084311/*.json --schema --auto-key CREATE TABLE [BallotTypeContestManifest] ( [BallotTypeId] INTEGER, [ContestId] INTEGER ); CREATE VIEW t1 AS select * from [BallotTypeContestManifest]; CREATE VIEW t AS select * from [BallotTypeContestManifest]; CREATE TABLE [BallotTypeManifest] ( [Description] TEXT, [Id] INTEGER, [ExternalId] TEXT ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248512739,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248512739,IC_kwDOCGYnMM5Kasrj,9599,simonw,2022-09-15T19:18:24Z,2022-09-15T19:21:01Z,OWNER,"Why doesn't `sqlite-utils insert` use the `rows_from_file` function I wonder? https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841 says: > I can refactor `sqlite-utils insert` to use this new code too. Maybe I forgot to do that?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248522618,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248522618,IC_kwDOCGYnMM5KavF6,9599,simonw,2022-09-15T19:29:20Z,2022-09-15T19:29:20Z,OWNER,I think refactoring `sqlite-utils insert` to use `rows_from_file` needs to happen as part of this work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/489#issuecomment-1248621072,https://api.github.com/repos/simonw/sqlite-utils/issues/489,1248621072,IC_kwDOCGYnMM5KbHIQ,9599,simonw,2022-09-15T20:56:09Z,2022-09-15T20:56:09Z,OWNER,"Prototype so far: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 767b170..d96c507 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1762,6 +1762,17 @@ def query( is_flag=True, help=""Analyze resulting tables and output results"", ) +@click.option(""--key"", help=""read data from this key of the root object"") +@click.option( + ""--auto-key"", + is_flag=True, + help=""Find a key in the root object that is a list of objects"", +) +@click.option( + ""--analyze"", + is_flag=True, + help=""Analyze resulting tables and output results"", +) @load_extension_option def memory( paths, @@ -1784,6 +1795,8 @@ def memory( schema, dump, save, + key, + auto_key, analyze, load_extension, ): @@ -1838,7 +1851,9 @@ def memory( csv_table = stem stem_counts[stem] = stem_counts.get(stem, 1) + 1 csv_fp = csv_path.open(""rb"") - rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding) + rows, format_used = rows_from_file( + csv_fp, format=format, encoding=encoding, key=key, auto_key=auto_key + ) tracker = None if format_used in (Format.CSV, Format.TSV) and not no_detect_types: tracker = TypeTracker() diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index 8754554..2e69c26 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -231,6 +231,8 @@ def rows_from_file( encoding: Optional[str] = None, ignore_extras: Optional[bool] = False, extras_key: Optional[str] = None, + key: Optional[str] = None, + auto_key: Optional[bool] = False, ) -> Tuple[Iterable[dict], Format]: """""" Load a sequence of dictionaries from a file-like object containing one of four different formats. @@ -271,13 +273,31 @@ def rows_from_file( :param encoding: the character encoding to use when reading CSV/TSV data :param ignore_extras: ignore any extra fields on rows :param extras_key: put any extra fields in a list with this key + :param key: read data from this key of the root object + :param auto_key: find a key in the root object that is a list of objects """""" if ignore_extras and extras_key: raise ValueError(""Cannot use ignore_extras= and extras_key= together"") + if key and auto_key: + raise ValueError(""Cannot use key= and auto_key= together"") if format == Format.JSON: decoded = json.load(fp) if isinstance(decoded, dict): - decoded = [decoded] + if auto_key: + list_keys = [ + k + for k in decoded + if isinstance(decoded[k], list) + and decoded[k] + and all(isinstance(o, dict) for o in decoded[k]) + ] + if len(list_keys) == 1: + decoded = decoded[list_keys[0]] + elif key: + # Raises KeyError, I think that's OK + decoded = decoded[key] + if not isinstance(decoded, list): + decoded = [decoded] if not isinstance(decoded, list): raise RowsFromFileBadJSON(""JSON must be a list or a dictionary"") return decoded, Format.JSON @@ -305,7 +325,9 @@ def rows_from_file( first_bytes = buffered.peek(2048).strip() if first_bytes.startswith(b""["") or first_bytes.startswith(b""{""): # TODO: Detect newline-JSON - return rows_from_file(buffered, format=Format.JSON) + return rows_from_file( + buffered, format=Format.JSON, key=key, auto_key=auto_key + ) else: dialect = csv.Sniffer().sniff( first_bytes.decode(encoding or ""utf-8-sig"", ""ignore"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1374939463,Ability to load JSON records held in a file with a single top level key that is a list of objects, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1246971764,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1246971764,IC_kwDOCGYnMM5KU0d0,9599,simonw,2022-09-14T15:52:14Z,2022-09-14T15:52:14Z,OWNER,"Frustratingly I think this counts as a backwards-incompatible change. Could have it be opt-in with a new option / method parameter, and then change it to the default if I release a `sqlite-utils 4`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254029808,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1254029808,IC_kwDOCGYnMM5Kvvnw,9599,simonw,2022-09-21T17:45:20Z,2022-09-21T17:45:41Z,OWNER,"No, I'm going to say that this is a bug - it's WEIRD having a `integer` or `float` column containing an empty string. I'm OK changing that - I very much doubt anyone is relying on this functionality. So no need for a new option here - just fixing the bug is sensible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254032378,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1254032378,IC_kwDOCGYnMM5KvwP6,9599,simonw,2022-09-21T17:47:54Z,2022-09-21T17:47:54Z,OWNER,"New tests should go in: https://github.com/simonw/sqlite-utils/blob/main/tests/test_transform.py I think the implementation fix needs to go near here: https://github.com/simonw/sqlite-utils/blob/0b315d3fa83c1584eaeec32f24912898621e437a/sqlite_utils/db.py#L1770-L1775 The trick is going to be teaching that generated SQL to know which columns are `integer` or `float` and to convert `""""` to `null` as part of that operation. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1254033981,https://api.github.com/repos/simonw/sqlite-utils/issues/488,1254033981,IC_kwDOCGYnMM5Kvwo9,9599,simonw,2022-09-21T17:49:32Z,2022-09-21T17:50:10Z,OWNER,"It looks like SQLite has a `SELECT NULLIF(value, '')` function which returns `null` if that value is equal to `''`. We need to only apply that function to columns that we know to be of type integer or float though - text columns containing empty strings should not be rewritten to null.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373224657,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float, https://github.com/simonw/sqlite-utils/issues/487#issuecomment-1242409766,https://api.github.com/repos/simonw/sqlite-utils/issues/487,1242409766,IC_kwDOCGYnMM5KDasm,9599,simonw,2022-09-09T20:04:30Z,2022-09-09T20:04:30Z,OWNER,"This isn't supported yet - there's an older issue for that here: - #117 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1367835380,Specify foreign key against compound key in other table, https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248565396,IC_kwDOCGYnMM5Ka5iU,9599,simonw,2022-09-15T20:12:50Z,2022-09-15T20:12:50Z,OWNER,"Annoying `mypy` test failure: ``` /Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636: error: Positional-only parameters are only supported in Python 3.8 and greater ``` Looks like this: - https://github.com/python/mypy/issues/13627","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248567323,IC_kwDOCGYnMM5Ka6Ab,9599,simonw,2022-09-15T20:14:45Z,2022-09-15T20:14:45Z,OWNER,"There's a fix for `mypy` that has landed but isn't out in a release yet: - https://github.com/python/mypy/issues/13385 For the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248568775,IC_kwDOCGYnMM5Ka6XH,9599,simonw,2022-09-15T20:16:14Z,2022-09-15T20:16:14Z,OWNER,"https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version: ``` - uses: actions/setup-python@v4 with: python-version: ""3.10.6"" ``` I'll try that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248582147,IC_kwDOCGYnMM5Ka9oD,9599,simonw,2022-09-15T20:29:17Z,2022-09-15T20:29:17Z,OWNER,This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248591268,IC_kwDOCGYnMM5Ka_2k,9599,simonw,2022-09-15T20:36:02Z,2022-09-15T20:40:03Z,OWNER,"I had a big CSV file lying around, I converted it to other formats like this: sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl sqlite-utils rows /tmp/t.db t > /tmp/big.json Then tested the progress bar like this: sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl Output: ``` sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl [------------------------------------] 0% [#######-----------------------------] 20% 00:00:20 ``` With `--silent` it is silent. And for regular JSON: ``` sqlite-utils insert /tmp/t3.db t /tmp/big.json [####################################] 100% ``` This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835,https://api.github.com/repos/simonw/sqlite-utils/issues/486,1248593835,IC_kwDOCGYnMM5KbAer,9599,simonw,2022-09-15T20:37:14Z,2022-09-15T20:37:14Z,OWNER,"I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366512990,"progressbar for inserts/upserts of all fileformats, closes #485", https://github.com/simonw/sqlite-utils/issues/485#issuecomment-1248597643,https://api.github.com/repos/simonw/sqlite-utils/issues/485,1248597643,IC_kwDOCGYnMM5KbBaL,9599,simonw,2022-09-15T20:39:39Z,2022-09-15T20:39:52Z,OWNER,"A note from PR #486: https://github.com/simonw/sqlite-utils/issues/486#issuecomment-1248591268_ > ``` > sqlite-utils insert /tmp/t3.db t /tmp/big.json > [####################################] 100% > ``` > This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%. I decided to land this anyway. If a streaming JSON parser is added later it will start to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1366423176,Progressbar not shown when inserting/upserting jsonlines file, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1238607591,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1238607591,IC_kwDOCGYnMM5J06bn,9599,simonw,2022-09-06T20:16:39Z,2022-09-06T20:16:39Z,OWNER,"Here's the implementation for recipes at the moment: https://github.com/simonw/sqlite-utils/blob/5b969273f1244b1bcf3e4dc071cdf17dab35d5f8/sqlite_utils/utils.py#L434-L441 And here's the `--functions` implementation that doesn't expose them: https://github.com/simonw/sqlite-utils/blob/5b969273f1244b1bcf3e4dc071cdf17dab35d5f8/sqlite_utils/cli.py#L3029-L3040","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239697643,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239697643,IC_kwDOCGYnMM5J5Ejr,9599,simonw,2022-09-07T17:48:00Z,2022-09-07T17:48:00Z,OWNER,Will also need to update documentation here: https://sqlite-utils.datasette.io/en/stable/cli.html#defining-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239699276,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239699276,IC_kwDOCGYnMM5J5E9M,9599,simonw,2022-09-07T17:49:49Z,2022-09-07T17:49:49Z,OWNER,"This feature is a tiny bit weird though: the recipe functions are not exposed to SQL by default, they are instead designed to be used with `sqlite-utils convert`. I guess with `--functions` support you could do something like this: sqlite-utils data.db ""update mytable set col1 = parsedate(col1)"" --functions ""parsedate = r.parsedate""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239759022,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239759022,IC_kwDOCGYnMM5J5Tiu,9599,simonw,2022-09-07T18:52:08Z,2022-09-07T18:52:08Z,OWNER,"It's not quite that simple. I tried applying this patch: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index c51b101..33e4d90 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -30,6 +30,7 @@ from .utils import ( Format, TypeTracker, ) +from . import recipes CONTEXT_SETTINGS = dict(help_option_names=[""-h"", ""--help""]) @@ -3029,7 +3030,7 @@ def _load_extensions(db, load_extension): def _register_functions(db, functions): # Register any Python functions as SQL functions: sqlite3.enable_callback_tracebacks(True) - globals = {} + globals = {""r"": recipes, ""recipes"": recipes} try: exec(functions, globals) except SyntaxError as ex: ``` Then got this: ``` % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate(""1st jan"")' Error: wrong number of arguments to function parsedate() % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate(""1st jan"", 0, 0, 0)' [{""parsedate(\""1st jan\"", 0, 0, 0)"": ""2022-01-01""}] ``` The problem here is that the `parsedate` function signature looks like this: https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/recipes.py#L8 But the code that register SQL functions introspects that signature, so creates a SQL function that requires four arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239760001,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239760001,IC_kwDOCGYnMM5J5TyB,9599,simonw,2022-09-07T18:53:17Z,2022-09-07T18:53:17Z,OWNER,"So you would need to do this instead: ``` sqlite-utils memory 'select parsedate(""1st jan"")' --functions ' def parsedate(s): return r.parsedate(s) ' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239761280,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239761280,IC_kwDOCGYnMM5J5UGA,9599,simonw,2022-09-07T18:54:51Z,2022-09-07T18:54:51Z,OWNER,"I could teach this code here to only register the function using arguments that don't have default parameters: https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/cli.py#L3037-L3040 Or even this code here: https://github.com/simonw/sqlite-utils/blob/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8/sqlite_utils/db.py#L398-L418 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239762031,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239762031,IC_kwDOCGYnMM5J5URv,9599,simonw,2022-09-07T18:55:30Z,2022-09-07T18:55:30Z,OWNER,That would be a breaking change though - existing code that registers functions with default parameters should continue to work unchanged (unless I want to ship `sqlite-utils` 4.0).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239762561,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239762561,IC_kwDOCGYnMM5J5UaB,9599,simonw,2022-09-07T18:56:13Z,2022-09-07T18:56:13Z,OWNER,"I could do this: ```python # Register all callables in the locals dict: for name, value in globals.items(): if callable(value) and not name.startswith(""_""): db.register_function(value, name=name, ignore_params_with_defaults=True) ``` Introducing a new `ignore_params_with_defaults` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239763997,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239763997,IC_kwDOCGYnMM5J5Uwd,9599,simonw,2022-09-07T18:57:59Z,2022-09-07T18:58:10Z,OWNER,"Here's how to detect defaults in the function signature: ```pycon >>> import inspect >>> def foo(a, b, c=1, d=2): ... pass ... >>> inspect.signature(foo) >>> inspect.signature(foo).parameters mappingproxy(OrderedDict([('a', ), ('b', ), ('c', ), ('d', )])) >>> inspect.signature(foo).parameters['c'] >>> dir(inspect.signature(foo).parameters['c']) ['KEYWORD_ONLY', 'POSITIONAL_ONLY', 'POSITIONAL_OR_KEYWORD', 'VAR_KEYWORD', 'VAR_POSITIONAL', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_annotation', '_default', '_kind', '_name', 'annotation', 'default', 'empty', 'kind', 'name', 'replace'] >>> inspect.signature(foo).parameters['c'].default 1 >>> inspect.signature(foo).parameters['a'].default ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239766987,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239766987,IC_kwDOCGYnMM5J5VfL,9599,simonw,2022-09-07T19:01:49Z,2022-09-07T19:01:49Z,OWNER,"OK with this: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index c51b101..93d82a9 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -30,6 +30,7 @@ from .utils import ( Format, TypeTracker, ) +from . import recipes CONTEXT_SETTINGS = dict(help_option_names=[""-h"", ""--help""]) @@ -3029,7 +3030,7 @@ def _load_extensions(db, load_extension): def _register_functions(db, functions): # Register any Python functions as SQL functions: sqlite3.enable_callback_tracebacks(True) - globals = {} + globals = {""r"": recipes, ""recipes"": recipes} try: exec(functions, globals) except SyntaxError as ex: @@ -3037,4 +3038,4 @@ def _register_functions(db, functions): # Register all callables in the locals dict: for name, value in globals.items(): if callable(value) and not name.startswith(""_""): - db.register_function(value, name=name) + db.register_function(value, name=name, ignore_defaults=True) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 27c46b0..1407d23 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -370,6 +370,7 @@ class Database: self, fn: Callable = None, deterministic: bool = False, + ignore_defaults: bool = False, replace: bool = False, name: Optional[str] = None, ): @@ -397,7 +398,10 @@ class Database: def register(fn): fn_name = name or fn.__name__ - arity = len(inspect.signature(fn).parameters) + params = inspect.signature(fn).parameters + if ignore_defaults: + params = [p for p in params if params[p].default is inspect._empty] + arity = len(params) if not replace and (fn_name, arity) in self._registered_functions: return fn kwargs = {} ``` I can now do this: ``` % sqlite-utils memory --functions 'parsedate = r.parsedate' 'select parsedate(""1st jan"")' [{""parsedate(\""1st jan\"")"": ""2022-01-01""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/484#issuecomment-1239772256,https://api.github.com/repos/simonw/sqlite-utils/issues/484,1239772256,IC_kwDOCGYnMM5J5Wxg,9599,simonw,2022-09-07T19:07:51Z,2022-09-07T19:09:52Z,OWNER,"Or... I could automatically register multiple copies of the function of different arities! If I'm going to do something like that though I need to think carefully about how functions that have keyword-only arguments should work: https://peps.python.org/pep-3102/ ```python def compare(a, b, *ignore, key=None): ... ``` I should think about how these work with `db.register_function()` anyway, since SQL functions cannot support keyword arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363766973,Expose convert recipes to `sqlite-utils --functions`, https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258451968,https://api.github.com/repos/simonw/sqlite-utils/issues/483,1258451968,IC_kwDOCGYnMM5LAnQA,9599,simonw,2022-09-26T18:37:54Z,2022-09-26T18:40:41Z,OWNER,"The implementation of this can be an almost exact copy of Datasette's, which was added in this commit: https://github.com/simonw/datasette/commit/01fe5b740171bfaea3752fc5754431dac53777e3 Current code for that is here: https://github.com/simonw/datasette/blob/0.62/datasette/cli.py#L319-L340 - which is improved to use the `from runpy import run_module` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363765916,`sqlite-utils install` command, https://github.com/simonw/sqlite-utils/issues/483#issuecomment-1258479462,https://api.github.com/repos/simonw/sqlite-utils/issues/483,1258479462,IC_kwDOCGYnMM5LAt9m,9599,simonw,2022-09-26T19:04:29Z,2022-09-26T19:04:43Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-uninstall - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#install - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#uninstall ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1363765916,`sqlite-utils install` command, https://github.com/simonw/sqlite-utils/issues/482#issuecomment-1236511153,https://api.github.com/repos/simonw/sqlite-utils/issues/482,1236511153,IC_kwDOCGYnMM5Js6mx,9599,simonw,2022-09-05T03:38:29Z,2022-09-05T03:38:29Z,OWNER,"This is an interesting research area! Running the same experiment as the one in https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/ could demonstrate one way or another if column ordering makes a difference with regards to performance or not. My hunch is that even if it does this wouldn't be a feature to bake into `sqlite-utils` itself, beyond the existing features for changing the order of columns in a table (https://sqlite-utils.datasette.io/en/stable/python-api.html#changing-column-order).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1361355564,balanced table default column_order, https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232089808,https://api.github.com/repos/simonw/sqlite-utils/issues/480,1232089808,IC_kwDOCGYnMM5JcDLQ,9599,simonw,2022-08-30T19:43:02Z,2022-08-30T19:43:02Z,OWNER,"Yeah this seems like a reasonable addition to me. Needs a test, which can go next to this one: https://github.com/simonw/sqlite-utils/blob/087753cd42c406f1e060c1822dcd9b5fda3d60f4/tests/test_fts.py#L561","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1355433619,search_sql add include_rank option, https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232419522,https://api.github.com/repos/simonw/sqlite-utils/issues/480,1232419522,IC_kwDOCGYnMM5JdTrC,9599,simonw,2022-08-31T03:33:27Z,2022-08-31T03:33:27Z,OWNER,"Tests look great, thank you!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1355433619,search_sql add include_rank option, https://github.com/simonw/sqlite-utils/issues/48#issuecomment-710402331,https://api.github.com/repos/simonw/sqlite-utils/issues/48,710402331,MDEyOklzc3VlQ29tbWVudDcxMDQwMjMzMQ==,9599,simonw,2020-10-16T18:41:06Z,2020-10-16T18:41:06Z,OWNER,I could use this demo from JupyterCon 2020 https://gist.github.com/simonw/656c21b5800d5e4624dec9930f00e093,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder", https://github.com/simonw/sqlite-utils/issues/48#issuecomment-1021790707,https://api.github.com/repos/simonw/sqlite-utils/issues/48,1021790707,IC_kwDOCGYnMM4850nz,9599,simonw,2022-01-26T02:08:39Z,2022-01-26T02:08:39Z,OWNER,This is a dupe of #308.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder", https://github.com/simonw/sqlite-utils/issues/478#issuecomment-1229563718,https://api.github.com/repos/simonw/sqlite-utils/issues/478,1229563718,IC_kwDOCGYnMM5JSadG,9599,simonw,2022-08-28T22:07:03Z,2022-08-28T22:07:03Z,OWNER,"What should happen if you run this: sqlite-utils tables data.db bad_table Where `bad_table` is a table that does not exist? I think it should error rather than failing silently. That way you can use the error code to confirm that a list of expected tables exist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353481513,`sqlite-utils tables data.db table1 table2`, https://github.com/simonw/sqlite-utils/issues/477#issuecomment-1238608921,https://api.github.com/repos/simonw/sqlite-utils/issues/477,1238608921,IC_kwDOCGYnMM5J06wZ,9599,simonw,2022-09-06T20:18:12Z,2022-09-06T20:18:12Z,OWNER,"Hi @thewchan - thanks for that! Yes I'm interested, can you invite `swillison` @ Google's email provider please?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353441389,Conda Forge, https://github.com/simonw/sqlite-utils/issues/476#issuecomment-1229320114,https://api.github.com/repos/simonw/sqlite-utils/issues/476,1229320114,IC_kwDOCGYnMM5JRe-y,9599,simonw,2022-08-27T23:26:48Z,2022-08-27T23:26:48Z,OWNER,"- #458 - the `register_function(name=...)` argument - New tutorial: [Cleaning data with sqlite-utils and Datasette](https://datasette.io/tutorials/clean-data) provides a tutorial introduction (and accompanying ten minute video) about using this tool. - New Discord community, https://discord.gg/Ass7bCAMDw - #469 `sqlite-utils rows --order` option - #471 `sqlite-utils query --functions` option - #472 Improved code compilation pattern - #473 Support entrypoints for `--load-extension` - #455 - #475 `table.default_values` property - #467","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353196970,Release notes for 3.29, https://github.com/simonw/sqlite-utils/issues/476#issuecomment-1229370823,https://api.github.com/repos/simonw/sqlite-utils/issues/476,1229370823,IC_kwDOCGYnMM5JRrXH,9599,simonw,2022-08-28T04:07:15Z,2022-08-28T04:07:15Z,OWNER,https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-29,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353196970,Release notes for 3.29, https://github.com/simonw/sqlite-utils/issues/475#issuecomment-1229283063,https://api.github.com/repos/simonw/sqlite-utils/issues/475,1229283063,IC_kwDOCGYnMM5JRV73,9599,simonw,2022-08-27T22:44:45Z,2022-08-27T22:44:45Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#default-values - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.default_values","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353189941,table.default_values introspection property, https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1229265285,https://api.github.com/repos/simonw/sqlite-utils/issues/474,1229265285,IC_kwDOCGYnMM5JRRmF,9599,simonw,2022-08-27T20:52:53Z,2022-08-27T20:52:53Z,OWNER,"A couple of tricks I use here. Firstly, I often create the table before the import using the `sqlite-utils create-table` command: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-tables The other current option is to use the `bulk` command, which lets you construct a custom SQL query to execute against every row from a CSV file: https://sqlite-utils.datasette.io/en/stable/cli.html#executing-sql-in-bulk Do either of those options work here or is there a useful new feature that would work better?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353074021,Add an option for specifying column names when inserting CSV data, https://github.com/simonw/sqlite-utils/issues/474#issuecomment-1232424050,https://api.github.com/repos/simonw/sqlite-utils/issues/474,1232424050,IC_kwDOCGYnMM5JdUxy,9599,simonw,2022-08-31T03:42:28Z,2022-08-31T03:42:28Z,OWNER,"Yeah, having a way of inserting a CSV file but saying ""only I want the `title` column to come first"" does make sense to me as a feature. It can use `-o` for consistency with https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search and https://sqlite-utils.datasette.io/en/stable/cli-reference.html#rows and https://sqlite-utils.datasette.io/en/stable/cli-reference.html#transform","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1353074021,Add an option for specifying column names when inserting CSV data, https://github.com/simonw/sqlite-utils/issues/472#issuecomment-1229125614,https://api.github.com/repos/simonw/sqlite-utils/issues/472,1229125614,IC_kwDOCGYnMM5JQvfu,9599,simonw,2022-08-27T05:12:59Z,2022-08-27T05:12:59Z,OWNER,"Annoyingly this means the `--import` option is no longer required, but removing it would be a backwards incompatible breakage so I need to leave it in. I can minimize it in the documentation though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352946135,Reuse the locals/globals fix from --functions for other code accepting options, https://github.com/simonw/sqlite-utils/issues/472#issuecomment-1229125890,https://api.github.com/repos/simonw/sqlite-utils/issues/472,1229125890,IC_kwDOCGYnMM5JQvkC,9599,simonw,2022-08-27T05:15:26Z,2022-08-27T05:15:26Z,OWNER,Here's the docs I get to update to remove the documented `global` hack: https://sqlite-utils.datasette.io/en/3.28/cli.html#cli-convert-complex,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352946135,Reuse the locals/globals fix from --functions for other code accepting options, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229116423,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229116423,IC_kwDOCGYnMM5JQtQH,9599,simonw,2022-08-27T04:00:52Z,2022-08-27T04:00:52Z,OWNER,"Alternative design would be `--function name definition` - like this: ``` sqlite-utils data.db 'update images set domain = extract_domain(url)' --function extract_domain ' from urllib.parse import urlparse return urlparse(url).netloc ' ``` I like the `--functions` design better because it leaves space for import statements at the top of the code block, and allows more than one function to be defined in a single go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229118619,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229118619,IC_kwDOCGYnMM5JQtyb,9599,simonw,2022-08-27T04:14:52Z,2022-08-27T04:14:52Z,OWNER,"Quick prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 43e76fa..5dee4f6 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1633,6 +1633,9 @@ def drop_view(path, view, ignore, load_extension): type=(str, str), help=""Named :parameters for SQL query"", ) +@click.option( + ""--functions"", help=""Python code defining one or more custom SQL functions"" +) @load_extension_option def query( path, @@ -1649,6 +1652,7 @@ def query( raw, param, load_extension, + functions, ): """"""Execute SQL query and return the results as JSON @@ -1665,6 +1669,16 @@ def query( _load_extensions(db, load_extension) db.register_fts4_bm25() + # Register any Python functions as SQL functions: + if functions: + locals = {} + globals = {} + exec(functions, globals, locals) + # Register all callables in the locals dict: + for name, value in locals.items(): + if callable(value): + db.register_function(value, name=name) + _execute_query( db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols ) ``` Demo: ```bash % sqlite-utils :memory: 'select 1 + dog()' --functions ' quote> def dog(): quote> return 2 quote> ' [{""1 + dog()"": 3}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119171,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229119171,IC_kwDOCGYnMM5JQt7D,9599,simonw,2022-08-27T04:18:28Z,2022-08-27T04:18:28Z,OWNER,"I tried this: ``` sqlite-utils :memory: 'select extract_domain(""https://www.google.com/blah"")' --functions ' from urllib.parse import urlparse def extract_domain(url): from urllib.parse import urlparse return urlparse(url).netloc ' ``` And got: ``` NameError: name 'urlparse' is not defined Error: user-defined function raised exception ``` But this worked OK: ``` % sqlite-utils :memory: 'select extract_domain(""https://www.google.com/blah"")' --functions ' def extract_domain(url): from urllib.parse import urlparse return urlparse(url).netloc ' [{""extract_domain(\""https://www.google.com/blah\"")"": ""www.google.com""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229119999,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229119999,IC_kwDOCGYnMM5JQuH_,9599,simonw,2022-08-27T04:24:58Z,2022-08-27T04:24:58Z,OWNER,"I've encountered this problem before: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-convert-complex > ``` > $ sqlite-utils convert content.db articles score ' > import random > random.seed(10) > > def convert(value): > global random > return random.random() > ' > ``` > Note the `global random` line here. Due to the way the tool compiles Python code, this is necessary to ensure the `random` module is available within the `convert()` function. If you were to omit this you would see a `NameError: name 'random' is not defined` error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120104,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229120104,IC_kwDOCGYnMM5JQuJo,9599,simonw,2022-08-27T04:25:39Z,2022-08-27T04:25:39Z,OWNER,"This works: ``` sqlite-utils :memory: 'select extract_domain(""https://www.google.com/blah"")' --functions ' from urllib.parse import urlparse def extract_domain(url): global urlparse return urlparse(url).netloc ' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120653,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229120653,IC_kwDOCGYnMM5JQuSN,9599,simonw,2022-08-27T04:29:49Z,2022-08-27T04:30:03Z,OWNER,"Found a fix for that! I replaced this: ```python locals = {} globals = {} exec(functions, globals, locals) # Register all callables in the locals dict: for name, value in locals.items(): if callable(value): db.register_function(value, name=name) ``` With this: ```python globals = {} exec(functions, globals) # Register all callables in the globals dict: for name, value in globals.items(): if callable(value): db.register_function(value, name=name) ``` Because https://docs.python.org/3/library/functions.html#exec says: > If only *globals* is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If *globals* and *locals* are given, they are used for the global and local variables, respectively.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120899,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229120899,IC_kwDOCGYnMM5JQuWD,9599,simonw,2022-08-27T04:31:35Z,2022-08-27T04:32:38Z,OWNER,I should add this `--functions` feature to the `memory` and `bulk` commands too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124379,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229124379,IC_kwDOCGYnMM5JQvMb,9599,simonw,2022-08-27T05:02:21Z,2022-08-27T05:02:21Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#defining-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229124549,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229124549,IC_kwDOCGYnMM5JQvPF,9599,simonw,2022-08-27T05:03:39Z,2022-08-27T05:03:39Z,OWNER,"I don't think I need separate documentation for `sqlite-utils memory` and `sqlite-tils bulk` since they work the same, and the `--help` text provides the necessary hints.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229125114,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1229125114,IC_kwDOCGYnMM5JQvX6,9599,simonw,2022-08-27T05:08:58Z,2022-08-27T05:08:58Z,OWNER,"Testing `bulk --functions`: ``` % sqlite-utils create-table chickens.db chickens id integer name text name_upper text % echo 'id,name 1,Blue 2,Snowy 3,Azi 4,Lila 5,Suna 6,Cardi' | sqlite-utils bulk chickens.db ' insert into chickens (id, name, name_upper) values (:id, :name, myupper(:name)) ' - --functions ' def myupper(s): return s.upper() ' --csv % sqlite-utils rows chickens.db chickens [{""id"": 1, ""name"": ""Blue"", ""name_upper"": ""BLUE""}, {""id"": 2, ""name"": ""Snowy"", ""name_upper"": ""SNOWY""}, {""id"": 3, ""name"": ""Azi"", ""name_upper"": ""AZI""}, {""id"": 4, ""name"": ""Lila"", ""name_upper"": ""LILA""}, {""id"": 5, ""name"": ""Suna"", ""name_upper"": ""SUNA""}, {""id"": 6, ""name"": ""Cardi"", ""name_upper"": ""CARDI""}] ```","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1238873948,https://api.github.com/repos/simonw/sqlite-utils/issues/471,1238873948,IC_kwDOCGYnMM5J17dc,9599,simonw,2022-09-07T03:46:26Z,2022-09-07T03:46:26Z,OWNER,"> Is it still nfortunately slow and tricky when playing with floats ? Not sure what you mean here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932716,sqlite-utils query --functions mechanism for registering extra functions, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229115501,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229115501,IC_kwDOCGYnMM5JQtBt,9599,simonw,2022-08-27T03:54:26Z,2022-08-27T03:54:26Z,OWNER,I can borrow Alex's example extension for the tests too: https://github.com/simonw/datasette/blob/ba35105eee2d3ba620e4f230028a02b2e2571df2/tests/ext.c,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229128617,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229128617,IC_kwDOCGYnMM5JQwOp,9599,simonw,2022-08-27T05:39:06Z,2022-08-27T05:39:06Z,OWNER,"That didn't run the tests as expected on Ubuntu: https://github.com/simonw/sqlite-utils/runs/8048229213?check_suite_focus=true ``` tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] SKIPPED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] SKIPPED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] SKIPPED [ 19%] ``` That should have worked because of this: https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/.github/workflows/test.yml#L38-L41 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229128950,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229128950,IC_kwDOCGYnMM5JQwT2,9599,simonw,2022-08-27T05:42:04Z,2022-08-27T05:42:04Z,OWNER,"Extension looks to be correctly compiled: ``` Run (cd tests && gcc ext.c -fPIC -shared -o ext.so && ls -lah) total 412K drwxr-xr-x 3 runner docker 4.0K Aug 27 05:40 . drwxr-xr-x 8 runner docker 4.0K Aug 27 05:40 .. -rw-r--r-- 1 runner docker 0 Aug 27 05:40 __init__.py -rw-r--r-- 1 runner docker 760 Aug 27 05:40 conftest.py -rw-r--r-- 1 runner docker 1.6K Aug 27 05:40 ext.c -rwxr-xr-x 1 runner docker 16K Aug 27 05:40 ext.so drwxr-xr-x 2 runner docker 4.0K Aug 27 05:40 sniff ``` So why doesn't this check pass? https://github.com/simonw/sqlite-utils/blob/3f694e51a0212a297f663e2fc7f9cdad8775a550/tests/test_cli.py#L25-L30","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130158,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229130158,IC_kwDOCGYnMM5JQwmu,9599,simonw,2022-08-27T05:52:46Z,2022-08-27T05:52:46Z,OWNER,"To run the tests locally on my Mac I needed to run Ubuntu in Docker (for the compilation step). This worked: docker run -it -v ""$(pwd):/app"" ubuntu:latest /bin/bash Then: ``` apt-get update && apt-get install python3-pip gcc libsqlite3-dev -y cd /app/tests gcc ext.c -fPIC -shared -o ext.so cd /app pip3 install -e '.[test]' pytest -k test_load_extension ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130242,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229130242,IC_kwDOCGYnMM5JQwoC,9599,simonw,2022-08-27T05:53:31Z,2022-08-27T05:53:31Z,OWNER,"Tests passed in CI against Ubuntu: ``` tests/test_cli.py::test_load_extension[None-should_pass0-should_fail0] PASSED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_b_init-b-should_fail1] PASSED [ 19%] tests/test_cli.py::test_load_extension[sqlite3_ext_c_init-c-should_fail2] PASSED [ 19%] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/470#issuecomment-1229130422,https://api.github.com/repos/simonw/sqlite-utils/issues/470,1229130422,IC_kwDOCGYnMM5JQwq2,9599,simonw,2022-08-27T05:55:01Z,2022-08-27T05:55:01Z,OWNER,cc @asg017 since this feature was inspired by his work on Datasette.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352932038,Upgrade `--load-extension` to accept entrypoints like Datasette, https://github.com/simonw/sqlite-utils/issues/469#issuecomment-1229120779,https://api.github.com/repos/simonw/sqlite-utils/issues/469,1229120779,IC_kwDOCGYnMM5JQuUL,9599,simonw,2022-08-27T04:30:48Z,2022-08-27T04:30:48Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1352931464,sqlite-utils rows --order option, https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229207725,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229207725,IC_kwDOCGYnMM5JRDit,9599,simonw,2022-08-27T14:55:32Z,2022-08-27T22:03:50Z,OWNER,"The main challenge here is coming up with comprehensive tests. The cases I need to cover are from this block of code: https://github.com/simonw/sqlite-utils/blob/c5f8a2eb1a81a18b52825cc649112f71fe419b12/sqlite_utils/db.py#L1468-L1480","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229276554,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229276554,IC_kwDOCGYnMM5JRUWK,9599,simonw,2022-08-27T22:05:40Z,2022-08-27T22:05:51Z,OWNER,For the moment I'm not going to pay attention to `foreign_keys` changes - I will note that these are not modified in the documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229279539,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229279539,IC_kwDOCGYnMM5JRVEz,9599,simonw,2022-08-27T22:24:33Z,2022-08-27T22:24:33Z,OWNER,"Interesting challenge with `default_value`: I need to be able to tell if the default values passed to `.create()` differ from those in the database already. Introspecting that is a bit tricky: ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db[""blah""].create({""id"": int, ""name"": str}, not_null=(""name"",), defaults={""name"": ""bob""})
>>> db[""blah""].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=""'bob'"", is_pk=0)] ``` Note how a default value of the Python string `bob` is represented in the results of `PRAGMA table_info()` as `default_value=""'bob'""` - it's got single quotes added to it! So comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229284539,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229284539,IC_kwDOCGYnMM5JRWS7,9599,simonw,2022-08-27T22:54:07Z,2022-08-27T22:55:01Z,OWNER,"Just needs documentation now, which can go here: https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/pull/468#issuecomment-1229303691,https://api.github.com/repos/simonw/sqlite-utils/issues/468,1229303691,IC_kwDOCGYnMM5JRa-L,9599,simonw,2022-08-27T23:13:06Z,2022-08-27T23:13:06Z,OWNER,"Documentation: - https://sqlite-utils--468.org.readthedocs.build/en/468/python-api.html#explicitly-creating-a-table - https://sqlite-utils--468.org.readthedocs.build/en/468/cli.html#creating-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348294436,"db[table].create(..., transform=True) and create-table --transform", https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224264744,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224264744,IC_kwDOCGYnMM5I-Mwo,9599,simonw,2022-08-23T15:51:29Z,2022-08-23T15:53:29Z,OWNER,"Jeff suggested `db[table].(..., alter=True)` for this. ```python db[""urls""].create( { ""url"": str, ""crawled"": bool, ""body"": str, ""headers"": dict, ""status"": int, ""status_text"": str, }, pk=""url"", defaults={""crawled"": False}, if_not_exists=True, alter=True, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224268273,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224268273,IC_kwDOCGYnMM5I-Nnx,9599,simonw,2022-08-23T15:54:24Z,2022-08-23T15:54:24Z,OWNER,I'm not crazy about having to pass both `alter=True` and `if_not_exists=True` - maybe `alter` should imply `if_not_exists`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224271324,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224271324,IC_kwDOCGYnMM5I-OXc,9599,simonw,2022-08-23T15:56:58Z,2022-08-23T15:56:58Z,OWNER,"Should passing `alter=True` also drop any columns that aren't included in the new table structure? It could even spot column types that aren't correct and fix those. Is that consistent with the expectations set by how `alter=True` works elsewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224272854,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224272854,IC_kwDOCGYnMM5I-OvW,9599,simonw,2022-08-23T15:58:14Z,2022-08-23T15:58:14Z,OWNER,Could call it `ensure=True` here if it works differently enough from `alter=True` that the behavior could be confusing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224278280,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224278280,IC_kwDOCGYnMM5I-QEI,9599,simonw,2022-08-23T16:02:07Z,2022-08-23T16:02:07Z,OWNER,"Thinking about this more, I think `alter=True` is a good name for this option even if it does more than the same option on `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224280225,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224280225,IC_kwDOCGYnMM5I-Qih,9599,simonw,2022-08-23T16:03:33Z,2022-08-23T16:03:33Z,OWNER,"Maybe there should be a separate `table.alter(...)` method that does the actual work here, with `.create(..., alter=True)` as syntactic sugar for triggering that if the table exists already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224283367,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224283367,IC_kwDOCGYnMM5I-RTn,9599,simonw,2022-08-23T16:05:55Z,2022-08-23T16:05:55Z,OWNER,"... but that's what the `table.transform(...)` method does already! So maybe this is actually a `transform=True` parameter to `create()` that triggers `table.transform(...)` if necessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224385575,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224385575,IC_kwDOCGYnMM5I-qQn,9599,simonw,2022-08-23T17:19:00Z,2022-08-23T17:19:00Z,OWNER,"Initial prototype: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 18a442a..03fd345 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -875,6 +875,7 @@ class Database: hash_id_columns: Optional[Iterable[str]] = None, extracts: Optional[Union[Dict[str, str], List[str]]] = None, if_not_exists: bool = False, + transform: bool = False, ) -> ""Table"": """""" Create a table with the specified name and the specified ``{column_name: type}`` columns. @@ -892,7 +893,39 @@ class Database: :param hash_id_columns: List of columns to be used when calculating the hash ID for a row :param extracts: List or dictionary of columns to be extracted during inserts, see :ref:`python_api_extracts` :param if_not_exists: Use ``CREATE TABLE IF NOT EXISTS`` - """""" + :param transform: If table already exists, transform it to fit the specified schema + """""" + # Transform table to match the new definition if table already exists: + if transform and self[name].exists(): + # First add missing columns and columns to drop + existing_columns = self[name].columns_dict + missing_columns = dict( + (col_name, col_type) + for col_name, col_type in columns.items() + if col_name not in existing_columns + ) + columns_to_drop = [ + column for column in existing_columns if column not in columns + ] + if missing_columns: + for col_name, col_type in missing_columns.items(): + self[name].add_column(col_name, col_type) + # Do we need to reset the column order? + column_order = None + if list(existing_columns) != list(columns): + column_order = list(columns) + # Only run .transform() if there is something to do + # TODO: this misses changes like pk= without also column changes + if columns_to_drop or missing_columns or column_order: + self[name].transform( + types=columns, + drop=columns_to_drop, + column_order=column_order, + not_null=not_null, + defaults=defaults, + pk=pk, + ) + return cast(Table, self[name]) sql = self.create_table_sql( name=name, columns=columns, @@ -1477,6 +1510,7 @@ class Table(Queryable): hash_id_columns: Optional[Iterable[str]] = None, extracts: Optional[Union[Dict[str, str], List[str]]] = None, if_not_exists: bool = False, + transform: bool = False, ) -> ""Table"": """""" Create a table with the specified columns. @@ -1508,6 +1542,7 @@ class Table(Queryable): hash_id_columns=hash_id_columns, extracts=extracts, if_not_exists=if_not_exists, + transform=transform, ) return self ``` Needs more thought about how things like just a change to `pk=` should work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224386951,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224386951,IC_kwDOCGYnMM5I-qmH,9599,simonw,2022-08-23T17:20:07Z,2022-08-23T17:20:07Z,OWNER,"Example of that prototype working: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].create({""id"": int, ""name"": str}, pk=""id"")
>>> db[""dogs""].create({""id"": int, ""name"": str, ""age"": int}, pk=""id"", transform=True)
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224388810,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1224388810,IC_kwDOCGYnMM5I-rDK,9599,simonw,2022-08-23T17:21:16Z,2022-08-23T17:21:16Z,OWNER,Also needs comprehensive tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229206306,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1229206306,IC_kwDOCGYnMM5JRDMi,9599,simonw,2022-08-27T14:47:04Z,2022-08-27T14:47:04Z,OWNER,I could add a `--transform` option to `sqlite-utils create-table` too.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1229311612,https://api.github.com/repos/simonw/sqlite-utils/issues/467,1229311612,IC_kwDOCGYnMM5JRc58,9599,simonw,2022-08-27T23:19:41Z,2022-08-27T23:19:41Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table - https://sqlite-utils.datasette.io/en/latest/cli.html#creating-tables ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1348169997,Mechanism for ensuring a table has all the columns, https://github.com/simonw/sqlite-utils/pull/465#issuecomment-1218663096,https://api.github.com/repos/simonw/sqlite-utils/issues/465,1218663096,IC_kwDOCGYnMM5Io1K4,9599,simonw,2022-08-17T23:37:51Z,2022-08-17T23:37:51Z,OWNER,Source links work on the preview: https://sqlite-utils--465.org.readthedocs.build/en/465/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1342357149,beanbag-docutils>=2.0, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214184512,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214184512,IC_kwDOCGYnMM5IXvxA,9599,simonw,2022-08-13T16:20:41Z,2022-08-13T16:20:41Z,OWNER,"https://github.com/beanbaginc/beanbag-docutils/blob/master/beanbag_docutils/sphinx/ext/github.py via https://twitter.com/chipx86/status/1558271211901050880 looks like a good option. It works using this Sphinx mechanism: https://www.sphinx-doc.org/en/master/usage/extensions/linkcode.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214184631,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214184631,IC_kwDOCGYnMM5IXvy3,9599,simonw,2022-08-13T16:21:40Z,2022-08-13T16:21:40Z,OWNER,"I tried it out and ran into a bug, which I've reported and submitted a PR for: - https://github.com/beanbaginc/beanbag-docutils/pull/1 - https://github.com/beanbaginc/beanbag-docutils/pull/2 I'm going to ship a version that uses my fork of the repo here: https://github.com/simonw/beanbag-docutils/tree/bytes-in-url","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214187919,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214187919,IC_kwDOCGYnMM5IXwmP,9599,simonw,2022-08-13T16:46:01Z,2022-08-13T16:49:36Z,OWNER,"Source links are now live here: https://sqlite-utils.datasette.io/en/latest/reference.html ![CleanShot 2022-08-13 at 09 49 02@2x](https://user-images.githubusercontent.com/9599/184503393-6559ab2b-cd02-4c85-98b9-62d10b3bdab8.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214188615,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214188615,IC_kwDOCGYnMM5IXwxH,9599,simonw,2022-08-13T16:50:16Z,2022-08-13T16:50:16Z,OWNER,Extracted a TIL: https://til.simonwillison.net/python/setup-py-from-url,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/issues/464#issuecomment-1214199975,https://api.github.com/repos/simonw/sqlite-utils/issues/464,1214199975,IC_kwDOCGYnMM5IXzin,9599,simonw,2022-08-13T18:09:02Z,2022-08-13T18:09:02Z,OWNER,I may be able to do this without that extra dependency: https://www.sphinx-doc.org/en/master/usage/extensions/linkcode.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1338001039,Link from documentation to source code, https://github.com/simonw/sqlite-utils/pull/463#issuecomment-1218610320,https://api.github.com/repos/simonw/sqlite-utils/issues/463,1218610320,IC_kwDOCGYnMM5IooSQ,9599,simonw,2022-08-17T23:11:07Z,2022-08-17T23:11:07Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1334416486,Use Read the Docs action v1, https://github.com/simonw/sqlite-utils/pull/462#issuecomment-1203207043,https://api.github.com/repos/simonw/sqlite-utils/issues/462,1203207043,IC_kwDOCGYnMM5Ht3uD,9599,simonw,2022-08-02T20:57:30Z,2022-08-02T20:57:30Z,OWNER,I created a new link for this - https://discord.gg/Ass7bCAMDw - which should drop people straight into the `#sqlite-utils` channel in the Datasette Discord.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326391841,Discord badge, https://github.com/simonw/sqlite-utils/issues/461#issuecomment-1203168339,https://api.github.com/repos/simonw/sqlite-utils/issues/461,1203168339,IC_kwDOCGYnMM5HtuRT,9599,simonw,2022-08-02T20:12:14Z,2022-08-02T20:12:14Z,OWNER,"The neat thing about these is that they can be embedded directly in a GitHub README. Not sure if they show up on PyPI project pages, though I expect that they would.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326349129,Consider including animated SVG console demos, https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1202947222,https://api.github.com/repos/simonw/sqlite-utils/issues/460,1202947222,IC_kwDOCGYnMM5Hs4SW,9599,simonw,2022-08-02T16:20:47Z,2022-08-02T16:20:47Z,OWNER,"That preview link it added didn't work, maybe because I have a custom domain setup? It linked to: https://readthedocs-preview--460.org.readthedocs.build/en/460/ - which 404s Here's the correct link: https://sqlite-utils--460.org.readthedocs.build/en/460/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326087800,Cross-link CLI to Python docs, https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1203190312,https://api.github.com/repos/simonw/sqlite-utils/issues/460,1203190312,IC_kwDOCGYnMM5Htzoo,9599,simonw,2022-08-02T20:36:58Z,2022-08-02T20:36:58Z,OWNER,"> That preview link it added didn't work, maybe because I have a custom domain setup? Entirely my fault, fixed here :https://github.com/simonw/sqlite-utils/commit/98a28cbfe6cea67f6334b42b74f35b0ddd309561","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326087800,Cross-link CLI to Python docs, https://github.com/simonw/sqlite-utils/pull/460#issuecomment-1220010289,https://api.github.com/repos/simonw/sqlite-utils/issues/460,1220010289,IC_kwDOCGYnMM5It-Ex,9599,simonw,2022-08-18T21:58:00Z,2022-08-18T21:58:00Z,OWNER,I'm going to merge this as-is and add more links separately.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1326087800,Cross-link CLI to Python docs, https://github.com/simonw/sqlite-utils/issues/46#issuecomment-514273140,https://api.github.com/repos/simonw/sqlite-utils/issues/46,514273140,MDEyOklzc3VlQ29tbWVudDUxNDI3MzE0MA==,9599,simonw,2019-07-23T15:57:53Z,2019-07-23T15:57:53Z,OWNER,"This will play very well with the new ability to specify these arguments in the class constructor, like so: ```python table = db.table(""trees"", extracts={""species_id"": ""Species""}) table.insert_all([{ ""id"": 1, ""species_id"": ""Oak"", }, { ""id"": 2, ""species_id"": ""Oak"", }, { ""id"": 3, ""species_id"": ""Palm"", }])","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471780443,extracts= option for insert/update/etc, https://github.com/simonw/sqlite-utils/issues/46#issuecomment-593118471,https://api.github.com/repos/simonw/sqlite-utils/issues/46,593118471,MDEyOklzc3VlQ29tbWVudDU5MzExODQ3MQ==,9599,simonw,2020-03-01T16:53:40Z,2020-03-01T16:53:40Z,OWNER,That seems sensible to me. Opening a ticket for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471780443,extracts= option for insert/update/etc, https://github.com/simonw/sqlite-utils/issues/458#issuecomment-1197509096,https://api.github.com/repos/simonw/sqlite-utils/issues/458,1197509096,IC_kwDOCGYnMM5HYIno,9599,simonw,2022-07-28T00:14:22Z,2022-07-28T00:14:22Z,OWNER,Updated documentation: https://github.com/simonw/sqlite-utils/blob/1491b66dd7439dd87cd5cd4c4684f46eb3c5751b/docs/python-api.rst#registering-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1320243134,Support custom names for registered functions, https://github.com/simonw/sqlite-utils/pull/457#issuecomment-1197108865,https://api.github.com/repos/simonw/sqlite-utils/issues/457,1197108865,IC_kwDOCGYnMM5HWm6B,9599,simonw,2022-07-27T17:58:10Z,2022-07-27T17:58:10Z,OWNER,Here's the build from the PR: https://sqlite-utils--457.org.readthedocs.build/en/457/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1319881016,Link to installation instructions, https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1189698289,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1189698289,IC_kwDOCGYnMM5G6Vrx,9599,simonw,2022-07-20T01:13:47Z,2022-07-20T01:13:47Z,OWNER,"Really interesting idea! Can you flesh out what this might look like, maybe with a before and after example schema?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command, https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190578687,https://api.github.com/repos/simonw/sqlite-utils/issues/456,1190578687,IC_kwDOCGYnMM5G9sn_,9599,simonw,2022-07-20T17:50:50Z,2022-07-20T17:50:50Z,OWNER,Wow https://tidyr.tidyverse.org/articles/pivot.html is _fascinating_.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1310243385,feature request: pivot command, https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1190686273,https://api.github.com/repos/simonw/sqlite-utils/issues/455,1190686273,IC_kwDOCGYnMM5G-G5B,9599,simonw,2022-07-20T19:46:15Z,2022-07-20T19:46:15Z,OWNER,Can you add a new test for this? Something derived from the example in #423 would work great.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1309542173,"in extract code, check equality with IS instead of = for nulls", https://github.com/simonw/sqlite-utils/pull/455#issuecomment-1229205990,https://api.github.com/repos/simonw/sqlite-utils/issues/455,1229205990,IC_kwDOCGYnMM5JRDHm,9599,simonw,2022-08-27T14:44:45Z,2022-08-27T14:44:45Z,OWNER,"I did not know `is` worked against strings and integers and booleans! https://latest.datasette.io/_memory?sql=select+1+is+1%2C+2.3+is+2.3%2C+true+is+true%2C+true+is+false%2C+%27dog%27+is+%27dog%27%2C+%27dog%27+is+%27cat%27%2C+null+is+null%2C+null+is+not+null ```sql select 1 is 1, 2.3 is 2.3, true is true, true is false, 'dog' is 'dog', 'dog' is 'cat', null is null, null is not null ``` | 1 is 1 | 2.3 is 2.3 | true is true | true is false | 'dog' is 'dog' | 'dog' is 'cat' | null is null | null is not null | |----------|--------------|----------------|-----------------|------------------|------------------|----------------|--------------------| | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1309542173,"in extract code, check equality with IS instead of = for nulls", https://github.com/simonw/sqlite-utils/issues/454#issuecomment-1185971600,https://api.github.com/repos/simonw/sqlite-utils/issues/454,1185971600,IC_kwDOCGYnMM5GsH2Q,9599,simonw,2022-07-15T21:48:23Z,2022-07-15T21:48:23Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#duplicating-tables - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#duplicate","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1306548397,CLI command for duplicating tables, https://github.com/simonw/sqlite-utils/issues/453#issuecomment-1185974145,https://api.github.com/repos/simonw/sqlite-utils/issues/453,1185974145,IC_kwDOCGYnMM5GsIeB,9599,simonw,2022-07-15T21:52:18Z,2022-07-15T21:52:18Z,OWNER,"I should warn you that this isn't a supported API - I reserve the right to change how it works between release without a major version bump, because it's not part of the documented API surface. You'll be fine if you pin to exact versions of the library though! You may find this recently-documented function useful though: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file See: - #443 I'm going to close this issue for the moment, but if anyone wants to submit a PR that cleans up this I'll happily review it. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1303169663,'unclosed file' warning when using insert_upsert_implementation from Python, https://github.com/simonw/sqlite-utils/issues/451#issuecomment-1185975132,https://api.github.com/repos/simonw/sqlite-utils/issues/451,1185975132,IC_kwDOCGYnMM5GsItc,9599,simonw,2022-07-15T21:53:42Z,2022-07-15T21:53:42Z,OWNER,Documentation can go here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1298531653,Make sqlite_utils.utils.chunks a documented function, https://github.com/simonw/sqlite-utils/issues/451#issuecomment-1185987117,https://api.github.com/repos/simonw/sqlite-utils/issues/451,1185987117,IC_kwDOCGYnMM5GsLot,9599,simonw,2022-07-15T22:09:33Z,2022-07-15T22:09:33Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-chunks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1298531653,Make sqlite_utils.utils.chunks a documented function, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185982012,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185982012,IC_kwDOCGYnMM5GsKY8,9599,simonw,2022-07-15T22:02:59Z,2022-07-15T22:03:49Z,OWNER,"From https://sqlite-utils.datasette.io/en/latest/cli-reference.html Commands that currently have an `--ignore` flag, and their descriptions: - `insert`: Ignore records if pk already exists - `create-table`: If table already exists, do nothing - `add-foreign-key`: If foreign key already exists, do nothing - `drop-table`: *description is missing* - `create-view`: If view already exists, do nothing - `drop-view`: *description is missing* And `create-index` has: - `--if-not-exists`: Ignore if index already exists","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185983894,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185983894,IC_kwDOCGYnMM5GsK2W,9599,simonw,2022-07-15T22:06:29Z,2022-07-15T22:37:20Z,OWNER,"Commands that could have `--ignore` added to them: - [x] `enable-fts` - [x] `disable-fts` (not doing) - [x] `add-column` - [x] `add-foreign-keys` (decided not to do this) - [x] `duplicate` - [x] `add-geometry-column` (decided not to do this) - [x] `create-spatial-index` (decided not to do this)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185988277,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185988277,IC_kwDOCGYnMM5GsL61,9599,simonw,2022-07-15T22:11:52Z,2022-07-15T22:11:52Z,OWNER,"For `enable-fts` I should use `--replace` instead, since that matches how the Python API works: https://github.com/simonw/sqlite-utils/blob/9dd4cf891d9f4565019e030ddc712507ac87b998/sqlite_utils/db.py#L2088-L2106","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185988480,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185988480,IC_kwDOCGYnMM5GsL-A,9599,simonw,2022-07-15T22:12:20Z,2022-07-15T22:12:20Z,OWNER,`disable-fts` already fails silently if the table doesn't have FTS setup.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185993791,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185993791,IC_kwDOCGYnMM5GsNQ_,9599,simonw,2022-07-15T22:23:02Z,2022-07-15T22:23:02Z,OWNER,"`create-spatial-index` is tricky. For consistency with `create-index` it should really use `--if-not-exists` - but under the hood it's not using `IF NOT EXISTS` because those indexes are created using `select CreateSpatialIndex(...)`. Really this highlights that `--if-not-exists` is a bad name for that option on `create-index`. I'm tempted to add `--ignore` to `create-index` and mark `--if-not-exists` as deprecated - though it will still work because I don't want to release a major version bump.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185994282,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185994282,IC_kwDOCGYnMM5GsNYq,9599,simonw,2022-07-15T22:24:12Z,2022-07-15T22:24:12Z,OWNER,"I can do this for `--create-index` instead: ``` Options: --name TEXT Explicit name for the new index --unique Make this a unique index --if-not-exists, --ignore Ignore if index already exists --analyze Run ANALYZE after creating the index --load-extension TEXT SQLite extensions to load -h, --help Show this message and exit. ``` That supports both names for the option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1185997860,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1185997860,IC_kwDOCGYnMM5GsOQk,9599,simonw,2022-07-15T22:32:51Z,2022-07-15T22:32:51Z,OWNER,`add-foreign-keys` is enough of a power-feature that I'm happy not to add this there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1186001408,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1186001408,IC_kwDOCGYnMM5GsPIA,9599,simonw,2022-07-15T22:37:40Z,2022-07-15T22:37:40Z,OWNER,I'm going to skip `add-geometry-column` and `create-spatial-index` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/450#issuecomment-1186002019,https://api.github.com/repos/simonw/sqlite-utils/issues/450,1186002019,IC_kwDOCGYnMM5GsPRj,9599,simonw,2022-07-15T22:39:09Z,2022-07-15T22:39:09Z,OWNER,Here are all of the changes I made in this issue: https://github.com/simonw/sqlite-utils/compare/e10536c7f59abbb785f092bf83c4ab94c00e31a3...b9a89a0f2c3559989efe65f25a6e1f8fa76fe8b0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292060682,Add --ignore option to more commands, https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1172902594,https://api.github.com/repos/simonw/sqlite-utils/issues/449,1172902594,IC_kwDOCGYnMM5F6RLC,9599,simonw,2022-07-02T13:55:51Z,2022-07-02T13:55:51Z,OWNER,"I like the idea of this as a feature - design could look like this: db[""my_table""].duplicate(""new_table"") Then for the CLI tool: sqlite-utils duplicate data.db my_table new_table Additional features that might be useful: - Duplicate across connection aliases - so you can copy a table to another database using the `duplicate` method or command - Ability to execute an arbitrary SQL query and save the results to a new table - basically a wrapper around `CREATE TABLE t AS SELECT ...`, again with alias support for copying to another database","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279863844,Utilities for duplicating tables and creating a table with the results of a query, https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1185970114,https://api.github.com/repos/simonw/sqlite-utils/issues/449,1185970114,IC_kwDOCGYnMM5GsHfC,9599,simonw,2022-07-15T21:46:13Z,2022-07-15T21:46:13Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#duplicating-tables - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.duplicate","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279863844,Utilities for duplicating tables and creating a table with the results of a query, https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1186002560,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1186002560,IC_kwDOCGYnMM5GsPaA,9599,simonw,2022-07-15T22:40:40Z,2022-07-15T22:40:40Z,OWNER,"This is very strange. GitHub Actions CI here runs against Windows and installs OK. Marking this as ""Help wanted"" to see if anyone can figure out what's going on here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1539109816,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1539109816,IC_kwDOCGYnMM5bvPO4,9599,simonw,2023-05-08T22:01:00Z,2023-05-08T22:01:00Z,OWNER,"This is being handled in: - #520","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1161857806,https://api.github.com/repos/simonw/sqlite-utils/issues/447,1161857806,IC_kwDOCGYnMM5FQIsO,9599,simonw,2022-06-21T14:55:51Z,2022-06-21T14:58:14Z,OWNER,"https://stackoverflow.com/a/44379513 suggests that the fix is: .. code-block:: text Or set this in `conf.py`: highlight_language = ""none"" I like that better - I don't like that all `::` blocks default to being treated as Python code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1278571700,Incorrect syntax highlighting in docs CLI reference, https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1161869859,https://api.github.com/repos/simonw/sqlite-utils/issues/447,1161869859,IC_kwDOCGYnMM5FQLoj,9599,simonw,2022-06-21T15:00:42Z,2022-06-21T15:00:42Z,OWNER,Deploying that to https://sqlite-utils.datasette.io/en/latest/cli-reference.html#insert,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1278571700,Incorrect syntax highlighting in docs CLI reference, https://github.com/simonw/sqlite-utils/issues/447#issuecomment-1162186856,https://api.github.com/repos/simonw/sqlite-utils/issues/447,1162186856,IC_kwDOCGYnMM5FRZBo,9599,simonw,2022-06-21T18:48:46Z,2022-06-21T18:48:46Z,OWNER,"That fixed it: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1278571700,Incorrect syntax highlighting in docs CLI reference, https://github.com/simonw/sqlite-utils/issues/446#issuecomment-1160798645,https://api.github.com/repos/simonw/sqlite-utils/issues/446,1160798645,IC_kwDOCGYnMM5FMGG1,9599,simonw,2022-06-20T19:55:34Z,2022-06-20T19:55:34Z,OWNER,"`just` now defaults to running the tests and linters. `just test` runs the tests - it can take arguments, e.g. `just test -k transform` `just lint` runs all of the linters. `just black` applies Black. In all case it assumes you are using `pipenv`, at least for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277328147,Use Just to automate running tests and linters locally, https://github.com/simonw/sqlite-utils/issues/446#issuecomment-1162234441,https://api.github.com/repos/simonw/sqlite-utils/issues/446,1162234441,IC_kwDOCGYnMM5FRkpJ,9599,simonw,2022-06-21T19:28:35Z,2022-06-21T19:28:35Z,OWNER,"`just -l` now does this: ``` % just -l Available recipes: black # Apply Black cog # Rebuild docs with cog default # Run tests and linters lint # Run linters: black, flake8, mypy, cog test *options # Run pytest with supplied options ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277328147,Use Just to automate running tests and linters locally, https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160763268,https://api.github.com/repos/simonw/sqlite-utils/issues/445,1160763268,IC_kwDOCGYnMM5FL9eE,9599,simonw,2022-06-20T19:09:21Z,2022-06-20T19:09:21Z,OWNER,Code to document: https://github.com/simonw/sqlite-utils/blob/3fbe8a784cc2f3fa0bfa8612fec9752ff9068a2b/sqlite_utils/utils.py#L318-L331,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277295119,`sqlite_utils.utils.TypeTracker` should be a documented API, https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160793114,https://api.github.com/repos/simonw/sqlite-utils/issues/445,1160793114,IC_kwDOCGYnMM5FMEwa,9599,simonw,2022-06-20T19:47:36Z,2022-06-20T19:47:36Z,OWNER,I also added inline documentation and types: https://github.com/simonw/sqlite-utils/blob/773f2b6b20622bb986984a1c3161d5b3aaa1046b/sqlite_utils/utils.py#L318-L360,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277295119,`sqlite_utils.utils.TypeTracker` should be a documented API, https://github.com/simonw/sqlite-utils/issues/445#issuecomment-1160794175,https://api.github.com/repos/simonw/sqlite-utils/issues/445,1160794175,IC_kwDOCGYnMM5FMFA_,9599,simonw,2022-06-20T19:49:02Z,2022-06-20T19:49:02Z,OWNER,"New documentation: - https://sqlite-utils.datasette.io/en/latest/python-api.html#detecting-column-types-using-typetracker - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-typetracker","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1277295119,`sqlite_utils.utils.TypeTracker` should be a documented API, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155804459,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155804459,IC_kwDOCGYnMM5E5C0r,9599,simonw,2022-06-14T23:28:18Z,2022-06-14T23:28:18Z,OWNER,"I think these become part of the `_import_options` list which is used in a few places: https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L765-L800","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155804591,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155804591,IC_kwDOCGYnMM5E5C2v,9599,simonw,2022-06-14T23:28:36Z,2022-06-14T23:28:36Z,OWNER,I'm going with `--extras-key` and `--ignore-extras` as the two new options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155815186,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155815186,IC_kwDOCGYnMM5E5FcS,9599,simonw,2022-06-14T23:48:16Z,2022-06-14T23:48:16Z,OWNER,"This is tricky to implement because of this code: https://github.com/simonw/sqlite-utils/blob/b8af3b96f5c72317cc8783dc296a94f6719987d9/sqlite_utils/cli.py#L938-L945 It's reconstructing each document using the known headers here: `docs = (dict(zip(headers, row)) for row in reader)` So my first attempt at this - the diff here - did not have the desired result: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 86eddfb..00b920b 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -6,7 +6,7 @@ import hashlib import pathlib import sqlite_utils from sqlite_utils.db import AlterError, BadMultiValues, DescIndex -from sqlite_utils.utils import maximize_csv_field_size_limit +from sqlite_utils.utils import maximize_csv_field_size_limit, _extra_key_strategy from sqlite_utils import recipes import textwrap import inspect @@ -797,6 +797,15 @@ _import_options = ( ""--encoding"", help=""Character encoding for input, defaults to utf-8"", ), + click.option( + ""--ignore-extras"", + is_flag=True, + help=""If a CSV line has more than the expected number of values, ignore the extras"", + ), + click.option( + ""--extras-key"", + help=""If a CSV line has more than the expected number of values put them in a list in this column"", + ), ) @@ -885,6 +894,8 @@ def insert_upsert_implementation( sniff, no_headers, encoding, + ignore_extras, + extras_key, batch_size, alter, upsert, @@ -909,6 +920,10 @@ def insert_upsert_implementation( raise click.ClickException(""--flatten cannot be used with --csv or --tsv"") if encoding and not (csv or tsv): raise click.ClickException(""--encoding must be used with --csv or --tsv"") + if ignore_extras and extras_key: + raise click.ClickException( + ""--ignore-extras and --extras-key cannot be used together"" + ) if pk and len(pk) == 1: pk = pk[0] encoding = encoding or ""utf-8-sig"" @@ -935,7 +950,9 @@ def insert_upsert_implementation( csv_reader_args[""delimiter""] = delimiter if quotechar: csv_reader_args[""quotechar""] = quotechar - reader = csv_std.reader(decoded, **csv_reader_args) + reader = _extra_key_strategy( + csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key + ) first_row = next(reader) if no_headers: headers = [""untitled_{}"".format(i + 1) for i in range(len(first_row))] @@ -1101,6 +1118,8 @@ def insert( sniff, no_headers, encoding, + ignore_extras, + extras_key, batch_size, alter, detect_types, @@ -1176,6 +1195,8 @@ def insert( sniff, no_headers, encoding, + ignore_extras, + extras_key, batch_size, alter=alter, upsert=False, @@ -1214,6 +1235,8 @@ def upsert( sniff, no_headers, encoding, + ignore_extras, + extras_key, alter, not_null, default, @@ -1254,6 +1277,8 @@ def upsert( sniff, no_headers, encoding, + ignore_extras, + extras_key, batch_size, alter=alter, upsert=True, @@ -1297,6 +1322,8 @@ def bulk( sniff, no_headers, encoding, + ignore_extras, + extras_key, load_extension, ): """""" @@ -1331,6 +1358,8 @@ def bulk( sniff=sniff, no_headers=no_headers, encoding=encoding, + ignore_extras=ignore_extras, + extras_key=extras_key, batch_size=batch_size, alter=False, upsert=False, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155815956,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155815956,IC_kwDOCGYnMM5E5FoU,9599,simonw,2022-06-14T23:49:56Z,2022-07-07T16:39:18Z,OWNER,"Yeah my initial implementation there makes no sense: ```python csv_reader_args = {""dialect"": dialect} if delimiter: csv_reader_args[""delimiter""] = delimiter if quotechar: csv_reader_args[""quotechar""] = quotechar reader = _extra_key_strategy( csv_std.reader(decoded, **csv_reader_args), ignore_extras, extras_key ) first_row = next(reader) if no_headers: headers = [""untitled_{}"".format(i + 1) for i in range(len(first_row))] reader = itertools.chain([first_row], reader) else: headers = first_row docs = (dict(zip(headers, row)) for row in reader) ``` Because my `_extra_key_strategy()` helper function is designed to work against `csv.DictReader` - not against `csv.reader()` which returns a sequence of lists, not a sequence of dictionaries. In fact, what's happening here is that `dict(zip(headers, row))` is ignoring anything in the row that doesn't correspond to a header: ```pycon >>> list(zip([""a"", ""b""], [1, 2, 3])) [('a', 1), ('b', 2)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/444#issuecomment-1155966234,https://api.github.com/repos/simonw/sqlite-utils/issues/444,1155966234,IC_kwDOCGYnMM5E5qUa,9599,simonw,2022-06-15T04:18:05Z,2022-06-15T04:18:05Z,OWNER,I'm going to push a branch with my not-yet-working code (which does at least include a test).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1271426387,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool, https://github.com/simonw/sqlite-utils/issues/443#issuecomment-1155672522,https://api.github.com/repos/simonw/sqlite-utils/issues/443,1155672522,IC_kwDOCGYnMM5E4inK,9599,simonw,2022-06-14T20:18:58Z,2022-06-14T20:18:58Z,OWNER,New documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1269998342,Make `utils.rows_from_file()` a documented API, https://github.com/simonw/sqlite-utils/issues/443#issuecomment-1160794604,https://api.github.com/repos/simonw/sqlite-utils/issues/443,1160794604,IC_kwDOCGYnMM5FMFHs,9599,simonw,2022-06-20T19:49:37Z,2022-06-20T19:49:37Z,OWNER,Also now shows up here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-rows-from-file,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1269998342,Make `utils.rows_from_file()` a documented API, https://github.com/simonw/sqlite-utils/issues/442#issuecomment-1155714131,https://api.github.com/repos/simonw/sqlite-utils/issues/442,1155714131,IC_kwDOCGYnMM5E4sxT,9599,simonw,2022-06-14T21:07:50Z,2022-06-14T21:07:50Z,OWNER,"Here's the commit where I added that originally, including a test: https://github.com/simonw/sqlite-utils/commit/1a93b72ba710ea2271eaabc204685a27d2469374","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1269886084,`maximize_csv_field_size_limit()` utility function, https://github.com/simonw/sqlite-utils/issues/442#issuecomment-1155748444,https://api.github.com/repos/simonw/sqlite-utils/issues/442,1155748444,IC_kwDOCGYnMM5E41Jc,9599,simonw,2022-06-14T21:55:15Z,2022-06-14T21:55:15Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#setting-the-maximum-csv-field-size-limit,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1269886084,`maximize_csv_field_size_limit()` utility function, https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1154373361,https://api.github.com/repos/simonw/sqlite-utils/issues/441,1154373361,IC_kwDOCGYnMM5Ezlbx,9599,simonw,2022-06-13T20:01:25Z,2022-06-13T20:01:25Z,OWNER,"Yeah, at the moment the best way to do this is with `search_sql()`, but you're right it really isn't very intuitive. Here's how I would do this, using a CTE trick to combine the queries: ```python search_sql = db[""articles""].search_sql(columns=[""title"", ""author""])) sql = f"""""" with search_results as ({search_sql}) select * from search_results where owner = :owner """""" results = db.query(sql, {""query"": ""my search query"", ""owner"": ""my owner""}) ``` I'm not sure if `sqlite-utils` should ever evolve to provide a better way of doing this kind of thing to be honest - if it did, it would turn into more of an ORM. Something like [PeeWee](http://docs.peewee-orm.com/en/latest/) may be a better option here.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1257724585,Combining `rows_where()` and `search()` to limit which rows are searched, https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155421299,https://api.github.com/repos/simonw/sqlite-utils/issues/441,1155421299,IC_kwDOCGYnMM5E3lRz,9599,simonw,2022-06-14T16:23:52Z,2022-06-14T16:23:52Z,OWNER,Actually I have a thought for something that could help here: I could add a mechanism for inserting additional where filters and parameters into that `.search()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1257724585,Combining `rows_where()` and `search()` to limit which rows are searched, https://github.com/simonw/sqlite-utils/issues/441#issuecomment-1155750270,https://api.github.com/repos/simonw/sqlite-utils/issues/441,1155750270,IC_kwDOCGYnMM5E41l-,9599,simonw,2022-06-14T21:57:57Z,2022-06-14T21:57:57Z,OWNER,I added `where=` and `where_args=` parameters to that `.search()` method - updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1257724585,Combining `rows_where()` and `search()` to limit which rows are searched, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154385916,IC_kwDOCGYnMM5Ezof8,9599,simonw,2022-06-13T20:15:49Z,2022-06-13T20:15:49Z,OWNER,`rows_from_file()` isn't part of the documented API but maybe it should be!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154386795,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154386795,IC_kwDOCGYnMM5Ezotr,9599,simonw,2022-06-13T20:16:53Z,2022-06-13T20:16:53Z,OWNER,"Steps to demonstrate that `sqlite-utils insert` is not affected: ```bash curl -o artsdatabanken.csv https://artsdatabanken.no/Fab2018/api/export/csv sqlite-utils insert arts.db artsdatabanken artsdatabanken.csv --sniff --csv --encoding utf-16le ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154387591,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154387591,IC_kwDOCGYnMM5Ezo6H,9599,simonw,2022-06-13T20:17:51Z,2022-06-13T20:17:51Z,OWNER,I don't understand why that works but calling `insert_all()` does not.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154396400,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154396400,IC_kwDOCGYnMM5EzrDw,9599,simonw,2022-06-13T20:28:25Z,2022-06-13T20:28:25Z,OWNER,"Fixing that `key` thing (to ignore any key that is `None`) revealed a new bug: ``` File ~/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py:376, in hash_record(record, keys) 373 if keys is not None: 374 to_hash = {key: record[key] for key in keys} 375 return hashlib.sha1( --> 376 json.dumps(to_hash, separators=("","", "":""), sort_keys=True, default=repr).encode( 377 ""utf8"" 378 ) 379 ).hexdigest() File ~/.pyenv/versions/3.8.2/lib/python3.8/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw) 232 if cls is None: 233 cls = JSONEncoder --> 234 return cls( 235 skipkeys=skipkeys, ensure_ascii=ensure_ascii, 236 check_circular=check_circular, allow_nan=allow_nan, indent=indent, 237 separators=separators, default=default, sort_keys=sort_keys, 238 **kw).encode(obj) File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:199, in JSONEncoder.encode(self, o) 195 return encode_basestring(o) 196 # This doesn't pass the iterator directly to ''.join() because the 197 # exceptions aren't as detailed. The list call should be roughly 198 # equivalent to the PySequence_Fast that ''.join() would do. --> 199 chunks = self.iterencode(o, _one_shot=True) 200 if not isinstance(chunks, (list, tuple)): 201 chunks = list(chunks) File ~/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py:257, in JSONEncoder.iterencode(self, o, _one_shot) 252 else: 253 _iterencode = _make_iterencode( 254 markers, self.default, _encoder, self.indent, floatstr, 255 self.key_separator, self.item_separator, self.sort_keys, 256 self.skipkeys, _one_shot) --> 257 return _iterencode(o, 0) TypeError: '<' not supported between instances of 'NoneType' and 'str' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154449442,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154449442,IC_kwDOCGYnMM5Ez4Ai,9599,simonw,2022-06-13T21:18:26Z,2022-06-13T21:20:12Z,OWNER,"Here are full steps to replicate the bug: ```python from urllib.request import urlopen import sqlite_utils db = sqlite_utils.Database(memory=True) with urlopen(""https://artsdatabanken.no/Fab2018/api/export/csv"") as fab: reader, other = sqlite_utils.utils.rows_from_file(fab, encoding=""utf-16le"") db[""fab2018""].insert_all(reader, pk=""Id"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154453319,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154453319,IC_kwDOCGYnMM5Ez49H,9599,simonw,2022-06-13T21:23:16Z,2022-06-13T21:23:16Z,OWNER,"Aha! I think I see what's happening here. Here's what `DictReader` does if one of the lines has too many items in it: ```pycon >>> import csv, io >>> list(csv.DictReader(io.StringIO(""id,name\n1,Cleo,nohead\n2,Barry""))) [{'id': '1', 'name': 'Cleo', None: ['nohead']}, {'id': '2', 'name': 'Barry'}] ``` See how that row with too many items gets this: `[{'id': '1', 'name': 'Cleo', None: ['nohead']}` That's a `None` for the key and (weirdly) a list containing the single item for the value! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154454127,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154454127,IC_kwDOCGYnMM5Ez5Jv,9599,simonw,2022-06-13T21:24:18Z,2022-06-13T21:24:18Z,OWNER,"That weird behaviour is documented here: https://docs.python.org/3/library/csv.html#csv.DictReader > If a row has more fields than fieldnames, the remaining data is put in a list and stored with the fieldname specified by *restkey* (which defaults to `None`). If a non-blank row has fewer fields than fieldnames, the missing values are filled-in with the value of *restval* (which defaults to `None`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154456183,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154456183,IC_kwDOCGYnMM5Ez5p3,9599,simonw,2022-06-13T21:26:55Z,2022-06-13T21:26:55Z,OWNER,"So I need to make a design decision here: what should `sqlite-utils` do with CSV files that have rows with more values than there are headings? Some options: - Ignore those extra fields entirely - silently drop that data. I'm not keen on this. - Throw an error. The library does this already, but the error is incomprehensible - it could turn into a useful, human-readable error instead. - Put the data in a JSON list in a column with a known name (`None` is not a valid column name, so not that). This could be something like `_restkey` or `_values_with_no_heading`. This feels like a better option, but I'd need to carefully pick a name for it - and come up with an answer for the question of what to do if the CSV file being important already uses that heading name for something else.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457028,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154457028,IC_kwDOCGYnMM5Ez53E,9599,simonw,2022-06-13T21:28:03Z,2022-06-13T21:28:03Z,OWNER,"Whatever I decide, I can implement it in `rows_from_file()`, maybe as an optional parameter - then decide how to call it from the `sqlite-utils insert` CLI (perhaps with a new option there too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154457893,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154457893,IC_kwDOCGYnMM5Ez6El,9599,simonw,2022-06-13T21:29:02Z,2022-06-13T21:29:02Z,OWNER,"Here's the current function signature for `rows_from_file()`: https://github.com/simonw/sqlite-utils/blob/26e6d2622c57460a24ffdd0128bbaac051d51a5f/sqlite_utils/utils.py#L174-L179","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154474482,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154474482,IC_kwDOCGYnMM5Ez-Hy,9599,simonw,2022-06-13T21:50:59Z,2022-06-13T21:51:24Z,OWNER,"Decision: I'm going to default to raising an exception if a row has too many values in it. You'll be able to pass `ignore_extras=True` to ignore those extra values, or pass `restkey=""the_rest""` to stick them in a list in the `restkey` column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154475454,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1154475454,IC_kwDOCGYnMM5Ez-W-,9599,simonw,2022-06-13T21:52:03Z,2022-06-13T21:52:03Z,OWNER,The exception will be called `RowError`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155310521,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155310521,IC_kwDOCGYnMM5E3KO5,9599,simonw,2022-06-14T14:58:50Z,2022-06-14T14:58:50Z,OWNER,"Interesting challenge in writing tests for this: if you give `csv.Sniffer` a short example with an invalid row in it sometimes it picks the wrong delimiter! id,name\r\n1,Cleo,oops It decided the delimiter there was `e`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155317293,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155317293,IC_kwDOCGYnMM5E3L4t,9599,simonw,2022-06-14T15:04:01Z,2022-06-14T15:04:01Z,OWNER,"I think that's unavoidable: it looks like `csv.Sniffer` only works if you feed it a CSV file with an equal number of values in each row, which is understandable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155350755,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155350755,IC_kwDOCGYnMM5E3UDj,9599,simonw,2022-06-14T15:25:18Z,2022-06-14T15:25:18Z,OWNER,"That broke `mypy`: `sqlite_utils/utils.py:229: error: Incompatible types in assignment (expression has type ""Iterable[Dict[Any, Any]]"", variable has type ""DictReader[str]"")`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155358637,IC_kwDOCGYnMM5E3V-t,9599,simonw,2022-06-14T15:31:34Z,2022-06-14T15:31:34Z,OWNER,"Getting this past `mypy` is really hard! ``` % mypy sqlite_utils sqlite_utils/utils.py:189: error: No overload variant of ""pop"" of ""MutableMapping"" matches argument type ""None"" sqlite_utils/utils.py:189: note: Possible overload variants: sqlite_utils/utils.py:189: note: def pop(self, key: str) -> str sqlite_utils/utils.py:189: note: def [_T] pop(self, key: str, default: Union[str, _T] = ...) -> Union[str, _T] ``` That's because of this line: row.pop(key=None) Which is legit here - we have a dictionary where one of the keys is `None` and we want to remove that key. But the baked in type is apparently `def pop(self, key: str) -> str`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155389614,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155389614,IC_kwDOCGYnMM5E3diu,9599,simonw,2022-06-14T15:54:03Z,2022-06-14T15:54:03Z,OWNER,"Filed an issue against `python/typeshed`: - https://github.com/python/typeshed/issues/8075","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155666672,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155666672,IC_kwDOCGYnMM5E4hLw,9599,simonw,2022-06-14T20:11:52Z,2022-06-14T20:11:52Z,OWNER,I'm going to rename `restkey` to `extras_key` for consistency with `ignore_extras`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155672675,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155672675,IC_kwDOCGYnMM5E4ipj,9599,simonw,2022-06-14T20:19:07Z,2022-06-14T20:19:07Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#reading-rows-from-a-file,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915,https://api.github.com/repos/simonw/sqlite-utils/issues/440,1155767915,IC_kwDOCGYnMM5E455r,9599,simonw,2022-06-14T22:22:27Z,2022-06-14T22:22:27Z,OWNER,I forgot to add equivalents of `extras_key=` and `ignore_extras=` to the CLI tool - will do that in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250629388,CSV files with too many values in a row cause errors, https://github.com/simonw/sqlite-utils/issues/44#issuecomment-514162499,https://api.github.com/repos/simonw/sqlite-utils/issues/44,514162499,MDEyOklzc3VlQ29tbWVudDUxNDE2MjQ5OQ==,9599,simonw,2019-07-23T11:01:45Z,2019-07-23T11:01:45Z,OWNER,"`id = table.lookup({""name"":""Cleo""})` - If table does not exist, create it with id, name where name is unique - If table does exist, add unique name column - If table and column exist, add unique constraint - throw error if impossible - now either insert the new row or return the existing ID","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471628483,Utilities for building lookup tables, https://github.com/simonw/sqlite-utils/issues/44#issuecomment-514162930,https://api.github.com/repos/simonw/sqlite-utils/issues/44,514162930,MDEyOklzc3VlQ29tbWVudDUxNDE2MjkzMA==,9599,simonw,2019-07-23T11:03:03Z,2019-07-23T11:03:16Z,OWNER,"I considered keyword arguments for this, but I am going with a dictionary instead - for two reasons: * leaves the option to add extra keyword arguments for further options later * supports column names that are not valid keyword arguments ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471628483,Utilities for building lookup tables, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155767202,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155767202,IC_kwDOCGYnMM5E45ui,9599,simonw,2022-06-14T22:21:10Z,2022-06-14T22:21:10Z,OWNER,"I can't figure out why that error is being swallowed like that. The most likely culprit was this code: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L1021-L1043 But I tried changing it like this: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 86eddfb..ed26fdd 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1023,6 +1023,7 @@ def insert_upsert_implementation( docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs ) except Exception as e: + raise if ( isinstance(e, sqlite3.OperationalError) and e.args ``` And your steps to reproduce still got to 49% and then failed silently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155769216,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155769216,IC_kwDOCGYnMM5E46OA,9599,simonw,2022-06-14T22:24:49Z,2022-06-14T22:25:06Z,OWNER,"I have a hunch that this crash may be caused by a CSV value which is too long, as addressed at the library level in: - #440 But not yet addressed in the CLI tool, see: - #444 Either way though, I really don't like that errors like this are swallowed!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155771462,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155771462,IC_kwDOCGYnMM5E46xG,9599,simonw,2022-06-14T22:28:38Z,2022-06-14T22:28:38Z,OWNER,"Maybe this isn't a CSV field value problem - I tried this patch and didn't seem to hit the new breakpoints: ```diff diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index d2ccc5f..f1b823a 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -204,13 +204,17 @@ def _extra_key_strategy( # DictReader adds a 'None' key with extra row values if None not in row: yield row - elif ignore_extras: + continue + else: + breakpoint() + if ignore_extras: # ignoring row.pop(none) because of this issue: # https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155358637 row.pop(None) # type: ignore yield row elif not extras_key: extras = row.pop(None) # type: ignore + breakpoint() raise RowError( ""Row {} contained these extra values: {}"".format(row, extras) ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155772244,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155772244,IC_kwDOCGYnMM5E469U,9599,simonw,2022-06-14T22:30:03Z,2022-06-14T22:30:03Z,OWNER,"Tried this: ``` % python -i $(which sqlite-utils) insert --csv --delimiter "";"" --encoding ""utf-16-le"" test test.db csv [------------------------------------] 0% [#################-------------------] 49% 00:00:01Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1072, in main ctx.exit() File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 692, in exit raise Exit(code) click.exceptions.Exit: 0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1090, in main sys.exit(e.exit_code) SystemExit: 0 >>> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155776023,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155776023,IC_kwDOCGYnMM5E474X,9599,simonw,2022-06-14T22:36:07Z,2022-06-14T22:36:07Z,OWNER,"Wait! The arguments in that are the wrong way round. This is correct: sqlite-utils insert --csv --delimiter "";"" --encoding ""utf-16-le"" test.db test csv It still outputs the following: [------------------------------------] 0% [#################-------------------] 49% 00:00:02% But it creates a `test.db` file that is 6.2MB. That database has 3141 rows in it: ``` % sqlite-utils tables test.db --counts -t table count ------- ------- test 3142 ``` I converted that `csv` file to utf-8 like so: iconv -f UTF-16LE -t UTF-8 csv > utf8.csv And it contains 3142 lines: ``` % wc -l utf8.csv 3142 utf8.csv ``` So my hunch here is that the problem is actually that the progress bar doesn't know how to correctly measure files in `utf-16-le` encoding!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155781399,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155781399,IC_kwDOCGYnMM5E49MX,9599,simonw,2022-06-14T22:45:41Z,2022-06-14T22:45:41Z,OWNER,TIL how to use `iconv`: https://til.simonwillison.net/linux/iconv,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155782835,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155782835,IC_kwDOCGYnMM5E49iz,9599,simonw,2022-06-14T22:48:22Z,2022-06-14T22:49:53Z,OWNER,"Here's the code that implements the progress bar in question: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/cli.py#L918-L932 It calls `file_progress()` which looks like this: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L159-L175 Which uses this: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/utils.py#L148-L156","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155784284,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155784284,IC_kwDOCGYnMM5E495c,9599,simonw,2022-06-14T22:51:03Z,2022-06-14T22:52:13Z,OWNER,"Yes, this is the problem. The progress bar length is set to the length in bytes of the file - `os.path.getsize(file.name)` - but it's then incremented by the length of each DECODED line in turn. So if the file is in `utf-16-le` (twice the size of `utf-8`) the progress bar will finish at 50%!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155788944,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155788944,IC_kwDOCGYnMM5E4_CQ,9599,simonw,2022-06-14T23:00:24Z,2022-06-14T23:00:24Z,OWNER,"The progress bar only works if the file-like object passed to it has a `fp.fileno()` that isn't 0 (for stdin) - that's how it detects that the file is something which it can measure the size of in order to show progress. If we know the file size in bytes AND we know the character encoding, can we change `UpdateWrapper` to update the number of bytes-per-character instead? I don't think so: I can't see a way of definitively saying ""for this encoding the number of bytes per character is X"" - and in fact I'm pretty sure that question doesn't even make sense since variable-length encodings exist. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155789101,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155789101,IC_kwDOCGYnMM5E4_Et,9599,simonw,2022-06-14T23:00:45Z,2022-06-14T23:00:45Z,OWNER,"I'm going to mark this as ""help wanted"" and leave it open. I'm glad that it's not actually a bug where errors get swallowed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/issues/439#issuecomment-1155953345,https://api.github.com/repos/simonw/sqlite-utils/issues/439,1155953345,IC_kwDOCGYnMM5E5nLB,9599,simonw,2022-06-15T03:53:43Z,2022-06-15T03:53:43Z,OWNER,"I tried fixing this by using `.tell()` to read the file position as I was iterating through it: ```diff diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py index d2ccc5f..29ad12e 100644 --- a/sqlite_utils/utils.py +++ b/sqlite_utils/utils.py @@ -149,10 +149,13 @@ class UpdateWrapper: def __init__(self, wrapped, update): self._wrapped = wrapped self._update = update + self._tell = wrapped.tell() def __iter__(self): for line in self._wrapped: - self._update(len(line)) + tell = self._wrapped.tell() + self._update(self._tell - tell) + self._tell = tell yield line ``` This did not work - I get this error: ``` File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 206, in _extra_key_strategy for row in reader: File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 156, in __iter__ tell = self._wrapped.tell() OSError: telling position disabled by next() call ``` It looks like you can't use `.tell()` during iteration: https://stackoverflow.com/questions/29618936/how-to-solve-oserror-telling-position-disabled-by-next-call","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1250495688,Misleading progress bar against utf-16-le CSV input, https://github.com/simonw/sqlite-utils/pull/437#issuecomment-1141488533,https://api.github.com/repos/simonw/sqlite-utils/issues/437,1141488533,IC_kwDOCGYnMM5ECbuV,9599,simonw,2022-05-30T21:32:36Z,2022-05-30T21:32:36Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1244294227,docs to dogs, https://github.com/simonw/sqlite-utils/issues/435#issuecomment-1133416698,https://api.github.com/repos/simonw/sqlite-utils/issues/435,1133416698,IC_kwDOCGYnMM5DjpD6,9599,simonw,2022-05-20T21:54:43Z,2022-05-20T21:54:43Z,OWNER,Done: https://sqlite-utils.datasette.io/en/latest/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243704847,Switch to Furo documentation theme, https://github.com/simonw/sqlite-utils/issues/435#issuecomment-1133417432,https://api.github.com/repos/simonw/sqlite-utils/issues/435,1133417432,IC_kwDOCGYnMM5DjpPY,9599,simonw,2022-05-20T21:56:10Z,2022-05-20T21:56:10Z,OWNER,"Before: ![sqlite-utils-datasette-io-en-stable-reference-html](https://user-images.githubusercontent.com/9599/169617623-457b4c01-3713-4e5c-b3c3-8574a2214238.png) After: ![sqlite-utils-datasette-io-en-latest-reference-html](https://user-images.githubusercontent.com/9599/169617666-ba388167-36b8-4a3e-be35-931ee2ab6b3b.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243704847,Switch to Furo documentation theme, https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155791109,https://api.github.com/repos/simonw/sqlite-utils/issues/434,1155791109,IC_kwDOCGYnMM5E4_kF,9599,simonw,2022-06-14T23:04:40Z,2022-06-14T23:04:40Z,OWNER,"Definitely a bug - thanks for the detailed write-up! You're right, the code at fault is here: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L2213-L2231","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243151184,`detect_fts()` identifies the wrong table if tables have names that are subsets of each other, https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155794149,https://api.github.com/repos/simonw/sqlite-utils/issues/434,1155794149,IC_kwDOCGYnMM5E5ATl,9599,simonw,2022-06-14T23:09:54Z,2022-06-14T23:09:54Z,OWNER,"A test that demonstrates the problem: ```python @pytest.mark.parametrize(""reverse_order"", (True, False)) def test_detect_fts_similar_tables(fresh_db, reverse_order): # https://github.com/simonw/sqlite-utils/issues/434 table1, table2 = (""demo"", ""demo2"") if reverse_order: table1, table2 = table2, table1 fresh_db[table1].insert({""title"": ""Hello""}).enable_fts( [""title""], fts_version=""FTS4"" ) fresh_db[table2].insert({""title"": ""Hello""}).enable_fts( [""title""], fts_version=""FTS4"" ) assert fresh_db[table1].detect_fts() == ""{}_fts"".format(table1) assert fresh_db[table2].detect_fts() == ""{}_fts"".format(table2) ``` The order matters - so this test currently passes in one direction and fails in the other: ``` > assert fresh_db[table2].detect_fts() == ""{}_fts"".format(table2) E AssertionError: assert 'demo2_fts' == 'demo_fts' E - demo_fts E + demo2_fts E ? + tests/test_introspect.py:53: AssertionError ========================================================================================= short test summary info ========================================================================================= FAILED tests/test_introspect.py::test_detect_fts_similar_tables[True] - AssertionError: assert 'demo2_fts' == 'demo_fts' =============================================================================== 1 failed, 1 passed, 855 deselected in 1.00s =============================================================================== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243151184,`detect_fts()` identifies the wrong table if tables have names that are subsets of each other, https://github.com/simonw/sqlite-utils/issues/434#issuecomment-1155801812,https://api.github.com/repos/simonw/sqlite-utils/issues/434,1155801812,IC_kwDOCGYnMM5E5CLU,9599,simonw,2022-06-14T23:23:32Z,2022-06-14T23:23:32Z,OWNER,"Since table names can be quoted like this: ```sql CREATE VIRTUAL TABLE ""searchable_fts"" USING FTS4 (text1, text2, [name with . and spaces], content=""searchable"") ``` OR like this: ```sql CREATE VIRTUAL TABLE ""searchable_fts"" USING FTS4 (text1, text2, [name with . and spaces], content=[searchable]) ``` This fix looks to be correct to me (copying from the updated `test_with_trace()` test): ```python ( ""SELECT name FROM sqlite_master\n"" "" WHERE rootpage = 0\n"" "" AND (\n"" "" sql LIKE :like\n"" "" OR sql LIKE :like2\n"" "" OR (\n"" "" tbl_name = :table\n"" "" AND sql LIKE '%VIRTUAL TABLE%USING FTS%'\n"" "" )\n"" "" )"", { ""like"": ""%VIRTUAL TABLE%USING FTS%content=[dogs]%"", ""like2"": '%VIRTUAL TABLE%USING FTS%content=""dogs""%', ""table"": ""dogs"", }, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1243151184,`detect_fts()` identifies the wrong table if tables have names that are subsets of each other, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1155749696,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1155749696,IC_kwDOCGYnMM5E41dA,9599,simonw,2022-06-14T21:57:05Z,2022-06-14T21:57:05Z,OWNER,Marking this as help wanted because I can't figure out how to replicate it!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1793273968,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1793273968,IC_kwDOCGYnMM5q4zBw,9599,simonw,2023-11-04T00:45:19Z,2023-11-04T00:45:19Z,OWNER,"Here's an animated GIF that demonstrates the bug: ![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/18e54b38-a466-4384-9162-114eb01526a2) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1793274350,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1793274350,IC_kwDOCGYnMM5q4zHu,9599,simonw,2023-11-04T00:46:30Z,2023-11-04T00:46:30Z,OWNER,"And a GIF of the fix after applying: - #598 ![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/35829aec-c9ac-4925-a8e6-ffe7c2ab0d96) ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor, https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155756742,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155756742,IC_kwDOCGYnMM5E43LG,9599,simonw,2022-06-14T22:05:38Z,2022-06-14T22:05:49Z,OWNER,"I don't like the idea of `table_names()` returning names of tables from connected databases as well, because it feels like it could lead to surprising behaviour - especially if those connected databases turn to have table names that are duplicated in the main connected database. It would be neat if functions like `.rows_where()` worked though. One thought would be to support something like this: ```python rows = db[""otherdb.tablename""].rows_where() ``` But... `.` is a valid character in a SQLite table name. So `""otherdb.tablename""` might ambiguously refer to a table called `tablename` in a connected database with the alias `otherdb`, OR a table in the current database with the name `otherdb.tablename`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155758664,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155758664,IC_kwDOCGYnMM5E43pI,9599,simonw,2022-06-14T22:07:50Z,2022-06-14T22:07:50Z,OWNER,"Another potential fix: add a `alias=` parameter to `rows_where()` and other similar methods. Then you could do this: ```python rows = db[""tablename""].rows_where(alias=""otherdb"") ``` This feels wrong to me: `db[""tablename""]` is the bit that is supposed to return a table object. Having part of what that table object is exist as a parameter to other methods is confusing. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155759857,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155759857,IC_kwDOCGYnMM5E437x,9599,simonw,2022-06-14T22:09:07Z,2022-06-14T22:09:07Z,OWNER,"Third option, and I think the one I like the best: ```python rows = db.table(""tablename"", alias=""otherdb"").rows_where(alias=""otherdb"") ``` The `db.table(tablename)` method already exists as an alternative to `db[tablename]`: https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-table-configuration ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155764064,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155764064,IC_kwDOCGYnMM5E449g,9599,simonw,2022-06-14T22:15:44Z,2022-06-14T22:15:44Z,OWNER,"Implementing this would be a pretty big change - initial instinct is that I'd need to introduce a `self.alias` property to `Queryable` (the subclass of `Table` and `View`) and a new `self.name_with_alias` getter which returns `alias.tablename` if `alias` is set to a not-None value. Then I'd need to rewrite every piece of code like this: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1161 To look like this instead: ```python sql = ""select {} from [{}]"".format(select, self.name_with_alias) ``` But some parts would be harder - for example: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L1227-L1231 Would have to know to query `alias.sqlite_master` instead. The cached table counts logic like this would need a bunch of changes too: https://github.com/simonw/sqlite-utils/blob/1b09538bc6c1fda773590f3e600993ef06591041/sqlite_utils/db.py#L644-L657","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/432#issuecomment-1155764428,https://api.github.com/repos/simonw/sqlite-utils/issues/432,1155764428,IC_kwDOCGYnMM5E45DM,9599,simonw,2022-06-14T22:16:21Z,2022-06-14T22:16:21Z,OWNER,"Initial idea of how the `.table()` method would change: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 7a06304..3ecb40b 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -474,11 +474,12 @@ class Database: self._tracer(sql, None) return self.conn.executescript(sql) - def table(self, table_name: str, **kwargs) -> Union[""Table"", ""View""]: + def table(self, table_name: str, alias: Optional[str] = None, **kwargs) -> Union[""Table"", ""View""]: """""" Return a table object, optionally configured with default options. :param table_name: Name of the table + :param alias: The database alias to use, if referring to a table in another connected database """""" klass = View if table_name in self.view_names() else Table return klass(self, table_name, **kwargs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1236693079,"Support `rows_where()`, `delete_where()` etc for attached alias databases", https://github.com/simonw/sqlite-utils/issues/431#issuecomment-1155753397,https://api.github.com/repos/simonw/sqlite-utils/issues/431,1155753397,IC_kwDOCGYnMM5E42W1,9599,simonw,2022-06-14T22:01:38Z,2022-06-14T22:01:38Z,OWNER,"Yeah, I think it would be neat if the library could support self-referential many-to-many in a nice way. I'm not sure about the `left_name/right_name` design though. Would it be possible to have this work as the user intends, by spotting that the other table name `""people""` matches the name of the current table? ```python db[""people""].insert({""name"": ""Mary""}, pk=""name"").m2m( ""people"", [{""name"": ""Michael""}, {""name"": ""Suzy""}], m2m_table=""parent_child"", pk=""name"" ) ``` The created table could look like this: ```sql CREATE TABLE [parent_child] ( [people_id_1] TEXT REFERENCES [people]([name]), [people_id_2] TEXT REFERENCES [people]([name]), PRIMARY KEY ([people_id_1], [people_id_2]) ) ``` I've not thought very hard about this, so the design I'm proposing here might not work. Are there other reasons people might wan the `left_name=` and `right_name=` parameters? If so then I'm much happier with those.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1227571375,Allow making m2m relation of a table to itself, https://github.com/simonw/sqlite-utils/issues/430#issuecomment-1155803262,https://api.github.com/repos/simonw/sqlite-utils/issues/430,1155803262,IC_kwDOCGYnMM5E5Ch-,9599,simonw,2022-06-14T23:26:11Z,2022-06-14T23:26:11Z,OWNER,"It looks like `PRAGMA temp_store` was the right option to use here: https://www.sqlite.org/pragma.html#pragma_temp_store `temp_store_directory` is listed as deprecated here: https://www.sqlite.org/pragma.html#pragma_temp_store_directory I'm going to turn this into a help-wanted documentation issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1224112817,Document how to use `PRAGMA temp_store` to avoid errors when running VACUUM against huge databases, https://github.com/simonw/sqlite-utils/issues/43#issuecomment-513481493,https://api.github.com/repos/simonw/sqlite-utils/issues/43,513481493,MDEyOklzc3VlQ29tbWVudDUxMzQ4MTQ5Mw==,9599,simonw,2019-07-20T16:34:54Z,2019-07-20T16:34:54Z,OWNER,"Here's the implementation in `create_table()`: https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L235-L249 And here's where `add_column()` does its thing: https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L547-L552","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691999,.add_column() doesn't match indentation of initial creation, https://github.com/simonw/sqlite-utils/issues/43#issuecomment-513481607,https://api.github.com/repos/simonw/sqlite-utils/issues/43,513481607,MDEyOklzc3VlQ29tbWVudDUxMzQ4MTYwNw==,9599,simonw,2019-07-20T16:36:34Z,2019-07-20T16:36:34Z,OWNER,"Hmm... is it even possible to fix this within sqlite-utils? Maybe this is a SQLite implementation detail. It's possible to update the schema itself - that's how we add missing foreign key constraints - but it seems a little bit of overkill to use the writable schema when adding a new column JUST to get prettier indentation: https://github.com/simonw/sqlite-utils/blob/c88f0a4d4617ff832aa728aac1da3500548c3137/sqlite_utils/db.py#L316-L326","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691999,.add_column() doesn't match indentation of initial creation, https://github.com/simonw/sqlite-utils/issues/43#issuecomment-514202279,https://api.github.com/repos/simonw/sqlite-utils/issues/43,514202279,MDEyOklzc3VlQ29tbWVudDUxNDIwMjI3OQ==,9599,simonw,2019-07-23T13:09:05Z,2019-07-23T13:09:05Z,OWNER,"Won't fix, I don't think this can be done in an elegant way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470691999,.add_column() doesn't match indentation of initial creation, https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115196863,https://api.github.com/repos/simonw/sqlite-utils/issues/429,1115196863,IC_kwDOCGYnMM5CeI2_,9599,simonw,2022-05-02T18:03:47Z,2022-05-02T18:52:42Z,OWNER,"I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html ```pycon >>> import micropip >>> await micropip.install(""https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl"") >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> list(db.query(""select 32443 + 55"")) [{'32443 + 55': 32498}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223177069,Depend on click-default-group-wheel, https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115197644,https://api.github.com/repos/simonw/sqlite-utils/issues/429,1115197644,IC_kwDOCGYnMM5CeJDM,9599,simonw,2022-05-02T18:04:28Z,2022-05-02T18:04:28Z,OWNER,I'm going to ship this straight away as `3.26.1`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1223177069,Depend on click-default-group-wheel, https://github.com/simonw/sqlite-utils/issues/428#issuecomment-1109190401,https://api.github.com/repos/simonw/sqlite-utils/issues/428,1109190401,IC_kwDOCGYnMM5CHOcB,9599,simonw,2022-04-26T01:05:29Z,2022-04-26T01:05:29Z,OWNER,Django makes extensive use of savepoints for nested transactions: https://docs.djangoproject.com/en/4.0/topics/db/transactions/#savepoints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1215216249,Research adding support for savepoints, https://github.com/simonw/sqlite-utils/issues/427#issuecomment-1172903355,https://api.github.com/repos/simonw/sqlite-utils/issues/427,1172903355,IC_kwDOCGYnMM5F6RW7,9599,simonw,2022-07-02T13:59:30Z,2022-07-02T13:59:30Z,OWNER,"This should work to avoid that: sqlite-utils convert \ ${dbfile} details dob \ 'r.parsedate(value)' \ --where ""dob != '*'""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1212701569,"sqlite-utils convert date parsing recipe complains about trying to parse ""*""", https://github.com/simonw/sqlite-utils/issues/426#issuecomment-1646681386,https://api.github.com/repos/simonw/sqlite-utils/issues/426,1646681386,IC_kwDOCGYnMM5iJl0q,9599,simonw,2023-07-22T22:13:22Z,2023-07-22T22:13:22Z,OWNER,I'm happy with how this works on https://sqlite-utils.datasette.io/en/stable/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1205687423,CLI docs should link to Python docs and vice versa, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098537000,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1098537000,IC_kwDOCGYnMM5Belgo,9599,simonw,2022-04-13T22:18:22Z,2022-04-13T22:18:22Z,OWNER,"I figured out a workaround in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531 The current `register(fn)` method looks like this: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L403 This alternative implementation worked in the environment where that failed: ```python def register(fn): name = fn.__name__ arity = len(inspect.signature(fn).parameters) if not replace and (name, arity) in self._registered_functions: return fn kwargs = {} done = False if deterministic: # Try this, but fall back if sqlite3.NotSupportedError try: self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True)) done = True except sqlite3.NotSupportedError: pass if not done: self.conn.create_function(name, arity, fn, **kwargs) self._registered_functions.add((name, arity)) return fn ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1098545390,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1098545390,IC_kwDOCGYnMM5Benju,9599,simonw,2022-04-13T22:34:52Z,2022-04-13T22:34:52Z,OWNER,"That broke Python 3.7 because it doesn't support `deterministic=True` even being passed: > function takes at most 3 arguments (4 given)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1101594549,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1101594549,IC_kwDOCGYnMM5BqP-1,9599,simonw,2022-04-18T17:36:14Z,2022-04-18T17:36:14Z,OWNER,"Releated: - #408","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher, https://github.com/simonw/sqlite-utils/issues/424#issuecomment-1098548090,https://api.github.com/repos/simonw/sqlite-utils/issues/424,1098548090,IC_kwDOCGYnMM5BeoN6,9599,simonw,2022-04-13T22:40:15Z,2022-04-13T22:40:15Z,OWNER,"New error: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].create({}) Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1465, in create self.db.create_table( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 885, in create_table sql = self.create_table_sql( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 771, in create_table_sql assert columns, ""Tables must have at least one column"" AssertionError: Tables must have at least one column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200866134,Better error message if you try to create a table with no columns, https://github.com/simonw/sqlite-utils/issues/422#issuecomment-1079406708,https://api.github.com/repos/simonw/sqlite-utils/issues/422,1079406708,IC_kwDOCGYnMM5AVnB0,9599,simonw,2022-03-25T20:23:21Z,2022-03-25T20:23:21Z,OWNER,"Fixing this would require a bump to 4.0 because it would break existing code. The alternative would be to introduce a new `ignore_nulls=True` parameter which users can change to `ignore_nulls=False`. Or come up with better wording for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181236173,Reconsider not running convert functions against null values, https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1079407962,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1079407962,IC_kwDOCGYnMM5AVnVa,9599,simonw,2022-03-25T20:25:10Z,2022-03-25T20:25:18Z,OWNER,"Can you share either your whole `global.db` table or a shrunk down example that illustrates the bug? My hunch is that you may have a table or column with a name that triggers the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098288158,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098288158,IC_kwDOCGYnMM5Bdowe,9599,simonw,2022-04-13T17:07:53Z,2022-04-13T17:07:53Z,OWNER,"I can't replicate the bug I'm afraid: ``` % wget ""https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true"" ... 2022-04-13 10:06:29 (8.97 MB/s) - ‘global_power_plant_database.csv?raw=true’ saved [8856038/8856038] % sqlite-utils insert global.db power_plants \ 'global_power_plant_database.csv?raw=true' --csv [------------------------------------] 0% [###################################-] 99% 00:00:00% % sqlite-utils indexes global.db --table table index_name seqno cid name desc coll key ------- ------------ ------- ----- ------ ------ ------ ----- % sqlite-utils extract global.db power_plants country country_long \ --table countries \ --fk-column country_id \ --rename country_long name % sqlite-utils indexes global.db --table table index_name seqno cid name desc coll key --------- -------------------------- ------- ----- ------- ------ ------ ----- countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098295517,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098295517,IC_kwDOCGYnMM5Bdqjd,9599,simonw,2022-04-13T17:16:20Z,2022-04-13T17:16:20Z,OWNER,"Aha! I was able to replicate the bug using your `Dockerfile` - thanks very much for providing that. ``` (app-root) sqlite-utils indexes global.db --table Error: near ""("": syntax error ``` (That wa sbefore I even ran the `extract` command.) To build your `Dockerfile` I copied it into an empty folder and ran the following: ``` wget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz docker build . -t centos-sqlite-utils docker run -it centos-sqlite-utils /bin/bash ``` This gave me a shell in which I could replicate the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098531354,IC_kwDOCGYnMM5BekIa,9599,simonw,2022-04-13T22:08:20Z,2022-04-13T22:08:20Z,OWNER,"OK I figured out what's going on here. First I added an extra `print(sql)` statement to the `indexes` command to see what SQL it was running: ``` (app-root) sqlite-utils indexes global.db --table select sqlite_master.name as ""table"", indexes.name as index_name, xinfo.* from sqlite_master join pragma_index_list(sqlite_master.name) indexes join pragma_index_xinfo(index_name) xinfo where sqlite_master.type = 'table' and xinfo.key = 1 Error: near ""("": syntax error ``` This made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version: ``` (app-root) sqlite3 SQLite version 3.36.0 2021-06-18 18:36:39 ``` That version should be fine - it's the one you compiled in the Dockerfile. Then I checked the version that `sqlite-utils` itself was using: ``` (app-root) sqlite-utils memory 'select sqlite_version()' [{""sqlite_version()"": ""3.7.17""}] ``` It's running SQLite 3.7.17! So the problem here is that the Python in that Docker image is running a very old version of SQLite. I tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked: ``` (app-root) python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())' ('3.7.17',) (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())' ('3.36.0',) ``` But when I try to run `sqlite-utils` like that I get an error: ``` (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db ... File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py"", line 1624, in query db.register_fts4_bm25() File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 412, in register_fts4_bm25 self.register_function(rank_bm25, deterministic=True) File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 408, in register_function register(fn) File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 401, in register self.conn.create_function(name, arity, fn, **kwargs) sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098532220,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098532220,IC_kwDOCGYnMM5BekV8,9599,simonw,2022-04-13T22:09:52Z,2022-04-13T22:09:52Z,OWNER,That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098535531,IC_kwDOCGYnMM5BelJr,9599,simonw,2022-04-13T22:15:48Z,2022-04-13T22:15:48Z,OWNER,"Trying this alternative implementation of the `register()` method: ```python def register(fn): name = fn.__name__ arity = len(inspect.signature(fn).parameters) if not replace and (name, arity) in self._registered_functions: return fn kwargs = {} done = False if deterministic: # Try this, but fall back if sqlite3.NotSupportedError try: self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True)) done = True except sqlite3.NotSupportedError: pass if not done: self.conn.create_function(name, arity, fn, **kwargs) self._registered_functions.add((name, arity)) return fn ``` With that fix, the following worked! ``` LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table table index_name seqno cid name desc coll key --------- -------------------------- ------- ----- ------- ------ ------ ----- countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098548931,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098548931,IC_kwDOCGYnMM5BeobD,9599,simonw,2022-04-13T22:41:59Z,2022-04-13T22:41:59Z,OWNER,"I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the `LD_PRELOAD` workaround above should work OK with the next release of `sqlite-utils`. Thanks for the detailed bug report!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI", https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078315922,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078315922,IC_kwDOCGYnMM5ARcuS,9599,simonw,2022-03-24T21:09:27Z,2022-03-24T21:09:27Z,OWNER,"Yeah, this is WAY harder than it should be. There's a clumsy workaround you could use which looks something like this: create a file `my_enchant.py` containing: ```python import enchant d = enchant.Dict(""en_US"") def check(word): return d.check(word) ``` Then run `sqlite-utils` like this: ``` PYTHONPATH=. cat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert 'my_enchant.check(value)' --import my_enchant ``` Except I tried that and it doesn't work! I don't know the right pattern for getting `--import` to work with modules in the same directory. So yeah, this is definitely a big feature gap.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078322301,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078322301,IC_kwDOCGYnMM5AReR9,9599,simonw,2022-03-24T21:10:52Z,2022-03-24T21:10:52Z,OWNER,"I can think of three ways forward: - Figure out a pattern that gets that local file import workaround to work - Add another option such as `--convert-init` that lets you pass code that will be executed once at the start - Come up with a pattern where the `--convert` code can run some initialization code and then return a function which will be called against each value I quite like the idea of that third option - I'm going to prototype it and see if I can work something out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078328774,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078328774,IC_kwDOCGYnMM5ARf3G,9599,simonw,2022-03-24T21:12:33Z,2022-03-24T21:12:33Z,OWNER,"Here's how the `_compile_code()` mechanism works at the moment: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L308-L342 At the end it does this: ```python return locals[""fn""] ``` So it's already building and then returning a function. The question is if there's a sensible way to allow people to further customize that function by executing some code first, in a way that's easy to explain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078343231,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078343231,IC_kwDOCGYnMM5ARjY_,9599,simonw,2022-03-24T21:16:10Z,2022-03-24T21:17:20Z,OWNER,"Aha! This may be possible already: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L311-L316 And yes, this does indeed work - you can do something like this: ``` echo '{""name"": ""harry""}' | sqlite-utils insert db.db people - --convert ' import time # Simulate something expensive time.sleep(1) def convert(row): row[""upper""] = row[""name""].upper() ' ``` And after running that: ``` sqlite-utils dump db.db BEGIN TRANSACTION; CREATE TABLE [people] ( [name] TEXT, [upper] TEXT ); INSERT INTO ""people"" VALUES('harry','HARRY'); COMMIT; ``` So this is a documentation issue - there's a trick for it but I didn't know what the trick was!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079243535,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079243535,IC_kwDOCGYnMM5AU_MP,9599,simonw,2022-03-25T17:25:12Z,2022-03-25T17:25:12Z,OWNER,"That documentation is split across a few places. This is the only bit that talks about `def convert()` pattern right now: - https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns But that's for `sqlite-utils convert` - the documentation for `sqlite-utils insert --convert` at https://sqlite-utils.datasette.io/en/stable/cli.html#applying-conversions-while-inserting-data doesn't mention it. Since both `sqlite-utils convert` and `sqlite-utils insert --convert` apply the same rules to the code, they should link to a shared explanation in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079376283,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079376283,IC_kwDOCGYnMM5AVfmb,9599,simonw,2022-03-25T19:39:30Z,2022-03-25T19:43:35Z,OWNER,"Actually this doesn't work as I thought. This demo shows that the initialization code is run once per item, not a single time at the start of the run: ``` % sqlite-utils insert dogs.db dogs dogs.json --convert ' import random print(""seeding"") random.seed(10) print(random.random()) def convert(row): print(row) row[""random_score""] = random.random() ' seeding 0.5714025946899135 seeding 0.5714025946899135 seeding 0.5714025946899135 seeding 0.5714025946899135 ``` Also that `print(row)` line is not being printed anywhere that gets to the console for some reason. ... my mistake, that happened because I changed this line in order to try to get local imports to work: ```python try: exec(code, globals, locals) return globals[""convert""] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): ``` It should be `locals[""convert""]`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079384771,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079384771,IC_kwDOCGYnMM5AVhrD,9599,simonw,2022-03-25T19:51:34Z,2022-03-25T19:53:01Z,OWNER,"This works: ``` % sqlite-utils insert dogs.db dogs dogs.json --convert ' import random print(""seeding"") random.seed(10) print(random.random()) def convert(row): global random print(row) row[""random_score""] = random.random() ' seeding 0.5714025946899135 {'id': 1, 'name': 'Cleo'} {'id': 2, 'name': 'Pancakes'} {'id': 3, 'name': 'New dog'} (sqlite-utils) sqlite-utils % sqlite-utils rows dogs.db dogs [{""id"": 1, ""name"": ""Cleo"", ""random_score"": 0.4288890546751146}, {""id"": 2, ""name"": ""Pancakes"", ""random_score"": 0.5780913011344704}, {""id"": 3, ""name"": ""New dog"", ""random_score"": 0.20609823213950174}] ``` Having to use `global random` inside the function is frustrating but apparently necessary. https://stackoverflow.com/a/56552138/6083","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079404281,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079404281,IC_kwDOCGYnMM5AVmb5,9599,simonw,2022-03-25T20:19:50Z,2022-03-25T20:19:50Z,OWNER,Now documented here: https://sqlite-utils.datasette.io/en/latest/cli.html#using-a-convert-function-to-execute-initialization,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1080141111,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1080141111,IC_kwDOCGYnMM5AYaU3,9599,simonw,2022-03-28T03:25:57Z,2022-03-28T03:54:37Z,OWNER,"So now this should solve your problem: ``` echo '[{""name"": ""notaword""}, {""name"": ""word""}] ' | python3 -m sqlite_utils insert listings.db listings - --convert ' import enchant d = enchant.Dict(""en_US"") def convert(row): global d row[""is_dictionary_word""] = d.check(row[""name""]) ' ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1081047053,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1081047053,IC_kwDOCGYnMM5Ab3gN,9599,simonw,2022-03-28T19:22:37Z,2022-03-28T19:22:37Z,OWNER,Wrote about this in my weeknotes: https://simonwillison.net/2022/Mar/28/datasette-auth0/#new-features-as-documentation,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1229438242,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1229438242,IC_kwDOCGYnMM5JR70i,9599,simonw,2022-08-28T11:34:21Z,2022-08-28T11:34:37Z,OWNER,"I found a fix that makes that `global` workaround unnecessary: - #472","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,Document how to use a `--convert` function that runs initialization code first, https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513244121,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513244121,MDEyOklzc3VlQ29tbWVudDUxMzI0NDEyMQ==,9599,simonw,2019-07-19T14:13:33Z,2019-07-19T14:13:33Z,OWNER,"So what could the interface to this look like? Especially for the CLI? One option: sqlite-utils extract dea_sales company_name companies name Tricky thing here is that it's quite a large number of positional arguments: sqlite-utils extract dea_sales company_name companies name Table column New table New column (maybe optional?) It would be great if this could supported multiple columns - for if a spreadsheet has e.g. a “Company Name”, “Company Address” pair of fields that always match each other and areduplicated many times. This could be handled by creating the new table with two columns that are indexed as a unique compound key. Then you can easily get-or-create on the pairs (or triples or whatever) from the original table. Challenge here is what does the CLI syntax look like. Something like this? $ sqlite-utils extract dea_sales -c company_name -c company_address \ --to companies --to-col name --to-col address Perhaps the columns in the new table are FORCED to be the same as the old ones, hence avoiding some options? Bit restrictive… maybe they default to the same but you can customize? $ sqlite-utils extract dea_sales -c company_name -c company_address -t companies","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246124,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513246124,MDEyOklzc3VlQ29tbWVudDUxMzI0NjEyNA==,9599,simonw,2019-07-19T14:18:35Z,2019-07-19T14:19:40Z,OWNER,"How about the Python version? That should be easier to design. ```python db[""dea_sales""].extract( columns=[""company_name"", ""company_address""], to_table=""companies"" ) ``` If we want to transform the extracted data (e.g. rename those columns) maybe support a `transform=` argument? ```python db[""dea_sales""].extract( columns=[""company_name"", ""company_address""], to_table=""companies"", transform = lambda extracted: { ""name"": extracted[""company_name""], ""address"": extracted[""company_address""], } ) ``` This would create a new ""companies"" table with three columns: id, name and address. Would also be nice if there was a syntax for saying ""... and use the value from this column as the primary key column in the newly created table"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513246831,MDEyOklzc3VlQ29tbWVudDUxMzI0NjgzMQ==,9599,simonw,2019-07-19T14:20:15Z,2019-07-19T14:20:49Z,OWNER,"Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command. The operations are full table scans so calculating progress shouldn't be too difficult.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513262013,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513262013,MDEyOklzc3VlQ29tbWVudDUxMzI2MjAxMw==,9599,simonw,2019-07-19T14:58:23Z,2020-09-22T18:12:11Z,OWNER,"CLI design idea: $ sqlite-utils extract my.db \ dea_sales company_name Here we just specify the original table and column - the new extracted table will automatically be called ""company_name"" and will have ""id"" and ""value"" columns, by default. To set a custom extract table: $ sqlite-utils extract my.db \ dea_sales company_name \ --table companies And for extracting multiple columns and renaming them on the created table, maybe something like this: $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies \ --column company_name name \ --column company_address address ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-695698227,https://api.github.com/repos/simonw/sqlite-utils/issues/42,695698227,MDEyOklzc3VlQ29tbWVudDY5NTY5ODIyNw==,9599,simonw,2020-09-20T04:27:26Z,2020-09-20T04:28:26Z,OWNER,This is going to need #114 (the `transform_table()` method) in order to convert string columns into integer foreign key columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696567460,MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA==,9599,simonw,2020-09-22T07:56:42Z,2020-09-22T07:56:42Z,OWNER,`.transform()` has landed now which should make this a lot easier to solve.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893244,MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA==,9599,simonw,2020-09-22T18:14:33Z,2020-09-22T18:14:45Z,OWNER,"Thinking more about this one: ``` $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies ``` The goal here is to pull the company name and address pair out into a separate table. Some questions: - should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this. - what should the foreign key column that gets added to the `companies` table be called?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893774,MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA==,9599,simonw,2020-09-22T18:15:33Z,2020-09-22T18:15:33Z,OWNER,I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696976678,MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA==,9599,simonw,2020-09-22T20:57:57Z,2020-09-22T20:57:57Z,OWNER,"I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID. That integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979168,MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA==,9599,simonw,2020-09-22T21:02:24Z,2020-09-22T21:02:24Z,OWNER,"In Python it looks like this: ```python # Simple case - species column species_id pointing to species table db[""trees""].extract(""species"") # Setting a custom table db[""trees""].extract(""species"", table=""Species"") # Custom foreign key column on trees db[""trees""].extract(""species"", fk_column=""species"") # Extracting multiple columns db[""trees""].extract([""common_name"", ""latin_name""]) # (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id) # Or with explicit table (fk_column here defaults to species_id because of the table name) db[""trees""].extract([""common_name"", ""latin_name""], table=""species"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979626,MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg==,9599,simonw,2020-09-22T21:03:11Z,2020-09-22T21:03:11Z,OWNER,"And if you want to rename some of the columns in the new table: ```python db[""trees""].extract([""common_name"", ""latin_name""], table=""species"", rename={""common_name"": ""name""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980503,MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw==,9599,simonw,2020-09-22T21:04:45Z,2020-09-22T21:04:45Z,OWNER,"`table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980709,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980709,MDEyOklzc3VlQ29tbWVudDY5Njk4MDcwOQ==,9599,simonw,2020-09-22T21:05:07Z,2020-09-22T21:05:07Z,OWNER,"So `.extract()` probably takes a `batch_size=` argument too, which defaults to maybe 1000.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987257,MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw==,9599,simonw,2020-09-22T21:17:34Z,2020-09-22T21:17:34Z,OWNER,"What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987925,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987925,MDEyOklzc3VlQ29tbWVudDY5Njk4NzkyNQ==,9599,simonw,2020-09-22T21:19:04Z,2020-09-22T21:19:04Z,OWNER,Need to make sure this works correctly for `rowid` tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697012111,MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ==,9599,simonw,2020-09-22T22:18:13Z,2020-09-22T22:18:13Z,OWNER,"Here's how I'm generating the examples for the documentation: ``` In [2]: import sqlite_utils In [3]: db = sqlite_utils.Database(memory=True) In [4]: db[""Trees""].insert({""id"": 1, ""TreeAddress"": ""52 Vine St"", ""CommonName"": ...: ""Palm"", ""LatinName"": ""foo""}, pk=""id"") Out[4]:
In [5]: db[""Trees""].extract([""CommonName"", ""LatinName""], table=""Species"", fk_col ...: umn=""species_id"") In [6]: print(db[""Trees""].schema) CREATE TABLE ""Trees"" ( [id] INTEGER PRIMARY KEY, [TreeAddress] TEXT, [species_id] INTEGER, FOREIGN KEY(species_id) REFERENCES Species(id) ) In [7]: print(db[""Species""].schema) CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [CommonName] TEXT, [LatinName] TEXT ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697013681,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697013681,MDEyOklzc3VlQ29tbWVudDY5NzAxMzY4MQ==,9599,simonw,2020-09-22T22:22:49Z,2020-09-22T22:22:49Z,OWNER,"The command-line version of this needs to accept a table and one or more columns, then a `--table` and `--fk-column` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697019944,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697019944,MDEyOklzc3VlQ29tbWVudDY5NzAxOTk0NA==,9599,simonw,2020-09-22T22:40:00Z,2020-09-22T22:40:00Z,OWNER,"I tried out the prototype of the CLI on the Global Power Plants data: ``` wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long ``` This threw an error because `rowid` columns are not yet supported. I fixed that like so: ``` sqlite-utils transform global.db power_plants --rename rowid id sqlite-utils extract global.db power_plants country country_long ``` That worked! But it didn't play great with Datasette, because the resulting extracted table had columns `country` and `country_long` and neither of those are called `name` or `value` or `title`. Based on this I need to add `rowid` table support AND I need to implement the proposed `rename=` argument for renaming columns on their way into the new table. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697025403,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697025403,MDEyOklzc3VlQ29tbWVudDY5NzAyNTQwMw==,9599,simonw,2020-09-22T22:57:53Z,2020-09-22T22:57:53Z,OWNER,The documentation for the `.extract()` method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697031174,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697031174,MDEyOklzc3VlQ29tbWVudDY5NzAzMTE3NA==,9599,simonw,2020-09-22T23:16:00Z,2020-09-22T23:16:00Z,OWNER,"Trying this demo again: ``` wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name ``` It worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697037974,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697037974,MDEyOklzc3VlQ29tbWVudDY5NzAzNzk3NA==,9599,simonw,2020-09-22T23:39:31Z,2020-09-22T23:39:31Z,OWNER,Documentation for `sqlite-utils extract`: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1074243540,IC_kwDOCGYnMM5AB6fU,9599,simonw,2022-03-21T18:08:03Z,2022-03-21T18:08:03Z,OWNER,"I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON! http://ndjson.org/ is the one I've been using in `sqlite-utils` - and https://github.com/ndjson/ndjson-spec#31-serialization says: > The JSON texts MUST NOT contain newlines or carriage returns. https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this: > 2. Each Line is a Valid JSON Value > > The most common values will be objects or arrays, but any JSON value is permitted. My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed. So what's `jq` doing here? It looks to me like that `jq` format is its own thing - it's not actually compatible with either of those two loose specs described above. The `jq` docs seem to call this ""whitespace-separated JSON"": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through `json.loads()` and that's it. No need to try and unwrap JSON objects that might span multiple lines. Unless someone has written a robust Python implementation of a `jq`-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards `jq -c` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,insert fails on JSONL with whitespace, https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1079441621,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1079441621,IC_kwDOCGYnMM5AVvjV,9599,simonw,2022-03-25T21:18:37Z,2022-03-25T21:18:37Z,OWNER,Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-newline-delimited-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,insert fails on JSONL with whitespace, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072833174,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1072833174,IC_kwDOCGYnMM4_8iKW,9599,simonw,2022-03-18T21:34:06Z,2022-03-18T21:34:06Z,OWNER,"Good call-out: right now the `parsedate()` and `parsedatetime()` functions both terminate with an exception if they hit something invalid: https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes It would be better if this was configurable by the user (and properly documented) - options could include ""set null if date is invalid"" and ""leave the value as it is if invalid"" in addition to throwing an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1072834273,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1072834273,IC_kwDOCGYnMM4_8ibh,9599,simonw,2022-03-18T21:36:05Z,2022-03-18T21:36:05Z,OWNER,"Python's `str.encode()` method has a `errors=` parameter that does something along these lines: https://docs.python.org/3/library/stdtypes.html#str.encode > *errors* may be given to set a different error handling scheme. The default for *errors* is `'strict'`, meaning that encoding errors raise a [`UnicodeError`](https://docs.python.org/3/library/exceptions.html#UnicodeError ""UnicodeError""). Other possible values are `'ignore'`, `'replace'`, `'xmlcharrefreplace'`, `'backslashreplace'` and any other name registered via [`codecs.register_error()`](https://docs.python.org/3/library/codecs.html#codecs.register_error ""codecs.register_error""), Imitating this might be the way to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073448904,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073448904,IC_kwDOCGYnMM4_-4fI,9599,simonw,2022-03-21T03:28:12Z,2022-03-21T03:30:37Z,OWNER,"Generating a test database using a pattern from https://www.geekytidbits.com/date-range-table-sqlite/ ``` sqlite-utils create-database test-dates.db sqlite-utils create-table test-dates.db dates id integer date text --pk id sqlite-utils test-dates.db ""WITH RECURSIVE cnt(x) AS ( SELECT 0 UNION ALL SELECT x+1 FROM cnt LIMIT (SELECT ((julianday('2016-04-01') - julianday('2016-03-15'))) + 1) ) insert into dates (date) select date(julianday('2016-03-15'), '+' || x || ' days') as date FROM cnt;"" ``` After running that: ``` % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": ""2016-03-24""}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] ``` Then to make one of them invalid: sqlite-utils test-dates.db ""update dates set date = '//' where id = 10""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073450588,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073450588,IC_kwDOCGYnMM4_-45c,9599,simonw,2022-03-21T03:32:58Z,2022-03-21T03:32:58Z,OWNER,"Then I ran this to convert `2016-03-27` etc to `2016/03/27` so I could see which ones were later converted: sqlite-utils convert test-dates.db dates date 'value.replace(""-"", ""/"")' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073451659,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073451659,IC_kwDOCGYnMM4_-5KL,9599,simonw,2022-03-21T03:35:01Z,2022-03-21T03:35:01Z,OWNER,"I confirmed that if it fails for any value ALL values are left alone, since it runs in a transaction. Here's the code that does that: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L2523-L2526","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453230,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453230,IC_kwDOCGYnMM4_-5iu,9599,simonw,2022-03-21T03:40:37Z,2022-03-21T03:40:37Z,OWNER,"I think the options here should be: - On error, raise an exception and revert the transaction (the current default) - On error, leave the value as-is - On error, set the value to `None` These need to be indicated by parameters to the `r.parsedate()` function. Some design options: - `ignore=True` to ignore errors - but how does it know if it should leave the value or set it to `None`? This is similar to other `ignore=True` parameters elsewhere in the Python API. - `errors=""ignore""`, `errors=""set-null""` - I don't like magic string values very much, but this is similar to Python's `str.encode(errors=)` mechanism - `errors=r.IGNORE` - using constants, which at least avoids magic strings. The other one could be `errors=r.SET_NULL` - `error=lambda v: None` or `error=lambda v: v` - this is a bit confusing though, introducing another callback that gets to have a go at converting the error if the first callback failed? And what happens if that lambda itself raises an error?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073453370,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073453370,IC_kwDOCGYnMM4_-5k6,9599,simonw,2022-03-21T03:41:06Z,2022-03-21T03:41:06Z,OWNER,I'm going to try the `errors=r.IGNORE` option and see what that looks like once implemented.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073455905,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073455905,IC_kwDOCGYnMM4_-6Mh,9599,simonw,2022-03-21T03:44:47Z,2022-03-21T03:45:00Z,OWNER,"This is quite nice: ``` % sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.IGNORE)"" [####################################] 100% % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": ""//""}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] % sqlite-utils convert test-dates.db dates date ""r.parsedate(value, errors=r.SET_NULL)"" [####################################] 100% % sqlite-utils rows test-dates.db dates [{""id"": 1, ""date"": ""2016-03-15""}, {""id"": 2, ""date"": ""2016-03-16""}, {""id"": 3, ""date"": ""2016-03-17""}, {""id"": 4, ""date"": ""2016-03-18""}, {""id"": 5, ""date"": ""2016-03-19""}, {""id"": 6, ""date"": ""2016-03-20""}, {""id"": 7, ""date"": ""2016-03-21""}, {""id"": 8, ""date"": ""2016-03-22""}, {""id"": 9, ""date"": ""2016-03-23""}, {""id"": 10, ""date"": null}, {""id"": 11, ""date"": ""2016-03-25""}, {""id"": 12, ""date"": ""2016-03-26""}, {""id"": 13, ""date"": ""2016-03-27""}, {""id"": 14, ""date"": ""2016-03-28""}, {""id"": 15, ""date"": ""2016-03-29""}, {""id"": 16, ""date"": ""2016-03-30""}, {""id"": 17, ""date"": ""2016-03-31""}, {""id"": 18, ""date"": ""2016-04-01""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456155,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456155,IC_kwDOCGYnMM4_-6Qb,9599,simonw,2022-03-21T03:45:37Z,2022-03-21T03:45:37Z,OWNER,"Prototype: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 8255b56..0a3693e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2583,7 +2583,11 @@ def _generate_convert_help(): """""" ).strip() recipe_names = [ - n for n in dir(recipes) if not n.startswith(""_"") and n not in (""json"", ""parser"") + n + for n in dir(recipes) + if not n.startswith(""_"") + and n not in (""json"", ""parser"") + and callable(getattr(recipes, n)) ] for name in recipe_names: fn = getattr(recipes, name) diff --git a/sqlite_utils/recipes.py b/sqlite_utils/recipes.py index 6918661..569c30d 100644 --- a/sqlite_utils/recipes.py +++ b/sqlite_utils/recipes.py @@ -1,17 +1,38 @@ from dateutil import parser import json +IGNORE = object() +SET_NULL = object() -def parsedate(value, dayfirst=False, yearfirst=False): + +def parsedate(value, dayfirst=False, yearfirst=False, errors=None): ""Parse a date and convert it to ISO date format: yyyy-mm-dd"" - return ( - parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() - ) + try: + return ( + parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst) + .date() + .isoformat() + ) + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise -def parsedatetime(value, dayfirst=False, yearfirst=False): +def parsedatetime(value, dayfirst=False, yearfirst=False, errors=None): ""Parse a datetime and convert it to ISO datetime format: yyyy-mm-ddTHH:MM:SS"" - return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + try: + return parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).isoformat() + except parser.ParserError: + if errors is IGNORE: + return value + elif errors is SET_NULL: + return None + else: + raise def jsonsplit(value, delimiter="","", type=str): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/416#issuecomment-1073456222,https://api.github.com/repos/simonw/sqlite-utils/issues/416,1073456222,IC_kwDOCGYnMM4_-6Re,9599,simonw,2022-03-21T03:45:52Z,2022-03-21T03:45:52Z,OWNER,Needs tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173023272,Options for how `r.parsedate()` should handle invalid dates, https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073463375,https://api.github.com/repos/simonw/sqlite-utils/issues/415,1073463375,IC_kwDOCGYnMM4_-8BP,9599,simonw,2022-03-21T04:02:36Z,2022-03-21T04:02:36Z,OWNER,Thanks for the really clear steps to reproduce!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1171599874,Convert with `--multi` and `--dry-run` flag does not work, https://github.com/simonw/sqlite-utils/issues/415#issuecomment-1073468996,https://api.github.com/repos/simonw/sqlite-utils/issues/415,1073468996,IC_kwDOCGYnMM4_-9ZE,9599,simonw,2022-03-21T04:14:42Z,2022-03-21T04:14:42Z,OWNER,"I can fix this like so: ``` % sqlite-utils convert demo.db demo foo '{""foo"": ""bar""}' --multi --dry-run abc --- becomes: {""foo"": ""bar""} Would affect 1 row ``` Diff is this: ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 0cf0468..b2a0440 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -2676,7 +2676,10 @@ def convert( raise click.ClickException(str(e)) if dry_run: # Pull first 20 values for first column and preview them - db.conn.create_function(""preview_transform"", 1, lambda v: fn(v) if v else v) + preview = lambda v: fn(v) if v else v + if multi: + preview = lambda v: json.dumps(fn(v), default=repr) if v else v + db.conn.create_function(""preview_transform"", 1, preview) sql = """""" select [{column}] as value, ```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1171599874,Convert with `--multi` and `--dry-run` flag does not work, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065377926,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065377926,IC_kwDOCGYnMM4_gGCG,9599,simonw,2022-03-11T18:34:05Z,2022-03-11T18:34:05Z,OWNER,"Two options: - Delete and recreate the release on GitHub, triggering it to be fixed on Read The Docs (as the `stable` version) - but cancel the push to PyPI, since that platform doesn't allow package versions to be over-written and in this case since the changelog file isn't included in the PyPI package there should be no change at all - Push a 3.25.2 release. I'm going to try and do that first option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065378902,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065378902,IC_kwDOCGYnMM4_gGRW,9599,simonw,2022-03-11T18:35:26Z,2022-03-11T18:35:26Z,OWNER,I deleted both the release and the tag from GitHub using the web interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065379528,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065379528,IC_kwDOCGYnMM4_gGbI,9599,simonw,2022-03-11T18:36:17Z,2022-03-11T18:36:17Z,OWNER,"I created a new tag and release: https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 And I cancelled the publish workflow: https://github.com/simonw/sqlite-utils/actions/runs/1970415399","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065380286,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065380286,IC_kwDOCGYnMM4_gGm-,9599,simonw,2022-03-11T18:37:23Z,2022-03-11T18:37:23Z,OWNER,"On ReadTheDocs that triggered a new `stable` build but it didn't seem to trigger a new build of `3.25.1`: https://readthedocs.org/projects/sqlite-utils/builds/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065381047,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065381047,IC_kwDOCGYnMM4_gGy3,9599,simonw,2022-03-11T18:38:27Z,2022-03-11T18:38:27Z,OWNER,"OK that fixed it here: https://sqlite-utils.datasette.io/en/stable/changelog.html I'm going to trigger a rebuild of `3.25.1` too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065382145,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065382145,IC_kwDOCGYnMM4_gHEB,9599,simonw,2022-03-11T18:39:05Z,2022-03-11T18:39:05Z,OWNER,"https://sqlite-utils.datasette.io/en/3.25.1/changelog.html is currently wrong: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/414#issuecomment-1065384183,https://api.github.com/repos/simonw/sqlite-utils/issues/414,1065384183,IC_kwDOCGYnMM4_gHj3,9599,simonw,2022-03-11T18:40:39Z,2022-03-11T18:40:39Z,OWNER,"That fixed it: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166731361,I forgot to include the changelog in the 3.25.1 release, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065245831,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065245831,IC_kwDOCGYnMM4_flyH,9599,simonw,2022-03-11T15:59:14Z,2022-03-11T15:59:14Z,OWNER,"Hint from https://twitter.com/AdamChainz/status/1502311047612575745 > Try: > > `autodoc_typehints = 'description'` > > For a list-of-arguments format > > https://sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065247619,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065247619,IC_kwDOCGYnMM4_fmOD,9599,simonw,2022-03-11T16:01:20Z,2022-03-11T16:01:20Z,OWNER,"Definitely an improvement! It does highlight that I'm not currently using the `:param XXX: description` syntax though, which should move my descriptions of each parameter into that generated list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065249883,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065249883,IC_kwDOCGYnMM4_fmxb,9599,simonw,2022-03-11T16:03:35Z,2022-03-11T16:03:35Z,OWNER,"Applying this change fixes that: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index 3bc528f..2a79711 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -2293,18 +2293,18 @@ class Table(Queryable): """""" Apply conversion function ``fn`` to every value in the specified columns. - - ``columns`` - a single column or list of string column names to convert. - - ``fn`` - a callable that takes a single argument, ``value``, and returns it converted. - - ``output`` - optional string column name to write the results to (defaults to the input column). - - ``output_type`` - if the output column needs to be created, this is the type that will be used + :param columns: a single column or list of string column names to convert. + :param fn: a callable that takes a single argument, ``value``, and returns it converted. + :param output: optional string column name to write the results to (defaults to the input column). + :param output_type: if the output column needs to be created, this is the type that will be used for the new column. - - ``drop`` - boolean, should the original column be dropped once the conversion is complete? - - ``multi`` - boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a + :param drop: boolean, should the original column be dropped once the conversion is complete? + :param multi: boolean, if ``True`` the return value of ``fn(value)`` will be expected to be a dictionary, and new columns will be created for each key of that dictionary. - - ``where`` - a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion + :param where: a SQL fragment to use as a ``WHERE`` clause to limit the rows to which the conversion is applied, for example ``age > ?`` or ``age > :age``. - - ``where_args`` - a list of arguments (if using ``?``) or a dictionary (if using ``:age``). - - ``show_progress`` - boolean, should a progress bar be displayed? + :param where_args: a list of arguments (if using ``?``) or a dictionary (if using ``:age``). + :param show_progress: boolean, should a progress bar be displayed? See :ref:`python_api_convert`. """""" ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065345515,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065345515,IC_kwDOCGYnMM4_f-Hr,9599,simonw,2022-03-11T17:52:22Z,2022-03-11T17:52:22Z,OWNER,"Well this is a huge improvement! https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.insert I'm not crazy about the `extracts=` thing though - I wonder if there's a neat way to customize that to be less verbose?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/413#issuecomment-1065357081,https://api.github.com/repos/simonw/sqlite-utils/issues/413,1065357081,IC_kwDOCGYnMM4_gA8Z,9599,simonw,2022-03-11T18:07:10Z,2022-03-11T18:07:10Z,OWNER,I'm really happy with this improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1166587040,Display autodoc type information more legibly, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646247,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646247,IC_kwDOCGYnMM4_KOsn,9599,simonw,2022-03-05T01:51:03Z,2022-03-05T01:51:03Z,OWNER,"I considered two ways of doing this. First, have methods such as `db.query_df()` and `table.rows_df` which do the same as `.query()` and `table.rows` but return a DataFrame instead of a generator of dictionaries. Second, have a compatibility class that is imported separately such as: ```python from sqlite_utils.pandas import Database ``` Then have the `.query()` and `.rows` and other similar methods return dataframes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646543,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646543,IC_kwDOCGYnMM4_KOxP,9599,simonw,2022-03-05T01:52:47Z,2022-03-05T01:52:47Z,OWNER,"I built a prototype of that second option and it looks pretty good: Here's the `pandas.py` prototype: ```python from .db import Database as _Database, Table as _Table, View as _View import pandas as pd from typing import ( Iterable, Union, Optional, ) class Database(_Database): def query( self, sql: str, params: Optional[Union[Iterable, dict]] = None ) -> pd.DataFrame: return pd.DataFrame(super().query(sql, params)) def table(self, table_name: str, **kwargs) -> Union[""Table"", ""View""]: ""Return a table object, optionally configured with default options."" klass = View if table_name in self.view_names() else Table return klass(self, table_name, **kwargs) class PandasQueryable: def rows_where( self, where: str = None, where_args: Optional[Union[Iterable, dict]] = None, order_by: str = None, select: str = ""*"", limit: int = None, offset: int = None, ) -> pd.DataFrame: return pd.DataFrame( super().rows_where( where, where_args, order_by=order_by, select=select, limit=limit, offset=offset, ) ) class Table(PandasQueryable, _Table): pass class View(PandasQueryable, _View): pass ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059646645,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059646645,IC_kwDOCGYnMM4_KOy1,9599,simonw,2022-03-05T01:53:10Z,2022-03-05T01:53:10Z,OWNER,I'm not an experienced enough Pandas user to know if this design is right or not. I'm going to leave this open for a while and solicit some feedback.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649193,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649193,IC_kwDOCGYnMM4_KPap,9599,simonw,2022-03-05T02:00:02Z,2022-03-05T02:00:02Z,OWNER,"Yeah, I imagine there are plenty of ways to do this with Pandas already - I'm opportunistically looking for a way to provide better integration with the rest of the Pandas situation from the work I've done in `sqlite-utils` already. Might be that this isn't worth doing at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649213,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649213,IC_kwDOCGYnMM4_KPa9,9599,simonw,2022-03-05T02:00:10Z,2022-03-05T02:00:10Z,OWNER,Requested feedback on Twitter here :https://twitter.com/simonw/status/1499927075930578948,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059649803,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059649803,IC_kwDOCGYnMM4_KPkL,9599,simonw,2022-03-05T02:02:41Z,2022-03-05T02:02:41Z,OWNER,"It looks like the existing `pd.read_sql_query()` method has an optional dependency on SQLAlchemy: ``` ... import pandas as pd pd.read_sql_query(db.conn, ""select * from articles"") # ImportError: Using URI string without sqlalchemy installed. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059650190,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059650190,IC_kwDOCGYnMM4_KPqO,9599,simonw,2022-03-05T02:04:43Z,2022-03-05T02:04:54Z,OWNER,"To be honest, I'm having second thoughts about this now mainly because the idiom for turning a generator of dicts into a DataFrame is SO simple: ```python df = pd.DataFrame(db.query(""select * from articles"")) ``` Given it's that simple, I'm questioning if there's any value to adding this to `sqlite-utils` at all. This likely becomes a documentation thing instead!","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651056,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059651056,IC_kwDOCGYnMM4_KP3w,9599,simonw,2022-03-05T02:09:38Z,2022-03-05T02:09:38Z,OWNER,"OK, so reading results from existing `sqlite-utils` into a Pandas DataFrame turns out to be trivial. How about writing a DataFrame to a database table? That feels like it could a lot more useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059651306,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059651306,IC_kwDOCGYnMM4_KP7q,9599,simonw,2022-03-05T02:10:49Z,2022-03-05T02:10:49Z,OWNER,"I could teach `.insert_all()` and `.upsert_all()` to optionally accept a DataFrame. A challenge there is `mypy` - if Pandas is an optional dependency, is it possibly to declare types that accept a Union that includes DataFrame?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059652538,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1059652538,IC_kwDOCGYnMM4_KQO6,9599,simonw,2022-03-05T02:13:17Z,2022-03-05T02:13:17Z,OWNER,"> It looks like the existing `pd.read_sql_query()` method has an optional dependency on SQLAlchemy: > > ``` > ... > import pandas as pd > pd.read_sql_query(db.conn, ""select * from articles"") > # ImportError: Using URI string without sqlalchemy installed. > ``` Hah, no I was wrong about this: SQLAlchemy is not needed for SQLite to work, I just had the arguments the wrong way round: ```python pd.read_sql_query(""select * from articles"", db.conn) # Shows a DateFrame ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1155364367,https://api.github.com/repos/simonw/sqlite-utils/issues/412,1155364367,IC_kwDOCGYnMM5E3XYP,9599,simonw,2022-06-14T15:36:28Z,2022-06-14T15:36:28Z,OWNER,"Here's as far as I got with my initial prototype, in `sqlite_utils/pandas.py`: ```python from .db import Database as _Database, Table as _Table, View as _View import pandas as pd from typing import ( Iterable, Union, Optional, ) class Database(_Database): def query( self, sql: str, params: Optional[Union[Iterable, dict]] = None ) -> pd.DataFrame: return pd.DataFrame(super().query(sql, params)) def table(self, table_name: str, **kwargs) -> Union[""Table"", ""View""]: ""Return a table object, optionally configured with default options."" klass = View if table_name in self.view_names() else Table return klass(self, table_name, **kwargs) class PandasQueryable: def rows_where( self, where: str = None, where_args: Optional[Union[Iterable, dict]] = None, order_by: str = None, select: str = ""*"", limit: int = None, offset: int = None, ) -> pd.DataFrame: return pd.DataFrame( super().rows_where( where, where_args, order_by=order_by, select=select, limit=limit, offset=offset, ) ) class Table(PandasQueryable, _Table): pass class View(PandasQueryable, _View): pass ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160182768,Optional Pandas integration, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065386352,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065386352,IC_kwDOCGYnMM4_gIFw,9599,simonw,2022-03-11T18:41:37Z,2022-03-11T18:41:37Z,OWNER,"I like `add-generated-column` - feels very clear to me, and is a nice place for adding logic that checks if the DB version supports it or not and shows a useful error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065389386,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065389386,IC_kwDOCGYnMM4_gI1K,9599,simonw,2022-03-11T18:42:53Z,2022-03-11T21:40:51Z,OWNER,"The Python API could be: ```python db[table_name].add_generated_column(""field"", str, ""json_extract(data, '$.field')"", stored=True) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065402557,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065402557,IC_kwDOCGYnMM4_gMC9,9599,simonw,2022-03-11T19:01:08Z,2022-03-11T21:42:25Z,OWNER,"Just spotted this in https://www.sqlite.org/gencol.html > The only functional difference is that one cannot add new STORED columns using the [ALTER TABLE ADD COLUMN](https://www.sqlite.org/lang_altertable.html#altertabaddcol) command. Only VIRTUAL columns can be added using ALTER TABLE. So to add stored columns to an existing table we would need to use the `.transform()` trick. Which implies that this should actually be a capability of the various `.create()` methods, since transform works by creating a new table with those and then copying across the old data. Here's where `.transform()` calls `.create_table_sql()` under the hood: https://github.com/simonw/sqlite-utils/blob/9388edf57aa15719095e3cf0952c1653cd070c9b/sqlite_utils/db.py#L1627-L1637","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065440445,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065440445,IC_kwDOCGYnMM4_gVS9,9599,simonw,2022-03-11T19:52:15Z,2022-03-11T19:52:15Z,OWNER,"Two new parameters to `.create_table()` and friends: - `generated={...}` - generated column definitions - `generated_stored={...}` generated stored column definitions These columns will be added at the end of the table, but you can use the `column_order=` parameter to apply a different order.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065458729,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065458729,IC_kwDOCGYnMM4_gZwp,9599,simonw,2022-03-11T19:58:50Z,2022-03-11T20:00:25Z,OWNER,"I'm coming round to your suggestion to have this as extra arguments to `sqlite-utils add-column` now, especially since you also need to pass a column type. I'd like to come up with syntax for `sqlite-utils create-table` as well. https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table Maybe extra `--generated-stored colname expression` (and `--generated`) options would work there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065596417,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065596417,IC_kwDOCGYnMM4_g7YB,9599,simonw,2022-03-11T22:30:15Z,2022-03-11T22:30:15Z,OWNER,"I tried it out in Jupyter and it works as advertised: Introspection is a bit weird: there doesn't seem to be a way to introspect generated columns outside of parsing the stored SQL schema for the columns at the moment! And the `.columns` method doesn't return them at all: https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1207-L1213 Here's why: ``` >>> db.execute(""PRAGMA table_info('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0)] >>> db.execute(""PRAGMA table_xinfo('t')"").fetchall() [(0, 'body', 'TEXT', 0, None, 0, 0), (1, 'd', 'INT', 0, None, 0, 2)] ``` So `table_xinfo()` is needed to get back columns including generated columns: https://www.sqlite.org/pragma.html#pragma_table_xinfo > **PRAGMA** *schema.***table_xinfo(***table-name***);** > > This pragma returns one row for each column in the named table, including [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables. The output is the same as for [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that hidden columns are shown rather than being omitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065597709,https://api.github.com/repos/simonw/sqlite-utils/issues/411,1065597709,IC_kwDOCGYnMM4_g7sN,9599,simonw,2022-03-11T22:32:43Z,2022-03-11T22:32:43Z,OWNER,"Trying to figure out what that extra field in `table_info` compared to `table_xinfo` is: ``` >>> list(db.query(""PRAGMA table_xinfo('t')"")) [{'cid': 0, 'name': 'body', 'type': 'TEXT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 0}, {'cid': 1, 'name': 'd', 'type': 'INT', 'notnull': 0, 'dflt_value': None, 'pk': 0, 'hidden': 2}] `` Presumably `hidden` 0 v.s 2 v.s. other values has meaning.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1160034488,Support for generated columns, https://github.com/simonw/sqlite-utils/pull/410#issuecomment-1055856441,https://api.github.com/repos/simonw/sqlite-utils/issues/410,1055856441,IC_kwDOCGYnMM4-7xc5,9599,simonw,2022-03-01T21:05:21Z,2022-03-01T21:05:21Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149729902,Correct spelling mistakes (found with codespell), https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1055930639,https://api.github.com/repos/simonw/sqlite-utils/issues/409,1055930639,IC_kwDOCGYnMM4-8DkP,9599,simonw,2022-03-01T22:40:15Z,2022-03-01T22:40:15Z,OWNER,"This test fails and I don't understand why: ```python from sqlite_utils import Database def test_transaction(): db1 = Database(memory_name=""transaction_test"", tracer=print) db2 = Database(memory_name=""transaction_test"", tracer=print) with db1.conn: db1[""t""].insert({""foo"": 1}) assert list(db2[""t""].rows) == [] assert list(db2[""t""].rows) == [{""foo"": 1}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055993700,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055993700,IC_kwDOCGYnMM4-8S9k,9599,simonw,2022-03-02T00:08:10Z,2022-03-02T00:08:10Z,OWNER,"I thought I'd made it so `deterministic=True` would be silently ignored in environments that don't support it, but clearly I missed a case here!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055995100,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055995100,IC_kwDOCGYnMM4-8TTc,9599,simonw,2022-03-02T00:10:41Z,2022-03-02T00:10:41Z,OWNER,"Here's the code in question: https://github.com/simonw/sqlite-utils/blob/521921b849003ed3742338f76f9d47ff3d95eaf3/sqlite_utils/db.py#L384-L394 It's checking for Python 3.8, because that's the version of Python that added the `deterministic=True` option: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function But from your error message it looks like it should be checking the SQLite version too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1055996626,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1055996626,IC_kwDOCGYnMM4-8TrS,9599,simonw,2022-03-02T00:12:21Z,2022-03-02T00:12:21Z,OWNER,Here's the SQLite changelog mentioning that it was added in 3.8.3: https://www.sqlite.org/changes.html#version_3_8_3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1056001414,https://api.github.com/repos/simonw/sqlite-utils/issues/408,1056001414,IC_kwDOCGYnMM4-8U2G,9599,simonw,2022-03-02T00:20:26Z,2022-03-02T00:20:26Z,OWNER,I need a `db.sqlite_version` property to implement this check.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1145882578,`deterministic=True` fails on versions of SQLite prior to 3.8.3, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040595572,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040595572,IC_kwDOCGYnMM4-Bjp0,9599,simonw,2022-02-15T17:54:58Z,2022-02-15T17:54:58Z,OWNER,This PR looks fantastic.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040596969,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040596969,IC_kwDOCGYnMM4-Bj_p,9599,simonw,2022-02-15T17:56:22Z,2022-02-15T17:56:35Z,OWNER,"We should add SpatiaLite to the action that calculates code coverage - that way we can calculate coverage across the new GIS tests as well: https://github.com/simonw/sqlite-utils/blob/main/.github/workflows/test-coverage.yml Should just be a case of adding this to that workflow - we can do this in the same PR. ``` - name: Install SpatiaLite run: sudo apt-get install libsqlite3-mod-spatialite ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040598665,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040598665,IC_kwDOCGYnMM4-BkaJ,9599,simonw,2022-02-15T17:58:11Z,2022-02-15T17:58:11Z,OWNER,"Wow, just found out I can edit files in this PR branch by hitting `.` on my keyboard while looking at the PR, then making changes in the VS Code for web on `github.dev`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040959312,https://api.github.com/repos/simonw/sqlite-utils/issues/407,1040959312,IC_kwDOCGYnMM4-C8dQ,9599,simonw,2022-02-16T00:58:32Z,2022-02-16T00:58:32Z,OWNER,This is honestly one of the most complete PRs I've ever seen for a feature of this size. Thanks so much for this!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138948786,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1040974519,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1040974519,IC_kwDOCGYnMM4-DAK3,9599,simonw,2022-02-16T01:08:17Z,2022-02-16T01:08:17Z,OWNER,"I had no idea this was possible! I guess SQLite will allow any text string as the column type, defaulting to `TEXT` as the underlying default representation if it doesn't recognize the type.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/406#issuecomment-1040978032,https://api.github.com/repos/simonw/sqlite-utils/issues/406,1040978032,IC_kwDOCGYnMM4-DBBw,9599,simonw,2022-02-16T01:10:31Z,2022-02-16T01:10:31Z,OWNER,"Allowing custom strings in the `create()` method, as you suggest in your example, feels like a reasonable way to support this. ```python db[""dummy""].create({ ""title"": str, ""vector"": ""array"", }) ``` I'm slightly nervous about that just because people might accidentally use this without realizig what they are doing - passing `""column-name"": ""string""` for example when they should have used `""column-name"": str` in order to get a `TEXT` column. Alternatively, this could work: ```python db[""dummy""].create({ ""title"": str, ""vector"": CustomColumnType(""array"") }) ``` This would play better with `mypy` too I think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128466114,Creating tables with custom datatypes, https://github.com/simonw/sqlite-utils/issues/405#issuecomment-1033424454,https://api.github.com/repos/simonw/sqlite-utils/issues/405,1033424454,IC_kwDOCGYnMM49mM5G,9599,simonw,2022-02-09T07:18:25Z,2022-02-09T07:18:25Z,OWNER,Writing tests against this is always a tiny bit fiddly since the created databases persist across the lifetime of the test run. Using randomly generated names helps.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128139375,"`Database(memory_name=""name"")` constructor argument", https://github.com/simonw/sqlite-utils/issues/405#issuecomment-1033425512,https://api.github.com/repos/simonw/sqlite-utils/issues/405,1033425512,IC_kwDOCGYnMM49mNJo,9599,simonw,2022-02-09T07:20:11Z,2022-02-09T07:20:11Z,OWNER,"Datasette's implementation: https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/database.py#L73-L79 ```python if self.memory_name: uri = ""file:{}?mode=memory&cache=shared"".format(self.memory_name) conn = sqlite3.connect( uri, uri=True, check_same_thread=False, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128139375,"`Database(memory_name=""name"")` constructor argument", https://github.com/simonw/sqlite-utils/issues/404#issuecomment-1033407778,https://api.github.com/repos/simonw/sqlite-utils/issues/404,1033407778,IC_kwDOCGYnMM49mI0i,9599,simonw,2022-02-09T06:50:26Z,2022-02-09T06:50:26Z,OWNER,"I'll use this: ``` sqlite-utils insert plants.db plants plants.csv --csv --convert ' return { ""name"": row[""name""].upper(), ""latitude"": float(row[""latitude""]), ""longitude"": float(row[""longitude""]), }' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128120451,Add example of `--convert` to the help for `sqlite-utils insert`, https://github.com/simonw/sqlite-utils/issues/404#issuecomment-1033410970,https://api.github.com/repos/simonw/sqlite-utils/issues/404,1033410970,IC_kwDOCGYnMM49mJma,9599,simonw,2022-02-09T06:56:35Z,2022-02-09T06:56:35Z,OWNER,https://sqlite-utils.datasette.io/en/latest/cli-reference.html#insert,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1128120451,Add example of `--convert` to the help for `sqlite-utils insert`, https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032976720,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1032976720,IC_kwDOCGYnMM49kflQ,9599,simonw,2022-02-08T19:23:05Z,2022-02-08T19:23:27Z,OWNER,"This is already possible using `sqlite-utils transform` like so: ``` % echo '[{""name"": ""Barry""}, {""name"": ""Sandra""}]' | sqlite-utils insert rowid.db records - % sqlite-utils schema rowid.db CREATE TABLE [records] ( [name] TEXT ); % sqlite-utils rows rowid.db records [{""name"": ""Barry""}, {""name"": ""Sandra""}] % sqlite-utils transform rowid.db records --pk id % sqlite-utils rows rowid.db records [{""id"": 1, ""name"": ""Barry""}, {""id"": 2, ""name"": ""Sandra""}] % sqlite-utils schema rowid.db CREATE TABLE ""records"" ( [id] INTEGER PRIMARY KEY, [name] TEXT ); % echo '[{""name"": ""Barry 2""}, {""name"": ""Sandra 2""}]' | sqlite-utils insert rowid.db records - % sqlite-utils rows rowid.db records [{""id"": 1, ""name"": ""Barry""}, {""id"": 2, ""name"": ""Sandra""}, {""id"": 3, ""name"": ""Barry 2""}, {""id"": 4, ""name"": ""Sandra 2""}] ``` It's not covered in the documentation though: https://sqlite-utils.datasette.io/en/3.23/cli.html#transforming-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`, https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032987901,https://api.github.com/repos/simonw/sqlite-utils/issues/403,1032987901,IC_kwDOCGYnMM49kiT9,9599,simonw,2022-02-08T19:36:06Z,2022-02-08T19:36:06Z,OWNER,New documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#adding-a-primary-key-to-a-rowid-table,"{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1126692066,Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901189,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030901189,IC_kwDOCGYnMM49ck3F,9599,simonw,2022-02-06T19:48:36Z,2022-02-06T19:48:52Z,OWNER,"From [that thread](https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566), two extra ideas which it may be possible to support in a single implementation: ```python from sqlite_utils.conversions import LongitudeLatitude db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}, ) ``` And ```python db[""places""].insert( { ""name"": ""London"", ""point"": LongitudeLatitude(-0.118092, 51.509865) } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030901853,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030901853,IC_kwDOCGYnMM49clBd,9599,simonw,2022-02-06T19:52:10Z,2022-02-06T19:52:10Z,OWNER,"So the key idea here is to introduce a new abstract base class, `Conversion`, which has the following abilities: - Can wrap one or more Python values (if called using the constructor) such that the `.insert_all()` method knows how to transform those into a format that can be included in an insert - something like `GeomFromText(?, 4326)` with input `POINT(-0.118092 51.509865)` - Can be passed to `conversions={""point"": LongitudeLatitude}` in a way that then knows to apply that conversion to every value in the `""point""` key of the data being inserted. - Maybe also extend `conversions=` to allow the definition of additional keys that use as input other rows? That's the `conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}` example above - it may not be possible to get this working with the rest of the design though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030902102,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030902102,IC_kwDOCGYnMM49clFW,9599,simonw,2022-02-06T19:53:34Z,2022-02-08T07:40:34Z,OWNER,"I like the idea that the contract for `Conversion` (or rather for its subclasses) is that it can wrap a Python value and then return both the SQL fragment - e.g. `GeomFromText(?, 4326)` - and the values that should be used as the SQL parameters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1030904948,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1030904948,IC_kwDOCGYnMM49clx0,9599,simonw,2022-02-06T20:09:42Z,2022-02-08T07:40:44Z,OWNER,"I think this is the code that needs to become aware of this system: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2453-L2469 There's an earlier branch that runs for upserts which needs to be modified too: https://github.com/simonw/sqlite-utils/blob/fea8c9bcc509bcae75e99ae8870f520103b9aa58/sqlite_utils/db.py#L2417-L2440","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031787865,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1031787865,IC_kwDOCGYnMM49f9VZ,9599,simonw,2022-02-07T18:33:27Z,2022-02-07T18:33:27Z,OWNER,"Hah, that's interesting - I've never used that mechanism before so it wasn't something that came to mind. They seem to be using a pretty surprising trick there that takes advantage of SQLite allowing you to define a column ""type"" using a made-up type name, which you can then introspect later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032294365,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032294365,IC_kwDOCGYnMM49h4_d,9599,simonw,2022-02-08T07:32:09Z,2022-02-08T07:34:41Z,OWNER,"I have an idea for how that third option could work - the one that creates a new column using values from the existing ones: ```python db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={""point"": LongitudeLatitude(""lng"", ""lat"")}, ) ``` How about specifying that the values in that `conversion=` dictionary can be: - A SQL string fragment (as currently implemented) - A subclass of `Conversion` as described above - Or... a callable function that takes the row as an argument and returns either a `Conversion` subclass instance or a literal value to be jnserted into the database (a string, int or float) Then you could do this: ```python db[""places""].insert( { ""name"": ""London"", ""lng"": -0.118092, ""lat"": 51.509865, }, conversions={ ""point"": lambda row: LongitudeLatitude( row[""lng""], row[""lat""] ) } ) ``` Something I really like about this is that it expands the abilities of `conversions=` beyond the slightly obscure need to customize the SQL fragment into something that can solve other data insertion cleanup problems too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032296717,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1032296717,IC_kwDOCGYnMM49h5kN,9599,simonw,2022-02-08T07:35:46Z,2022-02-08T07:35:46Z,OWNER,"I'm going to write the documentation for this first, before the implementation, so I can see if it explains cleanly enough that the design appears to be sound.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1033366312,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1033366312,IC_kwDOCGYnMM49l-so,9599,simonw,2022-02-09T05:28:11Z,2022-02-09T07:28:48Z,OWNER,"My hunch is that the case where you want to consider input from more than one column will actually be pretty rare - the only case I can think of where I would want to do that is for latitude/longitude columns - everything else that I'd want to use it for (which admittedly is still mostly SpatiaLite stuff) works against a single value. The reason I'm leaning towards using the constructor for the values is that I really like the look of this variant for common conversions: ```python db[""places""].insert( { ""name"": ""London"", ""boundary"": GeometryFromGeoJSON({...}) } ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1033428967,https://api.github.com/repos/simonw/sqlite-utils/issues/402,1033428967,IC_kwDOCGYnMM49mN_n,9599,simonw,2022-02-09T07:25:44Z,2022-02-09T07:28:11Z,OWNER,"The CLI version of this could perhaps look like this: sqlite-utils insert a.db places places.json \ --conversion boundary GeometryGeoJSON This will treat the boundary key as GeoJSON. It's equivalent to passing `conversions={""boundary"": geometryGeoJSON}` The combined latitude/longitude case here can be handled by combining this with the existing `--convert` mechanism. Any `Conversion` subclass will be available to the CLI in this way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125297737,Advanced class-based `conversions=` mechanism, https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030734937,https://api.github.com/repos/simonw/sqlite-utils/issues/401,1030734937,IC_kwDOCGYnMM49b8RZ,9599,simonw,2022-02-06T02:02:24Z,2022-02-06T02:02:24Z,OWNER,The example also doesn't work right now - the code that fetches from Who's On First gets a 403 forbidden error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125081640,Update SpatiaLite example in the documentation, https://github.com/simonw/sqlite-utils/issues/401#issuecomment-1030735372,https://api.github.com/repos/simonw/sqlite-utils/issues/401,1030735372,IC_kwDOCGYnMM49b8YM,9599,simonw,2022-02-06T02:05:03Z,2022-02-06T02:05:03Z,OWNER,Improved version: https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-column-values-using-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125081640,Update SpatiaLite example in the documentation, https://github.com/simonw/sqlite-utils/issues/400#issuecomment-1030730748,https://api.github.com/repos/simonw/sqlite-utils/issues/400,1030730748,IC_kwDOCGYnMM49b7P8,9599,simonw,2022-02-06T01:34:46Z,2022-02-06T01:34:46Z,OWNER,"Actually this is not needed - there is already an option that does this, it's just called `--ignore` rather than `--if-not-exists`. The lack of consistency here is a little annoying, but not annoying enough to justify making a backwards incompatible change. ``` % sqlite-utils create-table --help Usage: sqlite-utils create-table [OPTIONS] PATH TABLE COLUMNS... Add a table with the specified columns. Columns should be specified using name, type pairs, for example: sqlite-utils create-table my.db people \ id integer \ name text \ height float \ photo blob --pk id Options: --pk TEXT Column to use as primary key --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --fk ... Column, other table, other column to set as a foreign key --ignore If table already exists, do nothing --replace If table already exists, replace it --load-extension TEXT SQLite extensions to load -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1125077063,`sqlite-utils create-table` ... `--if-not-exists`, https://github.com/simonw/sqlite-utils/pull/40#issuecomment-511266664,https://api.github.com/repos/simonw/sqlite-utils/issues/40,511266664,MDEyOklzc3VlQ29tbWVudDUxMTI2NjY2NA==,9599,simonw,2019-07-15T03:47:26Z,2019-07-15T03:47:26Z,OWNER,This will close #36 and #39 once it is finished.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",467928674,.get() method plus support for compound primary keys, https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030455715,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030455715,IC_kwDOCGYnMM49a4Gj,9599,simonw,2022-02-05T00:15:28Z,2022-02-05T00:15:28Z,OWNER,"The `conversions=` argument to `.insert()` and friends is designed to handle this case, but I don't think it's very elegant: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions ```python db[""places""].insert( {""name"": ""Wales"", ""geometry"": wkt}, conversions={""geometry"": ""GeomFromText(?, 4326)""}, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030461163,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030461163,IC_kwDOCGYnMM49a5br,9599,simonw,2022-02-05T00:30:18Z,2022-02-05T00:30:18Z,OWNER,"I wonder what the most developer-friendly way to insert geometry data into SpatiaLite is? From https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html it looks like these are the main options: - `GeomFromText( wkt String [ , SRID Integer] )` - `GeomFromWKB( wkbGeometry Binary [ , SRID Integer] )` - `GeomFromKml( KmlGeometry String )` - `GeomFromGML( gmlGeometry String )` - `GeomFromGeoJSON( geoJSONGeometry String )` - `GeomFromEWKB( ewkbGeometry String )` - `GeomFromEWKT( ewktGeometry String )` - `GeomFromFGF( fgfGeometry Binary [ , SRID Integer] )` - `GeomFromTWKB( twkbGeometry BLOB [ , SRID Integer] )` - `GeomFromGPB( geom GPKG Blob Geometry )` - GeoPackage format - `GeomFromExifGpsBlob( image BLOB )` Interesting that some accept an SRID and others do not - presumably `GeomFromGeoJSON()` always uses SRID=4326?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030465557,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030465557,IC_kwDOCGYnMM49a6gV,9599,simonw,2022-02-05T00:39:09Z,2022-02-05T00:39:09Z,OWNER,"I can't seem to get `GeomFromGeoJSON()` to work - example: https://calands.datasettes.com/calands?sql=select+IsValid%28SetSRID%28GeomFromGeoJSON%28%27%7B%0D%0A++++++++%22type%22%3A+%22Point%22%2C%0D%0A++++++++%22coordinates%22%3A+%5B%0D%0A++++++++++-94.921875%2C%0D%0A++++++++++45.460130637921004%0D%0A++++++++%5D%0D%0A++++++%7D%27%29%2C+4326%29%29 ```sql select IsValid(SetSRID(GeomFromGeoJSON('{ ""type"": ""Point"", ""coordinates"": [ -94.921875, 45.460130637921004 ] }'), 4326)) ``` Returns `-1` suggesting the geometry is not valid. Just doing this (with or without that `SetSRID()` function) returns null: ```sql select SetSRID(GeomFromGeoJSON('{ ""type"": ""Point"", ""coordinates"": [ -94.921875, 45.460130637921004 ] }'), 4326) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030466255,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030466255,IC_kwDOCGYnMM49a6rP,9599,simonw,2022-02-05T00:41:35Z,2022-02-05T00:42:23Z,OWNER,"Wow, it was the newlines that broke it! This works fine: ```sql select AsWKT(SetSRID(GeomFromGeoJSON('{""type"": ""Point"",""coordinates"": [-94.921875,45.460130637921004]}'), 4326)) ``` https://calands.datasettes.com/calands?sql=select+AsWKT%28SetSRID%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%2C+4326%29%29 And removing `SetSRID()` returns exactly the same result: https://calands.datasettes.com/calands?sql=select+AsWKT%28GeomFromGeoJSON%28%27%7B%22type%22%3A+%22Point%22%2C%22coordinates%22%3A+%5B-94.921875%2C45.460130637921004%5D%7D%27%29%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030468418,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030468418,IC_kwDOCGYnMM49a7NC,9599,simonw,2022-02-05T00:49:08Z,2022-02-05T22:59:06Z,OWNER,"I'm trying to think of ways to make this nicer from the perspective of someone calling the `.insert()` or `.insert_all()` methods against a table that has geometry columns. One option would be for the code to introspect the table (if it exists) before running the insert, looking for any geometry columns. This introspection isn't easy! The table schema just gives you `""name_of_column"" point` or similar - to figure out the SRID and suchlike you need to consult the `geometry_columns` table, I think - which throws a 500 error on https://calands.datasettes.com/calands/geometry_columns for some reason. Also does the shape of that table change between SpatiaLite versions? Assuming we can introspect the table, what would we do with that information? We could add code that detects if the user attempted to pass GeoJSON objects and automatically inserts a `GeomFromGeoJSON()` function call - but detecting GeoJSON is a bit weird, and GeoJSON also isn't necessarily the nicest format for populating e.g. latitude/longitude points. Maybe we just support the simplest possible case: a tuple of floats, which we assume is `latitude, longitude` (or should we expect `longitude, latitude`, the eternal debate?) - if those are used against a geometry table (especially a point table) we assume they are coordinates that need to be converted using `GeomFromText('POINT(...`. Not crazy about either of these ideas. Is there something better?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030712129,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030712129,IC_kwDOCGYnMM49b2tB,9599,simonw,2022-02-05T23:08:45Z,2022-02-05T23:08:45Z,OWNER,"Useful thoughts on Twitter regarding making coordinate pairs easy and more complex shapes possible: https://twitter.com/dbreunig/status/1490099303888547843 > That is exactly where I was going: two modes. > > 1. Heuristics and assumptions to get coordinates as a pair (in tuple) or as columns (look for lat, lon, latitude, longitude, etc). > 2. GIS mode with projections, polys, etc > > Make it easy for people with csvs of coordinates. If you're using Geojson or shp files, you have to specify.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030732909,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030732909,IC_kwDOCGYnMM49b7xt,9599,simonw,2022-02-06T01:47:06Z,2022-02-06T01:47:06Z,OWNER,"Here's an idea for an API design: ```python geojson_geometry = {} # ... GeoJSON goes here db[""places""].insert( {""name"": ""Wales"", ""geometry"": geojson_geometry}, geojson=""geometry"" ) ``` That `geojson=` parameter takes either a single column name or an iterable of column names. Any column in that list is expected to be a compatible `geometry` and the correct conversion functions will be applied. That solves for GeoJSON, but it's a bit ugly. Should I add `wkt=` and maybe even `kml=` and `gml=` and so-on too? Definitely not, that's way too many ugly and inscrutable new parameters. More importantly: if I want to support the following how would I do it? ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)} ) ``` Here I want to provide a `(latitude, longitude)` pair and have it inserted correctly into a `point` column. Could do this, but again it's messy: ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)}, point=""point"" ) ``` And again, what about those `(longitude, latitude)` people?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030735774,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030735774,IC_kwDOCGYnMM49b8ee,9599,simonw,2022-02-06T02:08:19Z,2022-02-06T02:08:59Z,OWNER,"Maybe I should leave this entirely up to documented patterns in the `conversions={}` dictionary? But even that's not ideal for the co-ordinate case. Consider the following: ```python db[""places""].insert( {""name"": ""London"", ""point"": (51.509865, -0.118092)}, conversions={""point"": ""GeomFromText(?, 4326)""}, ) ``` The challenge here is that the SpatiaLite function `GeomFromText()` expects a WKT string, which looks like this: POINT(-0.118092 51.509865) The existing `conversions=` mechanism doesn't support applying Python code to convert the `(lat, lon)` tuple to that value. It doesn't even support passing a Python tuple as a `?` parameter - so I don't think I could come up with a SQL string that would do the right thing here either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736047,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736047,IC_kwDOCGYnMM49b8iv,9599,simonw,2022-02-06T02:10:18Z,2022-02-06T02:10:18Z,OWNER,"So maybe back to that earlier idea where the code introspects the table, figures out that `""point""` is a geometry table of type POINT, then applies the necessary conversions to the raw Python data? That feels overly-complicated to me, especially since nothing else in the `.insert()` method currently relies on table introspection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736589,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736589,IC_kwDOCGYnMM49b8rN,9599,simonw,2022-02-06T02:14:52Z,2022-02-06T02:14:52Z,OWNER,"Another idea: introduce a helper function transform pattern, something a bit like this: ```python transformer = make_transformer({ ""point"": lambda pair: ""POINT({} {})"".format(pair[1], pair[0]) }) db[""places""].insert_all( transformer([{""name"": ""London"", ""point"": (51.509865, -0.118092)}]) conversions={""point"": ""GeomFromText(?, 4326)""}, ) ``` The `make_transformer(...)` function builds an object that can work as a wrapping iterator, applying those transform functions to everything in the sequence that it wraps. So the above code would handle converting `(lat, lon)` to `POINT(lon lat)` - then the `conversions=` applies `GeomFromText`. Naming is a challenge here: `.transform()` and `.convert()` and `conversions=` all have existing meanings within the `sqlite-utils` Python library. It's also a bit of a messy way of solving this. It's not exactly a smooth API for inserting a bunch of lat/lon coordinate pairs!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030736848,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030736848,IC_kwDOCGYnMM49b8vQ,9599,simonw,2022-02-06T02:17:35Z,2022-02-06T02:17:35Z,OWNER,"Note that GeoJSON itself uses `(longitude, latitude)` so I should probably stick to that order here too. https://datatracker.ietf.org/doc/html/rfc7946#section-3.1.1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030738023,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030738023,IC_kwDOCGYnMM49b9Bn,9599,simonw,2022-02-06T02:28:05Z,2022-02-06T02:29:24Z,OWNER,"Here's the definitive guide to `latitude, longitude` v.s. `longitude, latitude`: https://macwright.com/lonlat/ > Which is right? > > Neither. This is an opinion with no right answer. Geographical tradition favors lat, lon. Math and software prefer lon, lat. I asked on Twitter here: https://twitter.com/simonw/status/1490148001569906688","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030739566,IC_kwDOCGYnMM49b9Zu,9599,simonw,2022-02-06T02:45:25Z,2022-02-06T02:50:27Z,OWNER,"Another idea - my favourite option so far: ```python from sqlite_utils.utils import LongitudeLatitude db[""places""].insert( { ""name"": ""London"", ""point"": (-0.118092, 51.509865) }, conversions={""point"": LongitudeLatitude}, ) ``` Here `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string. This would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function. Best of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740570,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740570,IC_kwDOCGYnMM49b9pa,9599,simonw,2022-02-06T02:56:17Z,2022-02-06T02:57:00Z,OWNER,"Thinking about types. The type of the `conversions` parameter right now is a bit lazy: ```python conversions: Optional[dict] = None, ``` That becomes: ```python Optional[Dict[str, Union[str, Conversion]]] ``` Where `Conversion` is an abstract base class which expects implementations to have a `.sql() -> str` and a `.convert(value) -> str` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740771,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740771,IC_kwDOCGYnMM49b9sj,9599,simonw,2022-02-06T02:58:29Z,2022-02-06T02:58:29Z,OWNER,That example you have there is really neat - I like the idea that they can also be used to populate completely new columns that are derived from the other column inputs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740846,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740846,IC_kwDOCGYnMM49b9tu,9599,simonw,2022-02-06T02:59:21Z,2022-02-06T02:59:21Z,OWNER,I wonder if there are any interesting non-geospatial canned conversions that it would be worth including?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740963,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030740963,IC_kwDOCGYnMM49b9vj,9599,simonw,2022-02-06T03:00:33Z,2022-02-06T03:00:33Z,OWNER,"Yeah, having this be a general purpose mechanism which has a few canned examples for handling geospatial stuff is a lot neater than having a mechanism for this that's exclusive to SpatiaLite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030871591,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030871591,IC_kwDOCGYnMM49cdon,9599,simonw,2022-02-06T16:57:22Z,2022-02-06T16:57:22Z,OWNER,"I wonder if I could implement the above such that this *also* works: ```python db[""places""].insert( { ""name"": ""London"", ""point"": LongitudeLatitude(-0.118092, 51.509865) } ) ``` This feels like a very natural way to work with single inserts. The challenge is writing the code inside `.insert_all()` such that it can handle these special objects in the input column values in addition to them being passed in `conversions=`. I'm feeling very good about this direction in general though, it feels like it takes the existing but not particularly elegant `conversions=` mechanism and upgrades it to be far more useful, while maintaining backwards compatibility.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030902158,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1030902158,IC_kwDOCGYnMM49clGO,9599,simonw,2022-02-06T19:53:54Z,2022-02-06T19:53:54Z,OWNER,"Moving the design of this new `Conversion` subclass mechanism to: - https://github.com/simonw/sqlite-utils/issues/402","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code", https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030454114,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030454114,IC_kwDOCGYnMM49a3ti,9599,simonw,2022-02-05T00:14:47Z,2022-02-05T00:14:47Z,OWNER,"I like these designs a lot. I would suggest `sqlite-utils create database.db --init-spatialite` there for consistency with the `sqlite-utils init-spatialite database.db` command. The other part of this story is how we support actually inserting spatial data from the command-line. I opened an issue about the challenges in doing that for the Python API here - #399 - but we need a good answer for the CLI too. I don't yet have any good ideas here. The `conversions=` option in the Python library was designed to cover these kinds of cases but it's pretty clunky and I don't think it's very widely used: https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030456717,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030456717,IC_kwDOCGYnMM49a4WN,9599,simonw,2022-02-05T00:16:42Z,2022-02-05T00:16:42Z,OWNER,"> The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{""rowid"": int, ""geometry"": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns. That's a good call. I'm happy for `sqlite-utils add-geometry-column` to throw an error if the table doesn't exist yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030521533,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030521533,IC_kwDOCGYnMM49bIK9,9599,simonw,2022-02-05T04:25:49Z,2022-02-05T04:25:49Z,OWNER,For ingesting geometry data from the command-line maybe GeoJSON would be the best route?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030534868,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030534868,IC_kwDOCGYnMM49bLbU,9599,simonw,2022-02-05T06:03:38Z,2022-02-05T06:03:38Z,OWNER,@eyeseast how do you usually insert geometries at the moment?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732093,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030732093,IC_kwDOCGYnMM49b7k9,9599,simonw,2022-02-06T01:41:37Z,2022-02-06T01:41:37Z,OWNER,Yeah I'd like to avoid adding any geo-dependencies to `sqlite-utils` if I can avoid it. I'm fine using stuff that's going to be available in SpatiaLite itself (provided it's available as a SQLite module) since then I don't need to add any extra Python dependencies.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030732222,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1030732222,IC_kwDOCGYnMM49b7m-,9599,simonw,2022-02-06T01:42:19Z,2022-02-06T01:42:28Z,OWNER,"Adding some thoughts to: - #399 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1040965672,https://api.github.com/repos/simonw/sqlite-utils/issues/398,1040965672,IC_kwDOCGYnMM4-C-Ao,9599,simonw,2022-02-16T01:02:29Z,2022-02-16T01:02:29Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#create-database - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-geometry-column - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#create-spatial-index - https://sqlite-utils.datasette.io/en/latest/cli.html#spatialite-helpers","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124237013,Add SpatiaLite helpers to CLI, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030726991,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030726991,IC_kwDOCGYnMM49b6VP,9599,simonw,2022-02-06T01:13:58Z,2022-02-06T01:13:58Z,OWNER,This is a good idea. We already have that parameter for the `table.create_index()` method: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.create_index,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030727979,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030727979,IC_kwDOCGYnMM49b6kr,9599,simonw,2022-02-06T01:19:21Z,2022-02-06T01:19:21Z,OWNER,"Just noticed there's no explicit test coverage for the `db[""table""].create(...)` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/397#issuecomment-1030730108,https://api.github.com/repos/simonw/sqlite-utils/issues/397,1030730108,IC_kwDOCGYnMM49b7F8,9599,simonw,2022-02-06T01:30:46Z,2022-02-06T01:30:46Z,OWNER,Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/python-api.html#explicitly-creating-a-table,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1123903919,Support IF NOT EXISTS for table creation, https://github.com/simonw/sqlite-utils/issues/395#issuecomment-1029686150,https://api.github.com/repos/simonw/sqlite-utils/issues/395,1029686150,IC_kwDOCGYnMM49X8OG,9599,simonw,2022-02-04T06:03:51Z,2022-02-04T06:03:51Z,OWNER,I'm just going to run the SpatiaLite tests on Ubuntu.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1123849278,"""apt-get: command not found"" error on macOS", https://github.com/simonw/sqlite-utils/issues/394#issuecomment-1029402029,https://api.github.com/repos/simonw/sqlite-utils/issues/394,1029402029,IC_kwDOCGYnMM49W22t,9599,simonw,2022-02-03T21:06:35Z,2022-02-03T21:06:35Z,OWNER,"This broke on Windows: https://github.com/simonw/sqlite-utils/runs/5056912641 ``` if recreate and os.path.exists(filename_or_conn): > os.remove(filename_or_conn) E PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_False_True_0\\data.db' ``` I'm going to revert it from `main` for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122446693,Test against Python 3.11-dev, https://github.com/simonw/sqlite-utils/issues/393#issuecomment-1029450617,https://api.github.com/repos/simonw/sqlite-utils/issues/393,1029450617,IC_kwDOCGYnMM49XCt5,9599,simonw,2022-02-03T22:13:24Z,2022-02-03T22:13:24Z,OWNER,Much better: https://sqlite-utils.datasette.io/en/latest/python-api.html#insert-replacing-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1118585417,Better documentation for insert-replace, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021876914,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021876914,IC_kwDOCGYnMM486Jqy,9599,simonw,2022-01-26T05:17:49Z,2022-01-26T05:17:49Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877058,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021877058,IC_kwDOCGYnMM486JtC,9599,simonw,2022-01-26T05:18:12Z,2022-01-26T05:18:18Z,OWNER,"Help for `insert` says: ``` --batch-size INTEGER Commit every X records ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877769,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021877769,IC_kwDOCGYnMM486J4J,9599,simonw,2022-01-26T05:19:48Z,2022-01-26T05:19:48Z,OWNER,Can use this utility function: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/utils.py#L322-L325,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1022466476,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1022466476,IC_kwDOCGYnMM488Zms,9599,simonw,2022-01-26T18:17:43Z,2022-01-26T18:17:43Z,OWNER,"Manually tested it like this: ``` # Create database with an empty ""lines"" table sqlite-utils create-table bulk-test.db lines line text # Stream records every 0.5s, commit every 5 records stream-delay docs/python-api.rst -d 500 | \ sqlite-utils bulk bulk-test.db 'insert into lines (line) values (:line)' - \ --lines --batch-size 5 ``` Running `datasette bulk-test.db` showed that records would show up about every 2.5s five at a time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101,`sqlite-utils bulk --batch-size` option, https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055,https://api.github.com/repos/simonw/sqlite-utils/issues/391,1021876055,IC_kwDOCGYnMM486JdX,9599,simonw,2022-01-26T05:15:58Z,2022-01-26T05:15:58Z,OWNER,Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114638930,`sqlite-utils bulk` progress bar, https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876463,https://api.github.com/repos/simonw/sqlite-utils/issues/391,1021876463,IC_kwDOCGYnMM486Jjv,9599,simonw,2022-01-26T05:16:51Z,2022-01-26T05:16:51Z,OWNER,"Actually adding a progress bar may not make sense here: it's designed to work with streaming input from stdin, in which case it's impossible for it to know the overall number of rows to be processed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114638930,`sqlite-utils bulk` progress bar, https://github.com/simonw/sqlite-utils/issues/390#issuecomment-1021825322,https://api.github.com/repos/simonw/sqlite-utils/issues/390,1021825322,IC_kwDOCGYnMM4859Eq,9599,simonw,2022-01-26T03:20:12Z,2022-01-26T03:20:25Z,OWNER,"``` % sqlite-utils upsert trees.db blah - Usage: sqlite-utils upsert [OPTIONS] PATH TABLE FILE Try 'sqlite-utils upsert -h' for help. Error: Missing option '--pk'. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114557284,`sqlite-utils upsert` should require `--pk` more elegantly, https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021789016,https://api.github.com/repos/simonw/sqlite-utils/issues/389,1021789016,IC_kwDOCGYnMM4850NY,9599,simonw,2022-01-26T02:05:41Z,2022-01-26T02:05:41Z,OWNER,"Oops, wrong domain in that - https://plausible.io/sqlite-utils.datasette.io is not yet showing data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114544727,Plausible analytics for documentation, https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021790136,https://api.github.com/repos/simonw/sqlite-utils/issues/389,1021790136,IC_kwDOCGYnMM4850e4,9599,simonw,2022-01-26T02:07:41Z,2022-01-26T02:07:41Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114544727,Plausible analytics for documentation, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021784593,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021784593,IC_kwDOCGYnMM485zIR,9599,simonw,2022-01-26T01:56:48Z,2022-01-26T01:56:48Z,OWNER,"On https://readthedocs.org/dashboard/sqlite-utils/advanced/ I checked this box (previously unchecked): ![CleanShot 2022-01-25 at 17 56 22@2x](https://user-images.githubusercontent.com/9599/151090718-72e553a6-826a-43b6-98bd-f70b0cba8784.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021785268,IC_kwDOCGYnMM485zS0,9599,simonw,2022-01-26T01:58:19Z,2022-01-26T01:58:19Z,OWNER,"While I'm modifying that template I'm also going to add Plausible analytics: ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021787822,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021787822,IC_kwDOCGYnMM485z6u,9599,simonw,2022-01-26T02:03:25Z,2022-01-26T02:03:25Z,OWNER,"https://sqlite-utils.datasette.io/en/latest/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021788260,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021788260,IC_kwDOCGYnMM4850Bk,9599,simonw,2022-01-26T02:04:17Z,2022-01-26T02:04:17Z,OWNER,https://sqlite-utils.datasette.io/en/3.20/ now also shows the version warning banner.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484269326,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484269326,IC_kwDOCGYnMM5YeCcO,9599,simonw,2023-03-26T23:24:12Z,2023-03-26T23:28:14Z,OWNER,"This broke on the upgrade to Furo. I fixed it in Datasette like this: https://github.com/simonw/datasette/commit/5c1cfa451d78e3935193f5e10eba59bf741241a1 and https://github.com/simonw/datasette/commit/db8cf899e286fbaa0a40f3a9ae8d5aaa1478822e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484276302,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484276302,IC_kwDOCGYnMM5YeEJO,9599,simonw,2023-03-26T23:39:27Z,2023-03-26T23:39:27Z,OWNER,"Actually final Datasette fix was: ```html+jinja {% block scripts %} {{ super() }} {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484277416,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484277416,IC_kwDOCGYnMM5YeEao,9599,simonw,2023-03-26T23:43:12Z,2023-03-26T23:43:12Z,OWNER,Fixed: https://sqlite-utils.datasette.io/en/latest/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions, https://github.com/simonw/sqlite-utils/issues/387#issuecomment-1021773056,https://api.github.com/repos/simonw/sqlite-utils/issues/387,1021773056,IC_kwDOCGYnMM485wUA,9599,simonw,2022-01-26T01:37:17Z,2022-01-26T01:37:17Z,OWNER,https://sqlite-utils.datasette.io/en/latest/python-api.html#getting-started,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1111293050,Python library docs should start with a self contained example, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029177015,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029177015,IC_kwDOCGYnMM49V_63,9599,simonw,2022-02-03T16:38:02Z,2022-02-03T16:38:02Z,OWNER,Sorry had missed this - tests should run now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029273853,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029273853,IC_kwDOCGYnMM49WXj9,9599,simonw,2022-02-03T18:23:30Z,2022-02-03T18:31:21Z,OWNER,"OK, this change makes a bunch of sense to me - and also raises some interesting questions about future additions to `sqlite-utils` with regards to SpatiaLite. Would `sqlite-utils add-geometry-column ...` be a good CLI enhancement. for example? I see you've already talked about that in #79 - moving this conversation there!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029285985,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029285985,IC_kwDOCGYnMM49Wahh,9599,simonw,2022-02-03T18:37:48Z,2022-02-03T18:37:48Z,OWNER,"`from sqlite_utils.utils import find_spatialite` is part of the documented API already: https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029296782,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029296782,IC_kwDOCGYnMM49WdKO,9599,simonw,2022-02-03T18:51:21Z,2022-02-03T18:51:21Z,OWNER,"What do you think about adding these as methods on the `Database` class instead? Then you could do: ```python # This is with an optional argument, which if omitted runs find_spatialite() for you: db.init_spatialite() # Instead of: init_spatialite(db, find_spatialite()) ``` Likewise, the `add_geometry_column` and `create_spatial_index` methods could live on `Table`: ```python # Instead of this: add_geometry_column(db[""locations""], ""POINT"", ""geometry"") create_spatial_index(db[""locations""], ""geometry"") # Could have this: db[""locations""].add_geometry_column(""POINT"") db[""locations""].create_spatial_index(""geometry"") ``` On the one hand, this is much more consistent with the existing `sqlite-utils` Python API. But on the other hand... this is mixing SpatiaLite functionality directly into the core classes. Is that a good idea, seeing as SpatiaLite is both an optional extension (which can be tricky to install) AND something that has a very different release cadence and quality-of-documentation from SQLite itself? There's a third option: the SpatiaLite could exist on subclasses of `Database` and `Table` - so the above examples would look something like this: ```python from sqlite_utils.gis import SpatiaLiteDatabase db = SpatiaLiteDatabase(""geo.db"") db.init_spatialite() db[""locations""].add_geometry_column(""POINT"") db[""locations""].create_spatial_index(""geometry"") ``` On the one hand, this would keep the SpatiaLite-specific stuff out of the core Database/Table classes. But it feels a bit untidy to me, especially since it raises the spectre of someone who was already subclassing Database for some reason now needing to instead subclass `SpatiaLiteDatabase` (not too keen on that capitalization) - or even (horror) trying to dabble with multiple inheritance, which can only lead to pain. So I don't have a strong opinion formed on this question yet!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029297971,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029297971,IC_kwDOCGYnMM49Wdcz,9599,simonw,2022-02-03T18:52:50Z,2022-02-03T18:52:50Z,OWNER,"I'm not sure I like `name=""geometry""` as the default argument to `add_geometry_column` - mainly because of this example here: ```python add_geometry_column(db[""locations""], ""POINT"") create_spatial_index(db[""locations""], ""geometry"") ``` I had to go and look at the code to figure out if `""POINT""` was the name of the column - and I don't like how inconsistent it looks next to the following `create_spatial_index()` call where you DO need to pass the column name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029335225,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029335225,IC_kwDOCGYnMM49Wmi5,9599,simonw,2022-02-03T19:39:40Z,2022-02-03T19:39:40Z,OWNER,"> I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way. > > I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API. > > You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever. Yeah that's too clever. You know what? I'm pretty confident we are both massively over-thinking this. We should put the methods on `Database` and `Table`! API simplicity and consistency matters more than vague concerns about purity.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029682294,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029682294,IC_kwDOCGYnMM49X7R2,9599,simonw,2022-02-04T05:53:26Z,2022-02-04T05:53:26Z,OWNER,"This looks fantastic, thanks for all of the work you put into this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029703216,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029703216,IC_kwDOCGYnMM49YAYw,9599,simonw,2022-02-04T06:45:43Z,2022-02-04T06:45:43Z,OWNER,Shipped this as `sqlite-utils` 3.23: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-23,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312,Add new spatialite helper methods, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010333511,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010333511,IC_kwDOCGYnMM48OHdH,9599,simonw,2022-01-11T20:27:08Z,2022-01-11T20:27:08Z,OWNER,"I'll call the new page ""CLI reference"", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010386802,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010386802,IC_kwDOCGYnMM48OUdy,9599,simonw,2022-01-11T21:44:53Z,2022-01-11T21:44:53Z,OWNER,Here's the `cog` code I used: https://github.com/simonw/sqlite-utils/blob/1d44b0cc2784c94aed1bcf350225cd86ee1aa7e5/docs/cli-reference.rst#L11-L76,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010387223,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010387223,IC_kwDOCGYnMM48OUkX,9599,simonw,2022-01-11T21:45:32Z,2022-01-11T21:45:32Z,OWNER,The new page of documentation: https://sqlite-utils.datasette.io/en/latest/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010440166,https://api.github.com/repos/simonw/sqlite-utils/issues/383,1010440166,IC_kwDOCGYnMM48Ohfm,9599,simonw,2022-01-11T22:55:05Z,2022-01-11T22:55:05Z,OWNER,Twitter thread about this: https://twitter.com/simonw/status/1481020195074293761,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099586786,Add documentation page with the output of `--help`, https://github.com/simonw/sqlite-utils/issues/382#issuecomment-1010461844,https://api.github.com/repos/simonw/sqlite-utils/issues/382,1010461844,IC_kwDOCGYnMM48OmyU,9599,simonw,2022-01-11T23:33:14Z,2022-01-11T23:33:14Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099585611,`--where` option for `sqlite-rows`, https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010441118,https://api.github.com/repos/simonw/sqlite-utils/issues/381,1010441118,IC_kwDOCGYnMM48Ohue,9599,simonw,2022-01-11T22:56:53Z,2022-01-11T22:57:09Z,OWNER,"`sqlite-utils search` has `--limit` already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search ``` --limit INTEGER Number of rows to return - defaults to everything ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099584685,`sqlite-utils rows` options `--limit` and `--offset`, https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010462035,https://api.github.com/repos/simonw/sqlite-utils/issues/381,1010462035,IC_kwDOCGYnMM48Om1T,9599,simonw,2022-01-11T23:33:37Z,2022-01-11T23:33:37Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1099584685,`sqlite-utils rows` options `--limit` and `--offset`, https://github.com/simonw/sqlite-utils/issues/380#issuecomment-1009544785,https://api.github.com/repos/simonw/sqlite-utils/issues/380,1009544785,IC_kwDOCGYnMM48LG5R,9599,simonw,2022-01-11T02:32:56Z,2022-01-11T02:32:56Z,OWNER,"CLI and Python library improvements to help run [ANALYZE](https://www.sqlite.org/lang_analyze.html) after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes. Three new CLI commands: `create-database`, `analyze` and `bulk`. - New `sqlite-utils create-database` command for creating new empty database files. ([#348](https://github.com/simonw/sqlite-utils/issues/348)) - New Python methods for running `ANALYZE` against a database, table or index: `db.analyze()` and `table.analyze()`, see [Optimizing index usage with ANALYZE](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-analyze). ([#366](https://github.com/simonw/sqlite-utils/issues/366)) - New [sqlite-utils analyze command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-analyze) for running `ANALYZE` using the CLI. ([#379](https://github.com/simonw/sqlite-utils/issues/379)) - The `create-index`, `insert` and `update` commands now have a new `--analyze` option for running `ANALYZE` after the command has completed. ([#379](https://github.com/simonw/sqlite-utils/issues/379)) - New [sqlite-utils bulk command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-bulk) which can import records in the same way as `sqlite-utils insert` (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. ([#375](https://github.com/simonw/sqlite-utils/issues/375)) - The CLI tool can now also be run using `python -m sqlite_utils`. ([#368](https://github.com/simonw/sqlite-utils/issues/368)) - Using `--fmt` now implies `--table`, so you don't need to pass both options. ([#374](https://github.com/simonw/sqlite-utils/issues/374)) - The `--convert` function applied to rows can now modify the row in place. ([#371](https://github.com/simonw/sqlite-utils/issues/371)) - The [insert-files command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-files) supports two new columns: `stem` and `suffix`. ([#372](https://github.com/simonw/sqlite-utils/issues/372)) - The `--nl` import option now ignores blank lines in the input. ([#376](https://github.com/simonw/sqlite-utils/issues/376)) - Fixed bug where streaming input to the `insert` command with `--batch-size 1` would appear to only commit after several rows had been ingested, due to unnecessary input buffering. ([#364](https://github.com/simonw/sqlite-utils/issues/364))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1098574572,Release notes for 3.21, https://github.com/simonw/sqlite-utils/pull/38#issuecomment-511219187,https://api.github.com/repos/simonw/sqlite-utils/issues/38,511219187,MDEyOklzc3VlQ29tbWVudDUxMTIxOTE4Nw==,9599,simonw,2019-07-14T17:05:26Z,2019-07-14T17:05:26Z,OWNER,I think I need to solve #36 (compound primary key support) first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",467862459,table.update() method, https://github.com/simonw/sqlite-utils/pull/38#issuecomment-514339130,https://api.github.com/repos/simonw/sqlite-utils/issues/38,514339130,MDEyOklzc3VlQ29tbWVudDUxNDMzOTEzMA==,9599,simonw,2019-07-23T18:57:33Z,2019-07-23T18:57:33Z,OWNER,"This should switch to using `.get()`, introduced in https://github.com/simonw/sqlite-utils/commit/c65b67ca46f70e2da46a5b945f4ed358173262e9","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",467862459,table.update() method, https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009532125,IC_kwDOCGYnMM48LDzd,9599,simonw,2022-01-11T02:03:35Z,2022-01-11T02:03:35Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817,https://api.github.com/repos/simonw/sqlite-utils/issues/377,1009534817,IC_kwDOCGYnMM48LEdh,9599,simonw,2022-01-11T02:09:38Z,2022-01-11T02:09:38Z,OWNER,"I tested this like so: ``` % wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' % sqlite-utils create-database test.db % sqlite-utils create-table test.db power_plants url text owner text % sqlite-utils schema test.db CREATE TABLE [power_plants] ( [url] TEXT, [owner] TEXT ); % sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv [------------------------------------] 0% [###################################-] 99% % sqlite-utils tables --counts test.db -t table count ------------ ------- power_plants 33643 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097477582,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008338186,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1008338186,IC_kwDOCGYnMM48GgUK,9599,simonw,2022-01-09T17:13:33Z,2022-01-09T17:13:54Z,OWNER," cat blah.csv | sqlite-utils bulk blah.db - \ ""insert into blah (:foo, :bar)"" --csv ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008556706,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1008556706,IC_kwDOCGYnMM48HVqi,9599,simonw,2022-01-10T05:33:41Z,2022-01-10T05:33:41Z,OWNER,"I tested the prototype like this: sqlite-utils blah.db 'create table blah (id integer primary key, name text)' echo 'id,name 1,Cleo 2,Chicken' > blah.csv sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1009536276,https://api.github.com/repos/simonw/sqlite-utils/issues/375,1009536276,IC_kwDOCGYnMM48LE0U,9599,simonw,2022-01-11T02:12:58Z,2022-01-11T02:12:58Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-sql-in-bulk,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097251014,`sqlite-utils bulk` command, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008252732,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008252732,IC_kwDOCGYnMM48GLc8,9599,simonw,2022-01-09T08:25:30Z,2022-01-09T08:25:30Z,OWNER,Need to change `if table:` to `if table or fmt:` in a few places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008345267,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008345267,IC_kwDOCGYnMM48GiCz,9599,simonw,2022-01-09T17:56:37Z,2022-01-09T17:56:37Z,OWNER,"Better: ```python if fmt: table = True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008346338,IC_kwDOCGYnMM48GiTi,9599,simonw,2022-01-09T18:03:22Z,2022-01-09T18:03:22Z,OWNER,"Commands that support `--fmt` (via the `@output_options` decorator) are: - `tables` - `views` - `query` - `memory` - `search` - `rows` - `triggers` - `indexes` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841,https://api.github.com/repos/simonw/sqlite-utils/issues/374,1008346841,IC_kwDOCGYnMM48GibZ,9599,simonw,2022-01-09T18:06:50Z,2022-01-09T18:06:50Z,OWNER,"In addition to a unit test I manually tested all of the above, e.g. ``` % sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst ======= ============ ======= ===== ====== ====== ====== ===== table index_name seqno cid name desc coll key ======= ============ ======= ===== ====== ====== ====== ===== ======= ============ ======= ===== ====== ====== ====== ===== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135860,`--fmt` should imply `-t`, https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008341078,IC_kwDOCGYnMM48GhBW,9599,simonw,2022-01-09T17:31:12Z,2022-01-09T17:31:12Z,OWNER,"Found an example of using `cog` in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21 ``` .. [[[cog from coverage.sqldata import SCHEMA_VERSION print("".. code::"") print() print(f"" SCHEMA_VERSION = {SCHEMA_VERSION}"") print() .. ]]] .. code:: SCHEMA_VERSION = 7 .. [[[end]]] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344525,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008344525,IC_kwDOCGYnMM48Gh3N,9599,simonw,2022-01-09T17:52:22Z,2022-01-09T17:52:22Z,OWNER,Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980,https://api.github.com/repos/simonw/sqlite-utils/issues/373,1008344980,IC_kwDOCGYnMM48Gh-U,9599,simonw,2022-01-09T17:54:53Z,2022-01-09T17:54:53Z,OWNER,Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097135732,List `--fmt` options in the docs , https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370,https://api.github.com/repos/simonw/sqlite-utils/issues/372,1008247370,IC_kwDOCGYnMM48GKJK,9599,simonw,2022-01-09T07:51:18Z,2022-01-09T07:51:18Z,OWNER,"Pathlib says the stem of that would be `dogs.and.cats.jpg` - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix It calls the last bit `suffix` - maybe I should use that instead of `ext`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097129710,Idea: `suffix` and `stem` file columns, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008246239,IC_kwDOCGYnMM48GJ3f,9599,simonw,2022-01-09T07:41:24Z,2022-01-09T07:41:24Z,OWNER,"Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828 To: ```python docs = (fn(doc) or doc for doc in docs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246366,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008246366,IC_kwDOCGYnMM48GJ5e,9599,simonw,2022-01-09T07:42:14Z,2022-01-09T07:42:14Z,OWNER,Also need to update relevant docs for that example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008347768,IC_kwDOCGYnMM48Gip4,9599,simonw,2022-01-09T18:12:30Z,2022-01-09T18:12:30Z,OWNER,"Tried this test: ```python result = CliRunner().invoke( cli.cli, [ ""insert"", db_path, ""rows"", ""-"", ""--convert"", 'row[""is_chicken""] = True', ], input='{""name"": ""Azi""}', ) ``` And got this error: > `E + where 1 = ', 2, 30, ' return row[""is_chicken""] = True\n'))>.exit_code` The code snippet compilation isn't currently compatible with this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008348032,IC_kwDOCGYnMM48GiuA,9599,simonw,2022-01-09T18:14:02Z,2022-01-09T18:14:02Z,OWNER,Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008354207,IC_kwDOCGYnMM48GkOf,9599,simonw,2022-01-09T18:54:54Z,2022-01-09T18:54:54Z,OWNER,"This seems to work: ```python def _compile_code(code, imports, variable=""value""): locals = {} globals = {""r"": recipes, ""recipes"": recipes} # If user defined a convert() function, return that try: exec(code, globals, locals) return locals[""convert""] except (AttributeError, SyntaxError, NameError, KeyError, TypeError): pass # Try compiling their code as a function instead body_variants = [code] # If single line and no 'return', try adding the return if ""\n"" not in code and not code.strip().startswith(""return ""): body_variants.insert(0, ""return {}"".format(code)) for variant in body_variants: new_code = [""def fn({}):"".format(variable)] for line in variant.split(""\n""): new_code.append("" {}"".format(line)) try: code_o = compile(""\n"".join(new_code), """", ""exec"") break except SyntaxError: # Try another variant, e.g. for 'return row[""column""] = 1' continue for import_ in imports: globals[import_.split(""."")[0]] = __import__(import_) exec(code_o, globals, locals) return locals[""fn""] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701,https://api.github.com/repos/simonw/sqlite-utils/issues/371,1008364701,IC_kwDOCGYnMM48Gmyd,9599,simonw,2022-01-09T20:04:35Z,2022-01-09T20:04:35Z,OWNER,"The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added `return`, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic `Error: Could not compile code` message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097128334,Support mutating row in `--convert` without returning it, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509681590,https://api.github.com/repos/simonw/sqlite-utils/issues/37,509681590,MDEyOklzc3VlQ29tbWVudDUwOTY4MTU5MA==,9599,simonw,2019-07-09T15:07:12Z,2021-06-22T18:17:53Z,OWNER,"Here's a magic incantation for generating types detected through running the tests with https://github.com/Instagram/MonkeyType ``` pip install pytest-monkeytype pytest --monkeytype-output=./monkeytype.sqlite3 monkeytype list-modules monkeytype apply sqlite_utils.utils monkeytype apply sqlite_utils.cli monkeytype apply sqlite_utils.db ``` Here's the result: https://github.com/simonw/sqlite-utils/commit/d18c694fc25b7dd3d76e250c77ddf56d10ddf935","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509685610,https://api.github.com/repos/simonw/sqlite-utils/issues/37,509685610,MDEyOklzc3VlQ29tbWVudDUwOTY4NTYxMA==,9599,simonw,2019-07-09T15:16:52Z,2019-07-09T15:16:52Z,OWNER,"Also interesting: ``` pip install pytest-mypy pytest --mypy ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509686432,https://api.github.com/repos/simonw/sqlite-utils/issues/37,509686432,MDEyOklzc3VlQ29tbWVudDUwOTY4NjQzMg==,9599,simonw,2019-07-09T15:18:49Z,2019-07-09T15:18:49Z,OWNER,"Interestingly if I run that against my `monkeytype` branch I get the following errors (subset): ``` _______________________________________________________________________________________ mypy db.py ________________________________________________________________________________________ sqlite_utils/db.py:15: error: No library stub file for module 'numpy' sqlite_utils/db.py:15: note: (Stub files are from https://github.com/python/typeshed) sqlite_utils/db.py:197: error: Incompatible types in assignment (expression has type ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], Dict[, ]]"", variable has type ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"") sqlite_utils/db.py:204: error: Item ""None"" of ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"" has no attribute ""__iter__"" (not iterable) sqlite_utils/db.py:206: error: Argument 1 to ""set"" has incompatible type ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]""; expected ""Iterable[str]"" sqlite_utils/db.py:211: error: Item ""None"" of ""Optional[Tuple[str, str, str]]"" has no attribute ""index"" sqlite_utils/db.py:211: error: Unsupported right operand type for in (""Optional[Tuple[str, str, str]]"") sqlite_utils/db.py:232: error: Unsupported right operand type for in (""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"") sqlite_utils/db.py:234: error: Value of type ""Union[Dict[str, int], Dict[str, Union[int, str]], Dict[str, str], None]"" is not indexable sqlite_utils/db.py:307: error: Need type annotation for 'table_sql' (hint: ""table_sql: Dict[, ] = ..."") sqlite_utils/db.py:427: error: Item ""int"" of ""Union[int, None, str]"" has no attribute ""startswith"" sqlite_utils/db.py:427: error: Item ""None"" of ""Union[int, None, str]"" has no attribute ""startswith"" sqlite_utils/db.py:434: error: Incompatible types in assignment (expression has type ""List[Any]"", target has type ""int"") sqlite_utils/db.py:438: error: Incompatible types in assignment (expression has type ""object"", target has type ""int"") sqlite_utils/db.py:670: error: Need type annotation for 'all_column_types' (hint: ""all_column_types: Dict[, ] = ..."") sqlite_utils/db.py:757: error: Incompatible types in assignment (expression has type ""List[Any]"", variable has type ""chain[Any]"") sqlite_utils/db.py:761: error: Argument 1 to ""create"" of ""Table"" has incompatible type ""Dict[str, Union[Type[str], Type[int], Type[bool], Type[time], Type[float]]]""; expected ""Dict[str, Union[Type[str], Type[int], Type[bool], Type[time]]]"" sqlite_utils/db.py:770: error: Need type annotation for 'all_columns' (hint: ""all_columns: Set[] = ..."") sqlite_utils/db.py:773: error: Incompatible types in assignment (expression has type ""List[Any]"", variable has type ""Set[Any]"") sqlite_utils/db.py:775: error: ""Set[Any]"" has no attribute ""insert"" sqlite_utils/db.py:797: error: Need type annotation for 'values' (hint: ""values: List[] = ..."") sqlite_utils/db.py:811: error: Argument 1 to ""add_missing_columns"" of ""Table"" has incompatible type ""chain[Any]""; expected ""Union[List[Union[Dict[str, Union[int, str]], Dict[str, Union[int, str, List[str]]]]], List[Dict[str, Union[str, int, float]]], List[Dict[str, Union[int, str]]]]"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881033,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881033,MDEyOklzc3VlQ29tbWVudDg2ODg4MTAzMw==,9599,simonw,2021-06-25T23:23:49Z,2021-06-25T23:23:49Z,OWNER,"Twitter conversation about how to add types to the `.create_table(columns=)` parameter: https://twitter.com/simonw/status/1408532867592818693 > Anyone know how to write a mypy type definition for this? > > {""id"": int, ""name"": str, ""image"": bytes, ""weight"": float} > > It's a dict where keys are strings and values are one of int/str/bytes/float (weird API design I know, but I designed this long before I was thinking about mypy) Looks like this could work: ```python def create_table( self, name, columns: Dict[str, Union[Type[int], Type[bytes], Type[str], Type[float]]], pk=None, foreign_keys=None, column_order=None, not_null=None, defaults=None, hash_id=None, extracts=None, ): ``` Except... that method can optionally also accept `numpy` types if `numpy` is installed. I don't know if it's possible to dynamically change a signature based on an import, since `mypy` is a static type analyzer and doesn't ever execute the code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881190,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881190,MDEyOklzc3VlQ29tbWVudDg2ODg4MTE5MA==,9599,simonw,2021-06-25T23:24:28Z,2021-06-25T23:24:28Z,OWNER,Maybe I could release a separate Python package `types-sqlite-utils-numpy` which adds an over-ridden type definition that includes the `numpy` types?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/37#issuecomment-901452199,https://api.github.com/repos/simonw/sqlite-utils/issues/37,901452199,IC_kwDOCGYnMM41uxGn,9599,simonw,2021-08-18T21:48:57Z,2021-08-18T21:48:57Z,OWNER,"I did a bunch of work on this in #266. The library is now pretty thoroughly typed, and I even found a couple of bugs using `mypy` along the way: #313 and #315.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219191,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219191,IC_kwDOCGYnMM48GDQ3,9599,simonw,2022-01-09T03:03:53Z,2022-01-09T03:03:53Z,OWNER,"Refs: - #366 - #365","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219484,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219484,IC_kwDOCGYnMM48GDVc,9599,simonw,2022-01-09T03:05:44Z,2022-01-09T03:05:44Z,OWNER,I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219588,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219588,IC_kwDOCGYnMM48GDXE,9599,simonw,2022-01-09T03:06:42Z,2022-01-09T03:06:42Z,OWNER,"``` analyze % sqlite-utils indexes global-power-plants.db -t table index_name seqno cid name desc coll key ------------------------------ ------------------------------------------------- ------- ----- ------------ ------ ------ ----- global-power-plants ""global-power-plants_owner"" 0 12 owner 0 BINARY 1 global-power-plants ""global-power-plants_country_long"" 0 1 country_long 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 0 0 segid 0 BINARY 1 global-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 1 1 term 0 BINARY 1 global-power-plants_fts_config sqlite_autoindex_global-power-plants_fts_config_1 0 0 k 0 BINARY 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219844,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008219844,IC_kwDOCGYnMM48GDbE,9599,simonw,2022-01-09T03:08:09Z,2022-01-09T03:08:09Z,OWNER,"``` analyze % sqlite-utils global-power-plants-analyzed.db 'analyze' [{""rows_affected"": -1}] analyze % sqlite-utils tables global-power-plants-analyzed.db [{""table"": ""global-power-plants""}, {""table"": ""global-power-plants_fts""}, {""table"": ""global-power-plants_fts_data""}, {""table"": ""global-power-plants_fts_idx""}, {""table"": ""global-power-plants_fts_docsize""}, {""table"": ""global-power-plants_fts_config""}, {""table"": ""sqlite_stat1""}] analyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t tbl idx stat ------------------------------- ---------------------------------- --------- global-power-plants_fts_config global-power-plants_fts_config 1 1 global-power-plants_fts_docsize 33643 global-power-plants_fts_idx global-power-plants_fts_idx 199 40 1 global-power-plants_fts_data 136 global-power-plants ""global-power-plants_owner"" 33643 4 global-power-plants ""global-power-plants_country_long"" 33643 202 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008220270,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008220270,IC_kwDOCGYnMM48GDhu,9599,simonw,2022-01-09T03:12:38Z,2022-01-09T03:13:15Z,OWNER,"Basically no difference using this very basic benchmark: ``` analyze % python3 -m timeit '__import__(""sqlite3"").connect(""global-power-plants.db"").execute(""select country_long, count(*) from [global-power-plants] group by country_long"").fetchall()' 100 loops, best of 5: 2.39 msec per loop analyze % python3 -m timeit '__import__(""sqlite3"").connect(""global-power-plants-analyzed.db"").execute(""select country_long, count(*) from [global-power-plants] group by country_long"").fetchall()' 100 loops, best of 5: 2.38 msec per loop ``` I should try this against a much larger database. https://covid-19.datasettes.com/covid.db is 879MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226487,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008226487,IC_kwDOCGYnMM48GFC3,9599,simonw,2022-01-09T04:14:05Z,2022-01-09T04:14:05Z,OWNER,"Didn't manage to spot a meaningful difference with that database either: ``` analyze % python3 -m timeit '__import__(""sqlite3"").connect(""covid.db"").execute(""select fips, count(*) from [ny_times_us_counties] group by fips"").fetchall()' 2 loops, best of 5: 101 msec per loop analyze % python3 -m timeit '__import__(""sqlite3"").connect(""covid-analyzed.db"").execute(""select fips, count(*) from [ny_times_us_counties] group by fips"").fetchall()' 2 loops, best of 5: 103 msec per loop ``` Maybe `select fips, count(*) from [ny_times_us_counties] group by fips` isn't a good query for testing this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226862,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008226862,IC_kwDOCGYnMM48GFIu,9599,simonw,2022-01-09T04:17:55Z,2022-01-09T04:17:55Z,OWNER,"There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html Some quotes: > SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value. > > The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the ""shape"" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed. And > Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed. And > The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008227625,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008227625,IC_kwDOCGYnMM48GFUp,9599,simonw,2022-01-09T04:25:38Z,2022-01-09T04:25:38Z,OWNER,"```sql EXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long ``` https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long > SCAN TABLE global-power-plants USING COVERING INDEX ""global-power-plants_country_long""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008229341,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008229341,IC_kwDOCGYnMM48GFvd,9599,simonw,2022-01-09T04:45:38Z,2022-01-09T04:47:11Z,OWNER,"This is probably too fancy. I think maybe the way to do this is with `select * from [global-power-plants] where ""country_long"" = 'United Kingdom'` - then mess around with stats to see if I can get it to use the index or not based on them. Here's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008232075,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1008232075,IC_kwDOCGYnMM48GGaL,9599,simonw,2022-01-09T05:13:15Z,2022-01-09T05:13:56Z,OWNER,"I think the query that will help solve this is: `explain query plan select * from ny_times_us_counties where state = 1 and county = 2` In this case, the query planner needs to decide if it should use the index for the `state` column or the index for the `county` column. That's where the statistics come into play. In particular: | tbl | idx | stat | |----------------------|---------------------------------|---------------| | ny_times_us_counties | idx_ny_times_us_counties_date | 2092871 2915 | | ny_times_us_counties | idx_ny_times_us_counties_fips | 2092871 651 | | ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085 | | ny_times_us_counties | idx_ny_times_us_counties_state | 2092871 37373 | Those numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55 ``` ** There is normally one row per index, with the index identified by the ** name in the idx column. The tbl column is the name of the table to ** which the index belongs. In each such row, the stat column will be ** a string consisting of a list of integers. The first integer in this ** list is the number of rows in the index. (This is the same as the ** number of rows in the table, except for partial indices.) The second ** integer is the average number of rows in the index that have the same ** value in the first column of the index. ``` So that table is telling us that using a value in the `county` column will filter down to an average of 1,085 rows, whereas filtering on the `state` column will filter down to an average of 37,373 - so clearly the `county` index is the better index to use here! Just one catch: against both my` covid.db` and my `covid-analyzed.db` databases the `county` index is picked for both of them - so SQLite is somehow guessing that `county` is a better index even though it doesn't have statistics for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1029402837,https://api.github.com/repos/simonw/sqlite-utils/issues/369,1029402837,IC_kwDOCGYnMM49W3DV,9599,simonw,2022-02-03T21:07:35Z,2022-02-03T21:07:35Z,OWNER,"Closing this - it was something I was curious about, but evidently not curious enough to actually do the work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097091527,Research how much of a difference analyze / sqlite_stat1 makes, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008215912,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008215912,IC_kwDOCGYnMM48GCdo,9599,simonw,2022-01-09T02:30:59Z,2022-01-09T02:30:59Z,OWNER,"Even better, inspired by `rich`, support `python -m sqlite_utils`. https://github.com/Textualize/rich/blob/master/rich/__main__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216271,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216271,IC_kwDOCGYnMM48GCjP,9599,simonw,2022-01-09T02:35:09Z,2022-01-09T02:35:09Z,OWNER,"Test failure on Python 3.6: > `E TypeError: __init__() got an unexpected keyword argument 'capture_output'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216371,https://api.github.com/repos/simonw/sqlite-utils/issues/368,1008216371,IC_kwDOCGYnMM48GCkz,9599,simonw,2022-01-09T02:36:22Z,2022-01-09T02:36:22Z,OWNER,"In Python 3.6: https://docs.python.org/3.6/library/subprocess.html > This does not capture stdout or stderr by default. To do so, pass [`PIPE`](https://docs.python.org/3.6/library/subprocess.html#subprocess.PIPE ""subprocess.PIPE"") for the *stdout* and/or *stderr* arguments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097087280,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`, https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1009272446,https://api.github.com/repos/simonw/sqlite-utils/issues/367,1009272446,IC_kwDOCGYnMM48KEZ-,9599,simonw,2022-01-10T19:31:08Z,2022-01-10T19:31:08Z,OWNER,I'm going to implement this in a separate commit from this PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1097041471,Initial prototype of .analyze() methods, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007637963,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007637963,IC_kwDOCGYnMM48D1XL,9599,simonw,2022-01-07T18:30:13Z,2022-01-07T18:30:13Z,OWNER,"Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features: - #207 - #320 there's only one method with a similar name in the Python library though and that's this one: https://github.com/simonw/sqlite-utils/blob/6e46b9913411682f3a3ec66f4d58886c1db8654b/sqlite_utils/db.py#L2904-L2906","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007639860,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007639860,IC_kwDOCGYnMM48D100,9599,simonw,2022-01-07T18:32:59Z,2022-01-07T18:33:07Z,OWNER,"From the SQLite docs: > If no arguments are given, all attached databases are analyzed. If a schema name is given as the argument, then all tables and indices in that one database are analyzed. If the argument is a table name, then only that table and the indices associated with that table are analyzed. If the argument is an index name, then only that one index is analyzed. So I think this becomes two methods: - `db.analyze()` calls analyze on the whole database - `db.analyze(name_of_table_or_index)` for a specific named table or index - `table.analyze()` is a shortcut for `db.analyze(table.name)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007641634,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1007641634,IC_kwDOCGYnMM48D2Qi,9599,simonw,2022-01-07T18:35:35Z,2022-01-07T18:35:35Z,OWNER,"Since the existing CLI feature is this: $ sqlite-utils analyze-tables github.db tags I can add `sqlite-utils analyze` to reflect the Python library method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008157132,IC_kwDOCGYnMM48F0HM,9599,simonw,2022-01-08T21:23:08Z,2022-01-08T21:25:05Z,OWNER,"Running `ANALYZE` creates a new visible table called `sqlite_stat1`: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table This should be added to the default list of hidden tables in Datasette. It looks something like this: | tbl | idx | stat | |---------------------------------|------------------------------------|-----------| | _counts | sqlite_autoindex__counts_1 | 5 1 | | global-power-plants_fts_config | global-power-plants_fts_config | 1 1 | | global-power-plants_fts_docsize | | 33643 | | global-power-plants_fts_idx | global-power-plants_fts_idx | 199 40 1 | | global-power-plants_fts_data | | 136 | | global-power-plants | ""global-power-plants_owner"" | 33643 4 | | global-power-plants | ""global-power-plants_country_long"" | 33643 202 | > In each such row, the sqlite_stat.stat column will be a string consisting of a list of integers followed by zero or more arguments. The first integer in this list is the approximate number of rows in the index. (The number of rows in the index is the same as the number of rows in the table, except for partial indexes.) The second integer is the approximate number of rows in the index that have the same value in the first column of the index. The third integer is the number number of rows in the index that have the same value for the first two columns. The N-th integer (for N>1) is the estimated average number of rows in the index which have the same value for the first N-1 columns. For a K-column index, there will be K+1 integers in the stat column. If the index is unique, then the last integer will be 1. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008158616,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1008158616,IC_kwDOCGYnMM48F0eY,9599,simonw,2022-01-08T21:35:32Z,2022-01-08T21:35:32Z,OWNER,"Built a prototype in a branch, see #367.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009273525,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009273525,IC_kwDOCGYnMM48KEq1,9599,simonw,2022-01-10T19:32:39Z,2022-01-10T19:32:39Z,OWNER,"I'm going to implement the Python library methods based on the prototype: ```diff commit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc Author: Simon Willison Date: Sat Jan 8 13:34:01 2022 -0800 Initial prototype of .analyze(), refs #366 diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index dfc4723..1348b4a 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -923,6 +923,13 @@ class Database: ""Run a SQLite ``VACUUM`` against the database."" self.execute(""VACUUM;"") + def analyze(self, name=None): + ""Run ``ANALYZE`` against the entire database or a named table or index."" + sql = ""ANALYZE"" + if name is not None: + sql += "" [{}]"".format(name) + self.execute(sql) + class Queryable: def exists(self) -> bool: @@ -2902,6 +2909,10 @@ class Table(Queryable): ) return self + def analyze(self): + ""Run ANALYZE against this table"" + self.db.analyze(self.name) + def analyze_column( self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None ) -> ""ColumnDetails"": ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009285627,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009285627,IC_kwDOCGYnMM48KHn7,9599,simonw,2022-01-10T19:49:19Z,2022-01-10T19:51:25Z,OWNER,Documentation for those two new methods: https://sqlite-utils.datasette.io/en/latest/python-api.html#optimizing-index-usage-with-analyze,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009286373,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009286373,IC_kwDOCGYnMM48KHzl,9599,simonw,2022-01-10T19:50:22Z,2022-01-10T19:50:22Z,OWNER,"With respect to #365, I'm now thinking that having the ability to say ""... and then run ANALYZE"" could be useful for a bunch of Python methods. For example: ```python db[""dogs""].insert_all(list_of_dogs, analyze=True) db[""dogs""].create_index([""name""], analyze=True) ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009288898,IC_kwDOCGYnMM48KIbC,9599,simonw,2022-01-10T19:54:04Z,2022-01-10T19:54:04Z,OWNER,"Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are: - `db.create_index` - `table.insert_all` - `table.upsert_all` - `table.delete_where`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865,https://api.github.com/repos/simonw/sqlite-utils/issues/366,1009508865,IC_kwDOCGYnMM48K-IB,9599,simonw,2022-01-11T01:08:51Z,2022-01-11T01:08:51Z,OWNER,"The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096563265,Python library methods for calling ANALYZE, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007633376,IC_kwDOCGYnMM48D0Pg,9599,simonw,2022-01-07T18:24:07Z,2022-01-07T18:24:07Z,OWNER,Relevant documentation: https://www.sqlite.org/lang_analyze.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007634999,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007634999,IC_kwDOCGYnMM48D0o3,9599,simonw,2022-01-07T18:26:22Z,2022-01-07T18:26:22Z,OWNER,"I've not used the `ANALYZE` feature in SQLite at all before. Should probably add Python library methods for it. Annoyingly I use the word ""analyze"" to mean something else in the CLI - for these features: - #207 - #320","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007642831,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007642831,IC_kwDOCGYnMM48D2jP,9599,simonw,2022-01-07T18:37:18Z,2022-01-07T18:37:18Z,OWNER,"After implementing #366 I can make it so `sqlite-utils create-index` automatically runs `db.analyze(index_name)` afterwards, maybe with a `--no-analyze` option in case anyone wants to opt out of that for specific performance reasons.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007643254,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1007643254,IC_kwDOCGYnMM48D2p2,9599,simonw,2022-01-07T18:37:56Z,2022-01-07T18:37:56Z,OWNER,Or I could leave off `--no-analyze` and tell people that if they want to add an index without running analyze they can execute the `CREATE INDEX` themselves.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008158357,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008158357,IC_kwDOCGYnMM48F0aV,9599,simonw,2022-01-08T21:33:07Z,2022-01-08T21:33:07Z,OWNER,"The one thing that worries me a little bit about doing this by default is that it adds a surprising new table to the database - it may be confusing to users if they run `create-index` and their database suddenly has a new `sqlite_stat1` table, see https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132 Options here are: - Do it anyway. People can tolerate a surprise table appearing when they create an index. - Only run `ANALYZE` if the user says `sqlite-utils create-index ... --analyze` - Use the `--analyze` option, but also automatically run `ANALYZE` if they create an index and the database they are working with already has a `sqlite_stat1` table I'm currently leading towards that third option - @fgregg any thoughts?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163050,IC_kwDOCGYnMM48F1jq,9599,simonw,2022-01-08T22:10:51Z,2022-01-08T22:10:51Z,OWNER,"Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it? Imagine the following sequence of events: - User imports a few records, creating the table, using `sqlite-utils insert` - User runs `sqlite-utils create-index ...` which also creates and populates the `sqlite_stat1` table - User runs `insert` again to populate several million new records The user now has a database file with several million records and a statistics table that is wildly out of date, having been populated when they only had a few. Will this result in surprisingly bad query performance compared to it that statistics table did not exist at all? If so, I lean much harder towards `ANALYZE` as a strictly opt-in optimization, maybe with the `--analyze` option added to `sqlite-utils insert` top to help users opt in to updating their statistics after running big inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163585,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008163585,IC_kwDOCGYnMM48F1sB,9599,simonw,2022-01-08T22:14:39Z,2022-01-09T03:03:07Z,OWNER,"The reason I'm hesitating on this is that I've not actually used ANALYZE at all in nearly five years of messing around with SQLite! So I'm nervous that there are surprise downsides I haven't thought of. My hunch is that ANALYZE is only worth worrying about on much larger databases, in which case I'm OK supporting it as a thoroughly documented power-user feature rather than a default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008229839,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1008229839,IC_kwDOCGYnMM48GF3P,9599,simonw,2022-01-09T04:51:44Z,2022-01-09T04:51:44Z,OWNER,"Found one report on Stack Overflow from 9 years ago of someone seeing broken performance after running `ANALYZE`, hard to say that's a trend and not a single weird edge-case though! https://stackoverflow.com/q/12947214/6083","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009521921,https://api.github.com/repos/simonw/sqlite-utils/issues/365,1009521921,IC_kwDOCGYnMM48LBUB,9599,simonw,2022-01-11T01:37:53Z,2022-01-11T01:37:53Z,OWNER,"I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see: - #379 - #366 You can now run the following: sqlite-utils create-index mydb.db mytable mycolumn --analyze And ``ANALYZE`` will be run on the index once it has been created.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096558279,create-index should run analyze after creating index, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008129841,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008129841,IC_kwDOCGYnMM48Ftcx,9599,simonw,2022-01-08T20:04:42Z,2022-01-08T20:04:42Z,OWNER,"It would be easier to test this if I had a utility for streaming out a file one line at a time. A few recipes for this in https://superuser.com/questions/526242/cat-file-to-terminal-at-particular-speed-of-lines-per-second - I'm going to build a quick `stream-delay` tool though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008143248,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008143248,IC_kwDOCGYnMM48FwuQ,9599,simonw,2022-01-08T20:34:12Z,2022-01-08T20:34:12Z,OWNER,Built that tool: https://github.com/simonw/stream-delay and https://pypi.org/project/stream-delay/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008151884,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008151884,IC_kwDOCGYnMM48Fy1M,9599,simonw,2022-01-08T20:59:21Z,2022-01-08T20:59:21Z,OWNER,"(That Heroku example doesn't record the timestamp, which limits its usefulness)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008153586,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008153586,IC_kwDOCGYnMM48FzPy,9599,simonw,2022-01-08T21:06:15Z,2022-01-08T21:06:15Z,OWNER,"I added a print statement after `for query, params in queries_and_params` and confirmed that something in the code is waiting until 16 records are available to be inserted and then executing the inserts, even with `--batch-size 1`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008154873,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008154873,IC_kwDOCGYnMM48Fzj5,9599,simonw,2022-01-08T21:11:55Z,2022-01-08T21:11:55Z,OWNER,"I'm suspicious that the `chunks()` utility function may not be working correctly: ```pycon In [10]: [list(d) for d in list(chunks('abc', 5))] Out[10]: [['a'], ['b'], ['c']] In [11]: [list(d) for d in list(chunks('abcdefghi', 5))] Out[11]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']] In [12]: [list(d) for d in list(chunks('abcdefghi', 3))] Out[12]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008155916,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008155916,IC_kwDOCGYnMM48Fz0M,9599,simonw,2022-01-08T21:16:46Z,2022-01-08T21:16:46Z,OWNER,"No, `chunks()` seems to work OK in the test I just added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214406,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008214406,IC_kwDOCGYnMM48GCGG,9599,simonw,2022-01-09T02:18:21Z,2022-01-09T02:18:21Z,OWNER,"I'm having trouble figuring out the best way to write a unit test for this. Filed a relevant feature request for Click here: - https://github.com/pallets/click/issues/2171","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008214998,IC_kwDOCGYnMM48GCPW,9599,simonw,2022-01-09T02:23:20Z,2022-01-09T02:23:20Z,OWNER,"Possible way of running the test: add this to `sqlite_utils/cli.py`: ```python if __name__ == ""__main__"": cli() ``` Now the tool can be run using `python -m sqlite_utils.cli --help` Then in the test use `subprocess` to call `sys.executable` (the path to the current Python interpreter) and pass it `-m sqlite_utils.cli` to run the script!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008216201,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008216201,IC_kwDOCGYnMM48GCiJ,9599,simonw,2022-01-09T02:34:12Z,2022-01-09T02:34:12Z,OWNER,"I can now write tests that look like this: https://github.com/simonw/sqlite-utils/blob/539f5ccd90371fa87f946018f8b77d55929e06db/tests/test_cli.py#L2024-L2030 Which means I can write a test that exercises this bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008233910,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008233910,IC_kwDOCGYnMM48GG22,9599,simonw,2022-01-09T05:32:53Z,2022-01-09T05:35:45Z,OWNER,"This is strange. The following: ```pycon >>> import subprocess >>> p = subprocess.Popen([""sqlite-utils"", ""insert"", ""/tmp/stream.db"", ""stream"", ""-"", ""--nl""], stdin=subprocess.PIPE) >>> p.stdin.write(b'\n'.join(b'{""id"": %s}' % str(i).encode(""utf-8"") for i in range(1000))) 11889 >>> # At this point /tmp/stream.db is still 0 bytes - but if I then run this: >>> p.stdin.close() >>> # /tmp/stream.db is now 20K and contains the written data ``` No wait, mystery solved - I can add `p.stdin.flush()` instead of `p.stdin.close()` and the file suddenly jumps up in size.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008234293,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008234293,IC_kwDOCGYnMM48GG81,9599,simonw,2022-01-09T05:37:02Z,2022-01-09T05:37:02Z,OWNER,Calling `p.stdin.close()` and then `p.wait()` terminates the subprocess.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008526736,IC_kwDOCGYnMM48HOWQ,9599,simonw,2022-01-10T04:07:29Z,2022-01-10T04:07:29Z,OWNER,"I think this test is right: ```python def test_insert_streaming_batch_size_1(db_path): # https://github.com/simonw/sqlite-utils/issues/364 # Streaming with --batch-size 1 should commit on each record # Can't use CliRunner().invoke() here bacuse we need to # run assertions in between writing to process stdin proc = subprocess.Popen( [ sys.executable, ""-m"", ""sqlite_utils"", ""insert"", db_path, ""rows"", ""-"", ""--nl"", ""--batch-size"", ""1"", ], stdin=subprocess.PIPE, ) proc.stdin.write(b'{""name"": ""Azi""}') proc.stdin.flush() assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}] proc.stdin.write(b'{""name"": ""Suna""}') proc.stdin.flush() assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}, {""name"": ""Suna""}] proc.stdin.close() proc.wait() ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008537194,IC_kwDOCGYnMM48HQ5q,9599,simonw,2022-01-10T04:29:53Z,2022-01-10T04:31:29Z,OWNER,"After a bunch of debugging with `print()` statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time. I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770 The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008545140,IC_kwDOCGYnMM48HS10,9599,simonw,2022-01-10T05:01:34Z,2022-01-10T05:01:34Z,OWNER,"Urgh, tests are still failing intermittently - for example: ``` time.sleep(0.4) > assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}] E AssertionError: assert [] == [{'name': 'Azi'}] E Right contains one more item: {'name': 'Azi'} E Full diff: E - [{'name': 'Azi'}] E + [] ``` I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008546573,IC_kwDOCGYnMM48HTMN,9599,simonw,2022-01-10T05:05:15Z,2022-01-10T05:05:15Z,OWNER,"Bit nasty but it might work: ```python def try_until(expected): tries = 0 while True: rows = list(Database(db_path)[""rows""].rows) if rows == expected: return tries += 1 if tries > 10: assert False, ""Expected {}, got {}"".format(expected, rows) time.sleep(tries * 0.1) try_until([{""name"": ""Azi""}]) proc.stdin.write(b'{""name"": ""Suna""}\n') proc.stdin.flush() try_until([{""name"": ""Azi""}, {""name"": ""Suna""}]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008557414,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008557414,IC_kwDOCGYnMM48HV1m,9599,simonw,2022-01-10T05:36:19Z,2022-01-10T05:36:19Z,OWNER,That did the trick.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,`--batch-size 1` doesn't seem to commit for every item, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006343303,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1006343303,IC_kwDOCGYnMM47-5SH,9599,simonw,2022-01-06T07:30:20Z,2022-01-06T07:30:20Z,OWNER,This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006344080,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1006344080,IC_kwDOCGYnMM47-5eQ,9599,simonw,2022-01-06T07:32:05Z,2022-01-06T07:32:05Z,OWNER,As part of this work I should add test coverage of this error message too: https://github.com/simonw/sqlite-utils/blob/413f8ed754e38d7b190de888c85fe8438336cb11/sqlite_utils/cli.py#L826,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1029469630,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1029469630,IC_kwDOCGYnMM49XHW-,9599,simonw,2022-02-03T22:42:36Z,2022-02-03T22:42:36Z,OWNER,"> This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error. Actually no that doesn't work, because this line causes an error before we even get to `.insert_all()`: https://github.com/simonw/sqlite-utils/blob/7d928f83085fb285f294dbdaeb93fd94a44d5d44/sqlite_utils/cli.py#L1012-L1013","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1029475387,https://api.github.com/repos/simonw/sqlite-utils/issues/363,1029475387,IC_kwDOCGYnMM49XIw7,9599,simonw,2022-02-03T22:52:30Z,2022-02-03T22:52:30Z,OWNER,"Demos: ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '[1]' --text Error: Rows must all be dictionaries, got: 1 % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '1' --text Error: --convert must return dict or iterator ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094981339,Better error message if `--convert` code fails to return a dict, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219848,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006219848,IC_kwDOCGYnMM47-bJI,9599,simonw,2022-01-06T01:51:36Z,2022-01-06T01:51:36Z,OWNER,"So far I've just implemented the new help: ``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE Insert records from FILE into a table, creating the table if it does not already exist. By default the input is expected to be a JSON array of objects. Or: - Use --nl for newline-delimited JSON objects - Use --csv or --tsv for comma-separated or tab-separated input - Use --lines to write each incoming line to a column called ""line"" - Use --all to write the entire input to a column called ""all"" You can also use --convert to pass a fragment of Python code that will be used to convert each input. Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. If you are using --lines your code will be passed a ""line"" variable, and for --all an ""all"" variable. Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {""a"": {""b"": 1}} becomes {""a_b"": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006220129,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006220129,IC_kwDOCGYnMM47-bNh,9599,simonw,2022-01-06T01:52:26Z,2022-01-06T01:52:26Z,OWNER,I'm going to refactor all of the tests for `sqlite-utils insert` into a new `test_cli_insert.py` module.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006230411,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006230411,IC_kwDOCGYnMM47-duL,9599,simonw,2022-01-06T02:17:35Z,2022-01-06T02:17:35Z,OWNER,"Documentation: https://github.com/simonw/sqlite-utils/blob/33223856ff7fe746b7b77750fbe5b218531d0545/docs/cli.rst#inserting-unstructured-data-with---lines-and---all - I went with a single section titled ""Inserting unstructured data with --lines and --all""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006232013,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006232013,IC_kwDOCGYnMM47-eHN,9599,simonw,2022-01-06T02:21:35Z,2022-01-06T02:21:35Z,OWNER,"I'm having second thoughts about this bit: > Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. > > If you are using `--lines` your code will be passed a ""line"" variable, and for `--all` an ""all"" variable. The code in question is this: https://github.com/simonw/sqlite-utils/blob/500a35ad4d91c8a6232134ce9406efec11bedff8/sqlite_utils/utils.py#L296-L303 Do I really want to add the complexity of supporting different variable names there? I think always using `value` might be better. Except... `value` made sense for the existing `sqlite-utils convert` command where you are running a conversion function against the value for the column in the current row - is it confusing if applied to lines or documents or `all`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006284673,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006284673,IC_kwDOCGYnMM47-q-B,9599,simonw,2022-01-06T04:55:52Z,2022-01-06T04:55:52Z,OWNER,"Test code that just worked for me: ``` sqlite-utils insert /tmp/blah.db blah /tmp/log.log --convert ' bits = line.split() return dict([(""b_{}"".format(i), bit) for i, bit in enumerate(bits)])' --lines ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006288444,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006288444,IC_kwDOCGYnMM47-r48,9599,simonw,2022-01-06T05:07:10Z,2022-01-06T05:07:10Z,OWNER,"And here's a demo of `--convert` used with `--all` - I added a custom error message for if the user's `--convert` code doesn't return a dict. ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Error: Records returned by your --convert function must be dicts % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert '{""all"": all.upper()}' --all % sqlite-utils dump /tmp/all.db BEGIN TRANSACTION; CREATE TABLE [blah] ( [all] TEXT ); INSERT INTO ""blah"" VALUES('INFO: 127.0.0.1:60581 - ""GET / HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FAVICON.ICO HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/TIDDLYWIKI HTTP/1.1"" 200 OK INFO: 127.0.0.1:60581 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/SQL-FORMATTER-2.3.3.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60585 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0.MIN.CSS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60588 - ""GET /FOO/-/STATIC/CODEMIRROR-5.57.0-SQL.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60587 - ""GET /FOO/-/STATIC/CM-RESIZE-1.0.1.MIN.JS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/TIDDLYWIKI/TIDDLERS HTTP/1.1"" 200 OK INFO: 127.0.0.1:60586 - ""GET /FOO/-/STATIC/APP.CSS?CEAD5A HTTP/1.1"" 200 OK INFO: 127.0.0.1:60584 - ""GET /FOO/-/STATIC/TABLE.JS HTTP/1.1"" 200 OK '); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006294777,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006294777,IC_kwDOCGYnMM47-tb5,9599,simonw,2022-01-06T05:24:54Z,2022-01-06T05:24:54Z,OWNER,"> I added a custom error message for if the user's `--convert` code doesn't return a dict. That turned out to be a bad idea because it meant exhausting the iterator early for the check - before we got to the `.insert_all()` code that breaks the iterator up into chunks. I tried fixing that with `itertools.tee()` to run the generator twice but that's grossly memory-inefficient for large imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006295276,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006295276,IC_kwDOCGYnMM47-tjs,9599,simonw,2022-01-06T05:26:11Z,2022-01-06T05:26:11Z,OWNER,"Here's the traceback if your `--convert` function doesn't return a dict right now: ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1062, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 763, in invoke return __callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 949, in insert insert_upsert_implementation( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 834, in insert_upsert_implementation db[table].insert_all( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2602, in insert_all first_record = next(records) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 3044, in fix_square_braces for record in records: File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 831, in docs = (decode_base64_values(doc) for doc in docs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 86, in decode_base64_values to_fix = [ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 89, in if isinstance(doc[k], dict) TypeError: string indices must be integers ``` I can live with that for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006299778,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006299778,IC_kwDOCGYnMM47-uqC,9599,simonw,2022-01-06T05:39:10Z,2022-01-06T05:39:10Z,OWNER,`all` is a bad variable name because it clashes with the Python `all()` built-in function.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006300280,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006300280,IC_kwDOCGYnMM47-ux4,9599,simonw,2022-01-06T05:40:45Z,2022-01-06T05:40:45Z,OWNER,"I'm going to rename `--all` to `--text`: > - Use `--text` to write the entire input to a column called ""text"" To avoid that clash with Python's `all()` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006301546,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006301546,IC_kwDOCGYnMM47-vFq,9599,simonw,2022-01-06T05:44:47Z,2022-01-06T05:44:47Z,OWNER,Just need documentation for `--convert` now against the various different types of input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006309834,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006309834,IC_kwDOCGYnMM47-xHK,9599,simonw,2022-01-06T06:08:01Z,2022-01-06T06:08:01Z,OWNER,"For `--text` the conversion function should be allowed to return an iterable instead of a dictionary, in which case it will be treated as the full list of records to be inserted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006311742,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006311742,IC_kwDOCGYnMM47-xk-,9599,simonw,2022-01-06T06:12:19Z,2022-01-06T06:12:19Z,OWNER,"Got that working: ``` % echo 'This is cool' | sqlite-utils insert words.db words - --text --convert '({""word"": w} for w in text.split())' % sqlite-utils dump words.db BEGIN TRANSACTION; CREATE TABLE [words] ( [word] TEXT ); INSERT INTO ""words"" VALUES('This'); INSERT INTO ""words"" VALUES('is'); INSERT INTO ""words"" VALUES('cool'); COMMIT; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006315145,https://api.github.com/repos/simonw/sqlite-utils/issues/361,1006315145,IC_kwDOCGYnMM47-yaJ,9599,simonw,2022-01-06T06:20:51Z,2022-01-06T06:20:51Z,OWNER,This is all documented. I'm going to rebase-merge it to keep the individual commits.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1094890366,--lines and --text and --convert and --import, https://github.com/simonw/sqlite-utils/issues/360#issuecomment-1006211113,https://api.github.com/repos/simonw/sqlite-utils/issues/360,1006211113,IC_kwDOCGYnMM47-ZAp,9599,simonw,2022-01-06T01:27:53Z,2022-01-06T01:27:53Z,OWNER,"It looks like you were using `sqlite-utils memory` - that works by loading the entire file into an in-memory database, so 170GB is very likely to run out of RAM. The line of code there exhibits another problem: it's reading the entire JSON file into a Python string, so it looks like it's going to run out of RAM even before it gets to the SQLite in-memory database section. To handle a file of this size you'd need to write it to a SQLite database on-disk first. The `sqlite-utils insert` command can do this, and it should be able to ""stream"" records in from a file without loading the entire thing into memory - but only for JSON-NL and CSV/TSV formats, not for JSON arrays. The code in question is here: https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L738-L773 That's using Python generators for the CSV/TSV/JSON-NL variants... but it's doing this for regular JSON which requires reading the entire thing into memory: https://github.com/simonw/sqlite-utils/blob/f3fd8613113d21d44238a6ec54b375f5aa72c4e0/sqlite_utils/cli.py#L767 If you have the ability to control how your 170GB file is generated you may have more luck converting it to CSV or TSV or newline-delimited JSON, then using `sqlite-utils insert` to insert it into a database file. To be honest though I've never tested this tooling with anything nearly that big, so it's possible you'll still run into problems. If you do I'd love to hear about them! I would be tempted to tackle this size of job by writing a custom Python script, either using the `sqlite_utils` Python library or even calling `sqlite3` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1091819089,MemoryError, https://github.com/simonw/sqlite-utils/issues/358#issuecomment-996232461,https://api.github.com/repos/simonw/sqlite-utils/issues/358,996232461,IC_kwDOCGYnMM47YU0N,9599,simonw,2021-12-16T22:10:39Z,2021-12-16T22:10:39Z,OWNER,"This goes beyond the `transform()` method - the curious methods that create new SQL tables could benefit from the ability to add `CHECK` constraints too. I haven't used these myself, do you have any `CREATE TABLE` examples that use them that you can share?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints, https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255332217,https://api.github.com/repos/simonw/sqlite-utils/issues/358,1255332217,IC_kwDOCGYnMM5K0tl5,9599,simonw,2022-09-22T17:27:34Z,2022-09-22T17:27:34Z,OWNER,"I've been thinking about this more recently. I think the first place to explore these will be in the `create-table` command (and underlying APIs). Relevant docs: https://www.sqlite.org/lang_createtable.html#check_constraints > A CHECK constraint may be attached to a column definition or specified as a table constraint. In practice it makes no difference. Each time a new row is inserted into the table or an existing row is updated, the expression associated with each CHECK constraint is evaluated and cast to a NUMERIC value in the same way as a [CAST expression](https://www.sqlite.org/lang_expr.html#castexpr). If the result is zero (integer value 0 or real value 0.0), then a constraint violation has occurred. If the CHECK expression evaluates to NULL, or any other non-zero value, it is not a constraint violation. The expression of a CHECK constraint may not contain a subquery. Something like this: sqlite-utils create-table data.db entries id integer title text tags text --pk id --check tags:json Where `--check tags:json` uses a pre-baked recipe for using the SQLite JSON function to check that the content is valid JSON and reject it otherwise. Then can bundle a bunch of other pre-baked recipes, but also support the following: --check 'x > 3' --check 'length(phone) >= 10' The besign reason for the `column:recipe` format here is to reuse `--check` for both pre-defined recipes that affect a single column AND for freeform expressions that get added to the end of the table. Detecting `column name:recipe` with a regex feels safe to me.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints, https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255333969,https://api.github.com/repos/simonw/sqlite-utils/issues/358,1255333969,IC_kwDOCGYnMM5K0uBR,9599,simonw,2022-09-22T17:29:09Z,2022-09-22T17:29:09Z,OWNER,"Quick demo of a check constraint for JSON validation: ``` sqlite> create table test (id integer primary key, tags text, check (json(tags) is not null)); sqlite> sqlite> insert into test (tags ('[""one"", ""two""]'); sqlite> insert into test (tags) values ('[""one"", ""two""'); Error: stepping, malformed JSON (1) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints, https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255340974,https://api.github.com/repos/simonw/sqlite-utils/issues/358,1255340974,IC_kwDOCGYnMM5K0vuu,9599,simonw,2022-09-22T17:34:45Z,2022-09-22T17:34:45Z,OWNER,"A few other recipes off the top of my head: - `title:maxlength:20` - set a max length, `length(title) <= 20` - `created:date` - check for `yyyy-mm-dd` date, `select :date == date(:date) is not null` ([demo](https://latest.datasette.io/_memory?sql=select+%3Adate+%3D%3D+date%28%3Adate%29+is+not+null&date=2022-01-01)) - `age:positiveint` - check `age` is a positive integer, `printf('%', age) = age and age > 0` (untested)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints, https://github.com/simonw/sqlite-utils/issues/358#issuecomment-1255341690,https://api.github.com/repos/simonw/sqlite-utils/issues/358,1255341690,IC_kwDOCGYnMM5K0v56,9599,simonw,2022-09-22T17:35:23Z,2022-09-22T17:35:23Z,OWNER,Make me think also that `sqlite-utils create-table` should have an option to dump out the SQL without actually creating the table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082651698,Support for CHECK constraints, https://github.com/simonw/sqlite-utils/issues/357#issuecomment-996179930,https://api.github.com/repos/simonw/sqlite-utils/issues/357,996179930,IC_kwDOCGYnMM47YH_a,9599,simonw,2021-12-16T20:43:19Z,2021-12-16T20:43:19Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1079422215,pytest-runner is not required, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-991517209,https://api.github.com/repos/simonw/sqlite-utils/issues/356,991517209,IC_kwDOCGYnMM47GVoZ,9599,simonw,2021-12-11T07:46:41Z,2021-12-11T07:46:41Z,OWNER,"By default this will accept single lines, but maybe there could be a `--all` option which instead grabs all of stdin into a single string against which the conversion function runs - like `git-history file`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997485361,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997485361,IC_kwDOCGYnMM47dGsx,9599,simonw,2021-12-19T23:45:30Z,2021-12-19T23:45:30Z,OWNER,Really interesting example input for this: https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/iOS13.txt - see https://blog.timac.org/2021/1219-state-of-swift-and-swiftui-ios15/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997486156,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997486156,IC_kwDOCGYnMM47dG5M,9599,simonw,2021-12-19T23:51:02Z,2021-12-19T23:51:02Z,OWNER,This is going to need a `--import` multi option too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997492872,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997492872,IC_kwDOCGYnMM47dIiI,9599,simonw,2021-12-20T00:23:31Z,2021-12-20T00:23:31Z,OWNER,"I think this should work on JSON, or CSV, or individual lines, or the entire content at once. So I'll require `--lines --convert ...` to import individual lines, or `--all --convert` to run the conversion against the entire input at once. What would `--lines` or `--all` do without `--convert`? Maybe insert records as `{""line"": ""line of text""}` or `{""all"": ""whole input}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496626,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997496626,IC_kwDOCGYnMM47dJcy,9599,simonw,2021-12-20T00:38:15Z,2022-01-06T01:29:03Z,OWNER,"The implementation of this gets a tiny bit complicated. Ignoring `--convert`, the `--lines` option can internally produce `{""line"": ...}` records and the `--all` option can produce `{""all"": ...}` records. But... when `--convert` is used, what should the code run against? It could run against those already-converted records but that's a little bit strange, since you'd have to do this: sqlite-utils insert blah.db blah myfile.txt --all --convert '{""item"": s for s in value[""all""].split(""-"")}' Having to use `value[""all""]` there is unintuitive. It would be nicer to have a `all` variable to work against. But then for `--lines` should the local variable be called `line`? And how best to summarize these different names for local variables in the inline help for the feature?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997496931,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997496931,IC_kwDOCGYnMM47dJhj,9599,simonw,2021-12-20T00:39:14Z,2021-12-20T00:39:52Z,OWNER,"``` % sqlite-utils insert --help Usage: sqlite-utils insert [OPTIONS] PATH TABLE JSON_FILE Insert records from JSON file into a table, creating the table if it does not already exist. Input should be a JSON array of objects, unless --nl or --csv is used. Options: --pk TEXT Columns to use as the primary key, e.g. id --nl Expect newline-delimited JSON --flatten Flatten nested JSON objects -c, --csv Expect CSV --tsv Expect TSV --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ``` I can add a bunch of extra help at the top there to explain all of this stuff. That ""Input should be a JSON array of objects"" bit could be expanded to several paragraphs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997497262,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997497262,IC_kwDOCGYnMM47dJmu,9599,simonw,2021-12-20T00:40:15Z,2021-12-20T00:40:15Z,OWNER,`--flatten` could do with a better description too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997507074,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997507074,IC_kwDOCGYnMM47dMAC,9599,simonw,2021-12-20T01:10:06Z,2021-12-20T01:16:11Z,OWNER,"Work-in-progress improved help: ``` Usage: sqlite-utils insert [OPTIONS] PATH TABLE FILE Insert records from FILE into a table, creating the table if it does not already exist. By default the input is expected to be a JSON array of objects. Or: - Use --nl for newline-delimited JSON objects - Use --csv or --tsv for comma-separated or tab-separated input - Use --lines to write each incoming line to a column called ""line"" - Use --all to write the entire input to a column called ""all"" You can also use --convert to pass a fragment of Python code that will be used to convert each input. Your Python code will be passed a ""row"" variable representing the imported row, and can return a modified row. If you are using --lines your code will be passed a ""line"" variable, and for --all an ""all"" variable. Options: --pk TEXT Columns to use as the primary key, e.g. id --flatten Flatten nested JSON objects, so {""a"": {""b"": 1}} becomes {""a_b"": 1} --nl Expect newline-delimited JSON -c, --csv Expect CSV input --tsv Expect TSV input --lines Treat each line as a single value called 'line' --all Treat input as a single value called 'all' --convert TEXT Python code to convert each item --import TEXT Python modules to import --delimiter TEXT Delimiter to use for CSV files --quotechar TEXT Quote character to use for CSV/TSV --sniff Detect delimiter and quote character --no-headers CSV file has no header row --batch-size INTEGER Commit every X records --alter Alter existing table to add any missing columns --not-null TEXT Columns that should be created as NOT NULL --default ... Default value that should be set for a column --encoding TEXT Character encoding for input, defaults to utf-8 -d, --detect-types Detect types for columns in CSV/TSV data --load-extension TEXT SQLite extensions to load --silent Do not show progress bar --ignore Ignore records if pk already exists --replace Replace records if pk already exists --truncate Truncate table before inserting records, if table already exists -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-997508728,https://api.github.com/repos/simonw/sqlite-utils/issues/356,997508728,IC_kwDOCGYnMM47dMZ4,9599,simonw,2021-12-20T01:14:43Z,2021-12-20T01:14:43Z,OWNER,(This makes me want `--extract` from #352 even more.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318007,https://api.github.com/repos/simonw/sqlite-utils/issues/356,1006318007,IC_kwDOCGYnMM47-zG3,9599,simonw,2022-01-06T06:28:53Z,2022-01-06T06:28:53Z,OWNER,Implemented in #361.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318443,https://api.github.com/repos/simonw/sqlite-utils/issues/356,1006318443,IC_kwDOCGYnMM47-zNr,9599,simonw,2022-01-06T06:30:13Z,2022-01-06T06:30:13Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-unstructured-data-with-lines-and-text - https://sqlite-utils.datasette.io/en/latest/cli.html#applying-conversions-while-inserting-data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077431957,`sqlite-utils insert --convert` option, https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991386841,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991386841,IC_kwDOCGYnMM47F1zZ,9599,simonw,2021-12-11T00:14:11Z,2021-12-11T00:15:15Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/7a43af232e4bc00bd227307665163614e225948b/sqlite_utils/cli.py#L2128-L2135 One way to implement this would be to look to see if the code starts with `def ...` - but that's not going to work for proper module that start with a docstring or imports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition, https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991387044,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991387044,IC_kwDOCGYnMM47F12k,9599,simonw,2021-12-11T00:14:45Z,2021-12-11T00:14:45Z,OWNER,"Maybe attempt to compile their code, and if it fails try again after adding `def fn(value):` to the start?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition, https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991393684,IC_kwDOCGYnMM47F3eU,9599,simonw,2021-12-11T00:42:19Z,2021-12-11T00:49:49Z,OWNER,"Ideally I'd like to show the perfect syntax error messages to the user - but I don't know if it's possible to do this cleanly because the error might occur with their originally entered code OR it might occur after I add `def fn(value)` to it. I'm going to punt on that for the moment and tolerate slightly confusing syntax errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition, https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991395494,https://api.github.com/repos/simonw/sqlite-utils/issues/355,991395494,IC_kwDOCGYnMM47F36m,9599,simonw,2021-12-11T00:50:22Z,2021-12-11T00:51:15Z,OWNER,"Here's an example of the new (slightly confusing) error message: ```bash sqlite-utils convert fixtures.db roadside_attractions name ' def foo(value) bar baz ' Error: Syntax error in code: def foo(value) invalid syntax ``` Another: ``` sqlite-utils convert fixtures.db roadside_attractions name '$' Error: Syntax error in code: return $ invalid syntax ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077322009,Allow users to pass a full convert() function definition, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991306712,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991306712,IC_kwDOCGYnMM47FiPY,9599,simonw,2021-12-10T21:28:27Z,2021-12-10T21:28:27Z,OWNER,"Failures started with this commit, which only touches documentation so is completely unrelated: https://github.com/simonw/sqlite-utils/commit/e328db8eba1fbf29a69eda95dfec861954f9e771","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991307422,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991307422,IC_kwDOCGYnMM47Fiae,9599,simonw,2021-12-10T21:29:34Z,2021-12-10T21:29:34Z,OWNER,Here's the test in question. The way it works is a bit weird (deleting everything in the `_fts_data` table in order to force errors that can be fixed with `.rebuild_fts()`): https://github.com/simonw/sqlite-utils/blob/8ae77a6961fed94ef2c9cc81fcfc7c81d222d9a2/tests/test_fts.py#L257-L285,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991309002,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991309002,IC_kwDOCGYnMM47FizK,9599,simonw,2021-12-10T21:32:14Z,2021-12-10T21:32:14Z,OWNER,"Here's what the method does: https://github.com/simonw/sqlite-utils/blob/e328db8eba1fbf29a69eda95dfec861954f9e771/sqlite_utils/db.py#L1941-L1952 Maybe I don't need a test that deliberately corrupts the database here? Not sure how to test that `rebuild` has been called though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991309759,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991309759,IC_kwDOCGYnMM47Fi-_,9599,simonw,2021-12-10T21:33:18Z,2021-12-10T21:33:18Z,OWNER,"https://www.sqlite.org/fts5.html#the_rebuild_command says: > This command first deletes the entire full-text index, then rebuilds it based on the contents of the table or [content table](https://www.sqlite.org/fts5.html#external_content_tables). It is not available with [contentless tables](https://www.sqlite.org/fts5.html#contentless_tables). > > `INSERT INTO ft(ft) VALUES('rebuild');`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991395919,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991395919,IC_kwDOCGYnMM47F4BP,9599,simonw,2021-12-11T00:52:31Z,2021-12-11T00:52:31Z,OWNER,"It turns out `rebuild` does indeed work against content tables, so I can put that in the test instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991398367,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991398367,IC_kwDOCGYnMM47F4nf,9599,simonw,2021-12-11T01:03:14Z,2021-12-11T01:03:14Z,OWNER,The new test: https://github.com/simonw/sqlite-utils/blob/ee13f98c2c7ca3b819bd0fc55da3108cb6a6434a/tests/test_fts.py#L270-L277,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/354#issuecomment-991399604,https://api.github.com/repos/simonw/sqlite-utils/issues/354,991399604,IC_kwDOCGYnMM47F460,9599,simonw,2021-12-11T01:08:46Z,2021-12-11T01:08:46Z,OWNER,That passed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077243232,Test failure in test_rebuild_fts, https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991376639,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991376639,IC_kwDOCGYnMM47FzT_,9599,simonw,2021-12-10T23:43:45Z,2021-12-10T23:43:45Z,OWNER,"There's a very non-obvious workaround for this at the moment. You can save your code in e.g. a file called` transform.py` - my test one looks like this: ```python def upper(value): return value.upper() ``` Then you can run the following to import and use that function: `PYTHONPATH=. sqlite-utils convert fixtures.db roadside_attractions name 'transform.upper(value)' --import transform` That `PYTHONPATH=. bit is necessary because otherwise the script won't look in the current directory for that `transform.py` module. Now that I've written this down, it's obviously bad! I think your suggestion here is a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991377288,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991377288,IC_kwDOCGYnMM47FzeI,9599,simonw,2021-12-10T23:45:53Z,2021-12-10T23:45:53Z,OWNER,"One challenge here: the current signature looks like this: ``` % sqlite-utils convert --help Usage: sqlite-utils convert [OPTIONS] DB_PATH TABLE COLUMNS... CODE ``` `CODE` is a positional argument which comes last - and since `COLUMNS` can be one or more items, making `CODE` optional isn't easy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991378346,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991378346,IC_kwDOCGYnMM47Fzuq,9599,simonw,2021-12-10T23:48:28Z,2021-12-10T23:48:28Z,OWNER,"One option: allow `CODE` to be a special value of `-` which means ""read from standard input"". It's a tiny bit of a hack but I think it would work here. If you wanted to replace a column entirely with hyphens you would still be able to do this: sqlite-utils convert my.db mytable col1 '""-""'","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381281,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991381281,IC_kwDOCGYnMM47F0ch,9599,simonw,2021-12-10T23:57:26Z,2021-12-10T23:57:26Z,OWNER,"My first attempt at building this looked a little bit strange, because you would end up having a file like this `convert.py`: ``` value = value.upper() return value ``` Which gets used like this: cat convert.py | sqlite-utils convert my.db mytable col1 - But... that `convert.py` code isn't actually valid Python - it's a weird thing where you have a partial snippet of Python code that gets wrapped in a function automatically. It would be better if you could write `convert.py` as a valid Python file with a function in it, something like this: ```python def convert(value): value = value.upper() return value ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991381679,IC_kwDOCGYnMM47F0iv,9599,simonw,2021-12-10T23:58:43Z,2021-12-10T23:59:35Z,OWNER,"I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too: ```sh sqlite-utils convert my.db mytable col1 ' def convert(value): return value.upper() ' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991399782,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991399782,IC_kwDOCGYnMM47F49m,9599,simonw,2021-12-11T01:09:37Z,2021-12-11T01:09:37Z,OWNER,"OK, this is implemented. Updated documentation is here: https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991400016,https://api.github.com/repos/simonw/sqlite-utils/issues/353,991400016,IC_kwDOCGYnMM47F5BQ,9599,simonw,2021-12-11T01:10:52Z,2021-12-11T01:11:02Z,OWNER,"This won't be in a release for a little while, but you can install it to try it out using: pip install https://github.com/simonw/sqlite-utils/archive/ee13f98c2c.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077102934,"Allow passing a file of code to ""sqlite-utils convert""", https://github.com/simonw/sqlite-utils/issues/352#issuecomment-987454872,https://api.github.com/repos/simonw/sqlite-utils/issues/352,987454872,IC_kwDOCGYnMM46212Y,9599,simonw,2021-12-07T00:56:29Z,2021-12-07T00:56:29Z,OWNER,"Thought about this due to this issue, which should stay consistent with how the `sqlite-utils` CLI works: - https://github.com/simonw/git-history/issues/41","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`, https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1024726879,https://api.github.com/repos/simonw/sqlite-utils/issues/352,1024726879,IC_kwDOCGYnMM49FBdf,9599,simonw,2022-01-28T22:59:44Z,2022-01-28T22:59:44Z,OWNER,"Rough prototype, seems to work:; ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 771d432..431b93e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -867,6 +867,12 @@ def insert_upsert_options(*, require_pk=False): ), load_extension_option, click.option(""--silent"", is_flag=True, help=""Do not show progress bar""), + click.option( + ""--extract"", + ""extracts"", + multiple=True, + help=""Columns to extract to another table"", + ), ) ): fn = decorator(fn) @@ -906,6 +912,7 @@ def insert_upsert_implementation( load_extension=None, silent=False, bulk_sql=None, + extracts=None, ): db = sqlite_utils.Database(path) _load_extensions(db, load_extension) @@ -1008,6 +1015,8 @@ def insert_upsert_implementation( extra_kwargs[""defaults""] = dict(default) if upsert: extra_kwargs[""upsert""] = upsert + if extracts is not None: + extra_kwargs[""extracts""] = extracts # Apply {""$base64"": true, ...} decoding, if needed docs = (decode_base64_values(doc) for doc in docs) @@ -1117,6 +1126,7 @@ def insert( truncate, not_null, default, + extracts, ): """""" Insert records from FILE into a table, creating the table if it @@ -1174,6 +1184,7 @@ def insert( silent=silent, not_null=not_null, default=default, + extracts=extracts, ) except UnicodeDecodeError as ex: raise click.ClickException(UNICODE_ERROR.format(ex)) @@ -1207,6 +1218,7 @@ def upsert( analyze, load_extension, silent, + extracts, ): """""" Upsert records based on their primary key. Works like 'insert' but if ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`, https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1024727476,https://api.github.com/repos/simonw/sqlite-utils/issues/352,1024727476,IC_kwDOCGYnMM49FBm0,9599,simonw,2022-01-28T23:01:06Z,2022-01-28T23:01:06Z,OWNER,"Manual test run with that prototype: ``` % echo '{""foo"": ""bar"", ""id"": 1}' | sqlite-utils insert insert-extract.db rows - --pk id --extract foo % sqlite-utils dump insert-extract.db BEGIN TRANSACTION; CREATE TABLE [foo] ( [id] INTEGER PRIMARY KEY, [value] TEXT ); INSERT INTO ""foo"" VALUES(1,'bar'); CREATE TABLE [rows] ( [foo] INTEGER REFERENCES [foo]([id]), [id] INTEGER PRIMARY KEY ); INSERT INTO ""rows"" VALUES(1,1); CREATE UNIQUE INDEX [idx_foo_value] ON [foo] ([value]); COMMIT; ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`, https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1029479388,https://api.github.com/repos/simonw/sqlite-utils/issues/352,1029479388,IC_kwDOCGYnMM49XJvc,9599,simonw,2022-02-03T22:59:35Z,2022-02-03T22:59:35Z,OWNER,"Ran into this bug again while writing tests for this: - #186","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507,`sqlite-utils insert --extract colname`, https://github.com/simonw/sqlite-utils/issues/351#issuecomment-987437043,https://api.github.com/repos/simonw/sqlite-utils/issues/351,987437043,IC_kwDOCGYnMM462xfz,9599,simonw,2021-12-07T00:41:02Z,2021-12-07T00:41:56Z,OWNER,"The fix there was: ```diff @@ -463,7 +473,7 @@ def compile_convert(convert, imports): locals = {} globals = {""json"": json} for import_ in imports: - globals[import_] = __import__(import_) + globals[import_.split(""."")[0]] = __import__(import_) exec(code_o, globals, locals) ``` With this example (needs to be modified here): ``` git-history file items.xml --convert ' tree = xml.etree.ElementTree.fromstring(content) return [el.attrib for el in tree.iter(""item"")] ' --import xml.etree.ElementTree ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072780607,Support `--import xml.etree.ElementTree` in `sqlite-utils convert`, https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987015063,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987015063,IC_kwDOCGYnMM461KeX,9599,simonw,2021-12-06T17:55:42Z,2021-12-06T17:55:42Z,OWNER,"API could be this: ```python id = db[""columns""].lookup( {""namespace"": namespace_id, ""name"": column}, cache=True ) ``` This could default to a 100 item LRU cache. You could perhaps modify that with `cache_size=500` or with `cache_size=None` to disable the size limit on that cache.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(), https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987015327,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987015327,IC_kwDOCGYnMM461Kif,9599,simonw,2021-12-06T17:56:05Z,2021-12-06T17:56:05Z,OWNER,Should I implement this remember to apply the optimization in `git-history`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(), https://github.com/simonw/sqlite-utils/issues/350#issuecomment-987016019,https://api.github.com/repos/simonw/sqlite-utils/issues/350,987016019,IC_kwDOCGYnMM461KtT,9599,simonw,2021-12-06T17:56:57Z,2021-12-06T17:56:57Z,OWNER,"Would be interesting to micro-benchmark this to get an idea for how much of a performance boost it is, since the indexed SQLite lookups used by `table.lookup()` should be really fast already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072435124,Optional caching mechanism for table.lookup(), https://github.com/simonw/sqlite-utils/issues/35#issuecomment-511216484,https://api.github.com/repos/simonw/sqlite-utils/issues/35,511216484,MDEyOklzc3VlQ29tbWVudDUxMTIxNjQ4NA==,9599,simonw,2019-07-14T16:26:49Z,2019-07-14T16:26:49Z,OWNER,"The last touched primary key should go in `last_pk`. https://github.com/simonw/sqlite-utils/blob/ba6bd47f15c154e348c3c85c05858053a4ab7bab/docs/python-api.rst#L126","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",462430920,table.update(...) method, https://github.com/simonw/sqlite-utils/issues/35#issuecomment-511216664,https://api.github.com/repos/simonw/sqlite-utils/issues/35,511216664,MDEyOklzc3VlQ29tbWVudDUxMTIxNjY2NA==,9599,simonw,2019-07-14T16:29:22Z,2019-07-14T16:29:22Z,OWNER,"Should this support both dictionaries AND keyword arguments? ```python db[""events_venues""].update((3, 2), custom_label=""Label"") ``` Pros: this is quite nice to look at. Cons: would have to continue supporting dictionaries as well, since otherwise you won't be able to update columns with names that are not valid Python identifiers. So now there are two ways. I'm going to stick with dictionaries only for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",462430920,table.update(...) method, https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987349633,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987349633,IC_kwDOCGYnMM462cKB,9599,simonw,2021-12-06T23:19:28Z,2021-12-06T23:19:28Z,OWNER,(I ended up not needing this here since `.lookup()` already creates a unique index on `_item_id` for you. Still could be a useful feature though.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables, https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987458772,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987458772,IC_kwDOCGYnMM4622zU,9599,simonw,2021-12-07T01:00:41Z,2021-12-07T01:00:41Z,OWNER,"I think the syntax design of this looks like: ```python item_pk = db[item_table].lookup( {""_item_id"": item_id}, item_to_insert, column_order=(""_id"", ""_item_id""), pk=""_id"", indexes=(""_version"",), ) ``` So it's a sequence of column names... or a sequence of tuples for creating compound indexes: ```python db[""dogs""].insert( {""name"": ""Cleo"", ""species"": ""Mutt"", ""hobbies"": ""Raiding picnics""}, indexes=((""name"", ""species""), ""hobbies""), ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables, https://github.com/simonw/sqlite-utils/issues/349#issuecomment-987461427,https://api.github.com/repos/simonw/sqlite-utils/issues/349,987461427,IC_kwDOCGYnMM4623cz,9599,simonw,2021-12-07T01:03:43Z,2021-12-07T01:04:37Z,OWNER,"In terms of types, I think that means it looks like this: ```python IndexesType = Iterable[ Union[str, Iterable[str]] ] def create( self, columns: Dict[str, Any], pk: Optional[Any] = None, ... indexes: Optional[IndexesType] = None, ): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1071531082,A way of creating indexes on newly created tables, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983120066,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983120066,IC_kwDOCGYnMM46mTjC,9599,simonw,2021-11-30T23:25:52Z,2021-11-30T23:26:11Z,OWNER,"Maybe this: sqlite-utils create-database my.db With options that include `--enable-wal` (actually that's the only option I can think of). This is consistent with the existing `create-table` and `create-view` commands. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983122733,https://api.github.com/repos/simonw/sqlite-utils/issues/348,983122733,IC_kwDOCGYnMM46mUMt,9599,simonw,2021-11-30T23:31:17Z,2021-11-30T23:31:17Z,OWNER,"Potential other options could include: - `--page-size` https://www.sqlite.org/pragma.html#pragma_page_size","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008367607,https://api.github.com/repos/simonw/sqlite-utils/issues/348,1008367607,IC_kwDOCGYnMM48Gnf3,9599,simonw,2022-01-09T20:22:43Z,2022-01-09T20:22:43Z,OWNER,I'm not going to implement `--page-size` unless someone specifically requests it - I don't like having features that I've never needed to use myself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008383293,https://api.github.com/repos/simonw/sqlite-utils/issues/348,1008383293,IC_kwDOCGYnMM48GrU9,9599,simonw,2022-01-09T20:38:17Z,2022-01-09T20:38:17Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#creating-an-empty-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1011855133,https://api.github.com/repos/simonw/sqlite-utils/issues/348,1011855133,IC_kwDOCGYnMM48T68d,9599,simonw,2022-01-13T07:06:59Z,2022-01-13T07:06:59Z,OWNER,Wrote a lot more about this feature here: https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/,"{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1067771698,Command for creating an empty database, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982126665,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982126665,IC_kwDOCGYnMM46ihBJ,9599,simonw,2021-11-29T23:26:01Z,2021-11-29T23:33:48Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/93b059dd230eae9eaae472b7fbabd4a66feeb79d/.github/workflows/test.yml#L11-L20 This configuration means that the numpy=0, Python=3.10, os=Ubuntu build will additionally use `pysqlite3` with the SQLite 3.37.0. It's failing right now: https://github.com/simonw/sqlite-utils/runs/4360593156 - because `pysqlite3` doesn't provide `.iterdump()`. I can use the workaround from this comment: https://github.com/coleifer/pysqlite3/issues/24#issuecomment-982081267","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129218,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982129218,IC_kwDOCGYnMM46ihpC,9599,simonw,2021-11-29T23:31:02Z,2021-11-29T23:31:02Z,OWNER,Here's the test run that's installing `pysqlite3` and that version of SQLite: https://github.com/simonw/sqlite-utils/runs/4360663292,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982129727,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982129727,IC_kwDOCGYnMM46ihw_,9599,simonw,2021-11-29T23:31:58Z,2021-11-29T23:31:58Z,OWNER,"It failed on other Python versions with `mypy`: ``` sqlite_utils/utils.py:8: error: Cannot find implementation or library stub for module named ""sqlite3.dump"" sqlite_utils/utils.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982133970,IC_kwDOCGYnMM46iizS,9599,simonw,2021-11-29T23:41:17Z,2021-11-29T23:41:17Z,OWNER,"Took a bit of experimenting to get both `mypy` AND `flake8` to ignore the same line. The incantation that worked was this one: https://github.com/simonw/sqlite-utils/blob/f990e134aa8219b687ff6c261330f36824b5df36/sqlite_utils/utils.py#L8 Order here matters - this did NOT work for both tools: ```python from sqlite3.dump import _iterdump as iterdump # noqa: F401 # type: ignore ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982136747,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982136747,IC_kwDOCGYnMM46ijer,9599,simonw,2021-11-29T23:48:05Z,2021-11-29T23:48:05Z,OWNER,"Some interesting test failures in the version that runs with `pysqlite3`: ``` =========================== short test summary info ============================ FAILED tests/test_cli.py::test_enable_wal - assert 0 == 1 FAILED tests/test_cli.py::test_disable_wal - pysqlite3.dbapi2.OperationalErro... FAILED tests/test_fts.py::test_rebuild_fts[searchable] - pysqlite3.dbapi2.Dat... FAILED tests/test_fts.py::test_rebuild_fts[searchable_fts] - pysqlite3.dbapi2... FAILED tests/test_wal.py::test_enable_disable_wal - pysqlite3.dbapi2.Operatio... ================== 5 failed, 750 passed, 3 skipped in 15.20s =================== ``` https://github.com/simonw/sqlite-utils/runs/4360759085 The WAL errors look like this: ``` E pysqlite3.dbapi2.OperationalError: cannot change into wal mode from within a transaction ``` Triggered by a call to `db.enable_wal()` The FTS errors are caused by tests that try to deliberately corrupt the FTS index by running `fresh_db[""searchable_fts_data""].delete_where()` - and then rebuilding it using `rebuild_fts()`: ``` @pytest.mark.parametrize(""table_to_fix"", [""searchable"", ""searchable_fts""]) def test_rebuild_fts(fresh_db, table_to_fix): table = fresh_db[""searchable""] table.insert(search_records[0]) table.enable_fts([""text"", ""country""]) # Run a search rows = list(table.search(""tanuki"")) assert len(rows) == 1 assert { ""rowid"": 1, ""text"": ""tanuki are running tricksters"", ""country"": ""Japan"", ""not_searchable"": ""foo"", }.items() <= rows[0].items() # Delete from searchable_fts_data fresh_db[""searchable_fts_data""].delete_where() # This should have broken the index with pytest.raises(sqlite3.DatabaseError): list(table.search(""tanuki"")) # Running rebuild_fts() should fix it > fresh_db[table_to_fix].rebuild_fts() tests/test_fts.py:277: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sqlite_utils/db.py:1947: in rebuild_fts self.db.execute( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = > sql = ""INSERT INTO [searchable_fts]([searchable_fts]) VALUES('rebuild');"" parameters = None def execute( self, sql: str, parameters: Optional[Union[Iterable, dict]] = None ) -> sqlite3.Cursor: ""Execute SQL query and return a ``sqlite3.Cursor``."" if self._tracer: self._tracer(sql, parameters) if parameters is not None: return self.conn.execute(sql, parameters) else: > return self.conn.execute(sql) E pysqlite3.dbapi2.DatabaseError: database disk image is malformed ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137293,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982137293,IC_kwDOCGYnMM46ijnN,9599,simonw,2021-11-29T23:49:29Z,2021-11-29T23:49:29Z,OWNER,"A short term fix would be to skip those tests against `pysqlite3` - but longer term it would be good to address the underlying issue, particularly for the WAL ones (the FTS ones aren't too worrying since if you deliberately try and break the FTS table it's not hugely problematic if you corrupt your database).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982137888,https://api.github.com/repos/simonw/sqlite-utils/issues/347,982137888,IC_kwDOCGYnMM46ijwg,9599,simonw,2021-11-29T23:50:54Z,2021-11-29T23:50:54Z,OWNER,If I'm going to `skipIf()` those tests I need a way to check if `pysqlite3` is being used.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/pull/347#issuecomment-991397907,https://api.github.com/repos/simonw/sqlite-utils/issues/347,991397907,IC_kwDOCGYnMM47F4gT,9599,simonw,2021-12-11T01:01:40Z,2021-12-11T01:01:40Z,OWNER,The change I made to that test in #354 might help with this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066603133,Test against pysqlite3 running SQLite 3.37, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982077873,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982077873,IC_kwDOCGYnMM46iVGx,9599,simonw,2021-11-29T22:22:05Z,2021-11-29T22:22:05Z,OWNER,"Ideally I'd like an extra set of matrix options for different versions of SQLite. I can use `pysqlite3` for this, but it isn't a completely compatible drop-in replacement - turns out it doesn't support the `iterdump()` method for example, see https://github.com/coleifer/pysqlite3/issues/24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982078527,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982078527,IC_kwDOCGYnMM46iVQ_,9599,simonw,2021-11-29T22:23:03Z,2021-11-29T22:23:03Z,OWNER,"Here's a modified version of the `dump` command that works even with `pysqlite3`: ```python @cli.command() @click.argument( ""path"", type=click.Path(exists=True, file_okay=True, dir_okay=False, allow_dash=False), required=True, ) @load_extension_option def dump(path, load_extension): """"""Output a SQL dump of the schema and full contents of the database"""""" db = sqlite_utils.Database(path) _load_extensions(db, load_extension) # pysqlite3 doesn't implement .iterdump() from sqlite3.dump import _iterdump for line in _iterdump(db.conn): click.echo(line) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982094020,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982094020,IC_kwDOCGYnMM46iZDE,9599,simonw,2021-11-29T22:50:11Z,2021-11-29T22:50:11Z,OWNER,"For the moment I think I'll combine two problems into one, and just add a single matrix alternative that uses `pysqlite3` running SQLite 3.37.0 - only on macOS and Linux so I don't have to figure out how to compile it for Windows.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982094370,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982094370,IC_kwDOCGYnMM46iZIi,9599,simonw,2021-11-29T22:50:49Z,2021-11-29T22:50:49Z,OWNER,I have a working recipe for compiling it for macOS here: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982006544,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/346#issuecomment-982111751,https://api.github.com/repos/simonw/sqlite-utils/issues/346,982111751,IC_kwDOCGYnMM46idYH,9599,simonw,2021-11-29T23:11:17Z,2021-11-29T23:12:49Z,OWNER,"To keep things simple for the moment I'm only going to add one extra thing to the matrix, and it will be a run of the tests against SQLite 3.37.0 using pysqlite3 on Linux only. I can use this mechanism: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#example-including-new-combinations","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066563554,Way to test SQLite 3.37 (and potentially other versions) in CI, https://github.com/simonw/sqlite-utils/issues/345#issuecomment-982090895,https://api.github.com/repos/simonw/sqlite-utils/issues/345,982090895,IC_kwDOCGYnMM46iYSP,9599,simonw,2021-11-29T22:44:36Z,2021-11-29T22:44:36Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#strict,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066501534,`table.strict` introspection boolean for identifying STRICT mode tables, https://github.com/simonw/sqlite-utils/issues/345#issuecomment-982091363,https://api.github.com/repos/simonw/sqlite-utils/issues/345,982091363,IC_kwDOCGYnMM46iYZj,9599,simonw,2021-11-29T22:45:26Z,2021-11-29T22:45:26Z,OWNER,This is the implementation: https://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L1236-L1241,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066501534,`table.strict` introspection boolean for identifying STRICT mode tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-981997973,https://api.github.com/repos/simonw/sqlite-utils/issues/344,981997973,IC_kwDOCGYnMM46iBmV,9599,simonw,2021-11-29T20:33:52Z,2021-11-29T20:33:52Z,OWNER,"From that page: > If you try to open a database containing the STRICT keyword in an earlier version of SQLite, it will not recognize the keyword and will report an error (except as noted below. > > [...] > > Because of a quirk in the SQL language parser, versions of SQLite prior to 3.37.0 can still read and write STRICT tables if they set ""PRAGMA writable_schema=ON"" immediately after opening the database file, prior to doing anything else that requires knowing the schema.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-981999025,https://api.github.com/repos/simonw/sqlite-utils/issues/344,981999025,IC_kwDOCGYnMM46iB2x,9599,simonw,2021-11-29T20:34:38Z,2021-11-29T20:35:58Z,OWNER,"I'm going to build my own `pysqlite3` wheel with the latest SQLite to try this out, following the instructions on https://github.com/coleifer/pysqlite3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982006544,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982006544,IC_kwDOCGYnMM46iDsQ,9599,simonw,2021-11-29T20:44:37Z,2021-11-29T20:48:43Z,OWNER,"This worked: ``` cd /tmp mkdir sqlite-3.37 cd sqlite-3.37 wget 'https://www.sqlite.org/2021/sqlite-amalgamation-3370000.zip' unzip sqlite-amalgamation-3370000.zip git clone https://github.com/coleifer/pysqlite3/ cp sqlite-amalgamation-3370000/sqlite3.[ch] pysqlite3 cd pysqlite3 python3 setup.py build_static build bdist_wheel ``` This gave me a file here: ``` pysqlite3 % ls -l dist total 1872 -rw-r--r-- 1 simon wheel 956557 Nov 29 12:38 pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl ``` That wheel only works when installed for Python 3.9 (it failed to install in a Python 3.10 virtual environment) - but `pip install /tmp/sqlite-3.37/pysqlite3/dist/pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl` gave me a working `pysqlite3` - and the following worked: ```pycon >>> import pysqlite3 >>> pysqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchall() [('3.37.0',)] ``` And if I install `sqlite-utils` in the same virtual environment this works: ``` % sqlite-utils memory 'select sqlite_version()' [{""sqlite_version()"": ""3.37.0""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982014776,IC_kwDOCGYnMM46iFs4,9599,simonw,2021-11-29T20:55:19Z,2021-11-29T20:55:19Z,OWNER,"There are a few places that the `strict=True` option could go: - `table.create()` and `table.create_table_sql()` - The `Database()` constructor, to turn it on for all created tables - The `.insert()` / `.insert_all()` etc family of methods that can implicitly create tables I'll definitely implement the first one, and likely the second one too. I'm on the fence with regards to the third one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982016594,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982016594,IC_kwDOCGYnMM46iGJS,9599,simonw,2021-11-29T20:57:42Z,2021-11-29T20:57:42Z,OWNER,"What should happen if you attempt to use `strict=True` against a SQLite version prior to 3.37.0? An obvious error would be best... but how about silently ignoring it on older versions instead? That would match how we handle `deterministic=True` for registering functions: https://github.com/simonw/sqlite-utils/blob/126703706ea153f63e6134ad14e5712e4bbcb8ae/sqlite_utils/db.py#L372-L380 https://github.com/simonw/sqlite-utils/blob/93c7fd9868fed3193a1732b39bfac539e5812b0b/tests/test_register_function.py#L34-L37 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982017994,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982017994,IC_kwDOCGYnMM46iGfK,9599,simonw,2021-11-29T20:59:37Z,2021-11-29T20:59:37Z,OWNER,"I'm leaning towards silently ignore if SQLite version doesn't support it. That means that the first time you attempt to use `strict=True` we will need to run a test against the database connection to see what version of SQLite it uses, then cache the result to avoid making the same call again for the same connection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982018304,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982018304,IC_kwDOCGYnMM46iGkA,9599,simonw,2021-11-29T21:00:02Z,2021-11-29T21:00:02Z,OWNER,Is there a need for an introspection function for telling if a SQLite table is in strict mode or not? How would that work?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982020757,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982020757,IC_kwDOCGYnMM46iHKV,9599,simonw,2021-11-29T21:03:34Z,2021-11-29T21:03:34Z,OWNER,"From the STRICT docs: > The SQLite parser accepts a comma-separated list of table options after the final close parenthesis in a CREATE TABLE statement. As of this writing (2021-08-23) only two options are recognized: > > - STRICT > - [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) So I think I need to read the `CREATE TABLE` statement from the `sqlite_master` table, split on the last `)`, split those tokens on `,` and see if `create` is in there (case insensitive).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982026918,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982026918,IC_kwDOCGYnMM46iIqm,9599,simonw,2021-11-29T21:11:42Z,2021-11-29T21:16:31Z,OWNER,"Made myself a test strict table like so: ```pycon >>> import pysqlite3 >>> conn = pysqlite3.connect(""/tmp/strict-table.db"") >>> conn.execute(""create table foo (id integer primary key, name text, weight real) strict"") >>> cursor = conn.cursor() >>> with conn: ... cursor.execute(""insert into foo (name, weight) values ('Lila', '2.31')"") >>> conn.close() ``` Uploaded that to: https://static.simonwillison.net/static/2021/strict-table.db","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982048918,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982048918,IC_kwDOCGYnMM46iOCW,9599,simonw,2021-11-29T21:40:42Z,2021-11-29T21:40:42Z,OWNER,"Here's a function that detects if `strict` is supported or not: ```python import secrets import sqlite3 def supports_strict_tables(db = None): if db is None: db = sqlite3.connect("":memory:"") try: table_name = 't{}'.format(secrets.token_hex(16)) with db: db.execute(""create table {} (name text) strict"".format(table_name)) db.execute(""drop table {}"".format(table_name)) return True except: return False ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982049148,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982049148,IC_kwDOCGYnMM46iOF8,9599,simonw,2021-11-29T21:40:59Z,2021-11-29T21:40:59Z,OWNER,I'm going to add that as `db.supports_strict`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076702,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982076702,IC_kwDOCGYnMM46iU0e,9599,simonw,2021-11-29T22:20:22Z,2021-11-29T22:20:22Z,OWNER,"I haven't documented `db.supports_strict` yet (I documented `table.strict`) because there wasn't an obvious section of the documentation for it. I need to remember to document it once I add documentation for the `strict=True` parameter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076924,https://api.github.com/repos/simonw/sqlite-utils/issues/344,982076924,IC_kwDOCGYnMM46iU38,9599,simonw,2021-11-29T22:20:44Z,2021-11-29T22:20:44Z,OWNER,Need to figure out a good pattern for testing this in CI too - it will currently skip the new tests if it doesn't have SQLite 3.37 or higher.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables, https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055854884,https://api.github.com/repos/simonw/sqlite-utils/issues/343,1055854884,IC_kwDOCGYnMM4-7xEk,9599,simonw,2022-03-01T21:03:45Z,2022-03-01T21:03:45Z,OWNER,"Just found myself needing this capability myself! Relevant code: https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2297-L2307 https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2996-L3001 So various functions could grow a `hash_id_columns=(""title"", ""date"")` argument which causes just those columns to be included in the hash. Bonus: if you use `hash_id_columns=...` without setting `hash_id=""id""` it could assume that you want the column to be called `id`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1063388037,Provide function to generate hash_id from specified columns, https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055855845,https://api.github.com/repos/simonw/sqlite-utils/issues/343,1055855845,IC_kwDOCGYnMM4-7xTl,9599,simonw,2022-03-01T21:04:45Z,2022-03-01T22:43:38Z,OWNER,"I'm going to make that `_hash()` utility function a documented, non-underscore-prefixed function too - called `hash_record()`.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1063388037,Provide function to generate hash_id from specified columns, https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055991226,https://api.github.com/repos/simonw/sqlite-utils/issues/343,1055991226,IC_kwDOCGYnMM4-8SW6,9599,simonw,2022-03-02T00:03:47Z,2022-03-02T00:03:47Z,OWNER,"Oops, broke mypy: ``` sqlite_utils/db.py:2600: error: Incompatible default for argument ""hash_id_columns"" (default has type ""Default"", argument has type ""Optional[Iterable[str]]"") Found 1 error in 1 file (checked 49 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1063388037,Provide function to generate hash_id from specified columns, https://github.com/simonw/sqlite-utils/issues/343#issuecomment-1055992544,https://api.github.com/repos/simonw/sqlite-utils/issues/343,1055992544,IC_kwDOCGYnMM4-8Srg,9599,simonw,2022-03-02T00:06:10Z,2022-03-02T00:06:10Z,OWNER,"Updated documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents Documentation for the renamed `utils.hash_record()` function: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-hash-record","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1063388037,Provide function to generate hash_id from specified columns, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973800795,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973800795,IC_kwDOCGYnMM46CwVb,9599,simonw,2021-11-19T06:54:08Z,2021-11-19T06:54:08Z,OWNER,"Looking at the code for `lookup()` it currently hard-codes `pk` to `""id""` - but it actually only calls `.insert()` in two places, both of which could be passed extra arguments. https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2756-L2763","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973801650,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973801650,IC_kwDOCGYnMM46Cwiy,9599,simonw,2021-11-19T06:55:56Z,2021-11-19T06:55:56Z,OWNER,"`pk` needs to be an explicit argument to `.lookup()`. The rest could be `**kwargs` passed through to `.insert()`, like this hacked together version (docstring removed for brevity): ```python def lookup( self, lookup_values: Dict[str, Any], extra_values: Optional[Dict[str, Any]] = None, pk=""id"", **insert_kwargs, ): """""" assert isinstance(lookup_values, dict) if extra_values is not None: assert isinstance(extra_values, dict) combined_values = dict(lookup_values) if extra_values is not None: combined_values.update(extra_values) if self.exists(): self.add_missing_columns([combined_values]) unique_column_sets = [set(i.columns) for i in self.indexes] if set(lookup_values.keys()) not in unique_column_sets: self.create_index(lookup_values.keys(), unique=True) wheres = [""[{}] = ?"".format(column) for column in lookup_values] rows = list( self.rows_where( "" and "".join(wheres), [value for _, value in lookup_values.items()] ) ) try: return rows[0][pk] except IndexError: return self.insert(combined_values, pk=pk, **insert_kwargs).last_pk else: pk = self.insert(combined_values, pk=pk, **insert_kwargs).last_pk self.create_index(lookup_values.keys(), unique=True) return pk ``` I think I'll explicitly list the parameters, mainly so they can be typed and covered by automatic documentation. I do worry that I'll add more keyword arguments to `.insert()` in the future and forget to mirror them to `.lookup()` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802308,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802308,IC_kwDOCGYnMM46CwtE,9599,simonw,2021-11-19T06:57:37Z,2021-11-19T06:57:37Z,OWNER,"Here's the current full method signature for `.insert()`: https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2462-L2477 I could add a test which uses introspection (`inspect.signature(method).parameters`) to confirm that `.lookup()` has a super-set of the arguments accepted by `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802469,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802469,IC_kwDOCGYnMM46Cwvl,9599,simonw,2021-11-19T06:58:03Z,2021-11-19T06:58:03Z,OWNER,Also: I don't think `ignore=` and `replace=` make sense in the context of `lookup()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802766,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802766,IC_kwDOCGYnMM46Cw0O,9599,simonw,2021-11-19T06:58:45Z,2021-11-19T06:58:45Z,OWNER,"And neither does `hash_id`. On that basis I'm going to specifically list the ones that DO make sense, and hope that I remember to add any new ones in the future. I can add a code comment hint to `.insert()` about that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802998,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802998,IC_kwDOCGYnMM46Cw32,9599,simonw,2021-11-19T06:59:22Z,2021-11-19T06:59:32Z,OWNER,"I don't think I need the `DEFAULT` defaults for `.insert()` either, since it just passes through to `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973820125,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973820125,IC_kwDOCGYnMM46C1Dd,9599,simonw,2021-11-19T07:25:55Z,2021-11-19T07:25:55Z,OWNER,"`alter=True` doesn't make sense to support here either, because `.lookup()` already adds missing columns: https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2743-L2746","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`, https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434425,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968434425,IC_kwDOCGYnMM45uSL5,9599,simonw,2021-11-15T01:42:36Z,2021-11-15T01:42:36Z,OWNER,"Here's the current signature of `table.lookup()`: https://github.com/simonw/sqlite-utils/blob/9cda5b070f885a7995f0c307bcc4f45f0812994a/sqlite_utils/db.py#L2716-L2729 I'm going to add a second positional argument which can provide a dictionary of column->value to use when creating the original table and populating the initial row. If the row already exists, those columns will be ignored entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record, https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434594,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968434594,IC_kwDOCGYnMM45uSOi,9599,simonw,2021-11-15T01:43:10Z,2021-11-15T01:43:10Z,OWNER,What should I call this parameter? Django has a similar feature where it calls them `defaults=` (for `get_or_create()`) but I'm not a huge fan of that name.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record, https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968435041,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968435041,IC_kwDOCGYnMM45uSVh,9599,simonw,2021-11-15T01:44:42Z,2021-11-15T01:44:42Z,OWNER,"`lookup(column_values, extra_values)` is one option. `column_values` isn't actually a great name for the first parameter any more, since the second parameter also takes column values. The first parameter is now all about the unique lookup values. Maybe this: lookup(lookup_values, extra_values)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record, https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968450579,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968450579,IC_kwDOCGYnMM45uWIT,9599,simonw,2021-11-15T02:02:34Z,2021-11-15T02:02:34Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/docs/python-api.rst#working-with-lookup-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record, https://github.com/simonw/sqlite-utils/issues/336#issuecomment-962411119,https://api.github.com/repos/simonw/sqlite-utils/issues/336,962411119,IC_kwDOCGYnMM45XTpv,9599,simonw,2021-11-06T07:21:04Z,2021-11-06T07:21:04Z,OWNER,I've never used `DEFAULT 'CURRENT_TIMESTAMP'` myself so this one should be an interesting bug to explore.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1044267332,"sqlite-util tranform --column-order mangles columns of type ""timestamp""", https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359137,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968359137,IC_kwDOCGYnMM45t_zh,9599,simonw,2021-11-14T20:37:00Z,2021-11-14T20:37:00Z,OWNER,This is strange - the code already checks that an index doesn't exist before attempting to create it: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L893-L902,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359868,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968359868,IC_kwDOCGYnMM45t_-8,9599,simonw,2021-11-14T20:41:42Z,2021-11-14T20:41:42Z,OWNER,"The ""index idx_generators_eia860_report_date already exists"" error suggests that the problem here is actually one of an index name collision. ```python table.create_index([fk.column]) ``` This will derive a name for the index automatically from the name of the table and the name of the passed in columns: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1536-L1539 So perhaps `.create_index()` should grow an extra option that creates the index even if the name already exists, by finding a new name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360387,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968360387,IC_kwDOCGYnMM45uAHD,9599,simonw,2021-11-14T20:45:44Z,2021-11-14T20:45:44Z,OWNER,"What would such an option be called? Some options: - `table.create_index([fk.column], force=True)` - not obvious what `force` means here - `table.create_index([fk.column], ignore_existing_name=True)` - not obvious what `ignore` means here - `table.create_index([fk.column], pick_unique_name=True)` - bit verbose If the user doesn't pass in an explicit name it seems like their intent is ""just create me the index, I don't care what name you use"" - so actually perhaps the default behaviour here should be to pick a new unique name if that name is already in use. Then maybe there should be an option to disable that - some options there: - `table.create_index([fk.column], error_on_existing_index_name=True)` - too verbose - `table.create_index([fk.column], force=False)` - not clear what `force` means ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360538,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968360538,IC_kwDOCGYnMM45uAJa,9599,simonw,2021-11-14T20:46:56Z,2021-11-14T20:46:56Z,OWNER,"I'm tempted to not provide an opt-out option either: if you call `table.create_index(...)` without specifying an index name I think the tool should create the index for you, quietly picking an index name that works. But... it feels wasteful to create an index that exactly duplicates an existing index. Would SQLite even let you do that or would it notice and NOT double the amount of disk space used for that index?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361285,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361285,IC_kwDOCGYnMM45uAVF,9599,simonw,2021-11-14T20:51:57Z,2021-11-14T20:51:57Z,OWNER,"SQLite will happily create multiple identical indexes on a table, using more disk space each time: ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(""dupes.db"") >>> db[""t""].insert_all({""id"": i} for i in range(10000))
# dupes.db is 98304 bytes >>> db[""t""].create_index([""id""])
# dupes.db is 204800 bytes >>> db[""t""].indexes [Index(seq=0, name='idx_t_id', unique=0, origin='c', partial=0, columns=['id'])] >>> db[""t""].create_index([""id""], index_name=""t_idx_t_id_2"")
# 311296 bytes >>> db[""t""].create_index([""id""], index_name=""t_idx_t_id_3"")
# 417792 bytes >>> db.vacuum() # Still 417792 bytes ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361409,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361409,IC_kwDOCGYnMM45uAXB,9599,simonw,2021-11-14T20:52:55Z,2021-11-14T20:52:55Z,OWNER,"Looking at the method signature: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1518-L1524 `if_not_exists` just adds a `IF NOT EXISTS` clause here: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1549-L1561","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361671,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361671,IC_kwDOCGYnMM45uAbH,9599,simonw,2021-11-14T20:54:53Z,2021-11-14T21:01:14Z,OWNER,"I'm leaning towards `table.create_index(columns, ignore_existing_name=True)` now. Or `resolve_existing_name` - or `skip_existing_name`? ""ignore"" sounds like it might not create the index if the name exists, but we want to still create the index but pick a new name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362214,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968362214,IC_kwDOCGYnMM45uAjm,9599,simonw,2021-11-14T20:59:15Z,2021-11-14T20:59:15Z,OWNER,"How to figure out if an index name is already in use? `PRAGMA index_list(t)` requires a table name. This does it: ```sql SELECT name FROM sqlite_master WHERE type = 'index'; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362285,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968362285,IC_kwDOCGYnMM45uAkt,9599,simonw,2021-11-14T20:59:44Z,2021-11-14T20:59:44Z,OWNER,I think I'll attempt to create the index and re-try if it fails with that error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968371112,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968371112,IC_kwDOCGYnMM45uCuo,9599,simonw,2021-11-14T21:57:43Z,2021-11-14T22:21:31Z,OWNER,"`create_index(..., find_unique_name=)` is good. Default to false. `index_foreign_keys` can set it to true.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968380387,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968380387,IC_kwDOCGYnMM45uE_j,9599,simonw,2021-11-14T22:55:56Z,2021-11-14T22:55:56Z,OWNER,"OK, this should fix it.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index, https://github.com/simonw/sqlite-utils/pull/333#issuecomment-974754412,https://api.github.com/repos/simonw/sqlite-utils/issues/333,974754412,IC_kwDOCGYnMM46GZJs,9599,simonw,2021-11-21T04:35:32Z,2021-11-21T04:35:32Z,OWNER,"Some other recent projects (like trying to get this library to work in JupyterLite) have made me much more cautious about adding new dependencies, especially dependencies like `pyarrow` which require custom C/Rust extensions. There are a few ways this could work though: - Have this as an optional dependency feature - so it only works if the user installs `pyarrow` as well - Implement this as a separate tool, `parquet-to-sqlite` - which could itself depend on `sqlite-utils` - Add a concept of ""plugins"" to `sqlite-utils`, similar to how those work in Datasette: https://docs.datasette.io/en/stable/plugins.html My favourite option is `parquet-to-sqlite` because that can be built without any additional changes to `sqlite-utils` at all! I find the concept of plugins for `sqlite-utils` interesting. I've so far not had quite enough potential use-cases to convince me this is worthwhile (especially since it should be very easy to build out separate tools entirely), but I'm ready to be convinced that a plugin mechanism would be worthwhile.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1039037439,Add functionality to read Parquet files., https://github.com/simonw/sqlite-utils/pull/333#issuecomment-979442854,https://api.github.com/repos/simonw/sqlite-utils/issues/333,979442854,IC_kwDOCGYnMM46YRym,9599,simonw,2021-11-25T19:47:26Z,2021-11-25T19:47:26Z,OWNER,"I just remembered that there's one other place that this could fit: as a Datasette ""insert"" plugin. This is vaporware at the moment, but the idea is that Datasette itself could grow a mechanism for importing data, that's driven by plugins. Out of the box Datasette would be able to import CSV and CSV files, similar to `sqlite-utils insert ... --csv` - but plugins would then be able to add support for additional format such as GeoJSON or - in this case - Parquet. The neat thing about having it as a Datasette plugin is that one plugin would enable three different ways of importing data: 1. Via a new `datasette insert ...` CLI option (similar to `sqlite-utils`) 2. Via a web form upload interface, where authenticated Datasette users would be able to upload files 3. Via an API interface, where files could be programatically submitted to a running Datasette server I started fleshing out this idea quite a while ago but didn't make much concrete progress, maybe I should revisit it: - https://github.com/simonw/datasette/issues/1160","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1039037439,Add functionality to read Parquet files., https://github.com/simonw/sqlite-utils/issues/332#issuecomment-968380675,https://api.github.com/repos/simonw/sqlite-utils/issues/332,968380675,IC_kwDOCGYnMM45uFED,9599,simonw,2021-11-14T22:57:56Z,2021-11-14T22:57:56Z,OWNER,This is a great idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1028056713,`sqlite-utils memory --flatten` option to flatten nested JSON, https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968381939,https://api.github.com/repos/simonw/sqlite-utils/issues/331,968381939,IC_kwDOCGYnMM45uFXz,9599,simonw,2021-11-14T23:06:20Z,2021-11-14T23:06:20Z,OWNER,Thanks - I didn't know this was needed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026794056,Mypy error: found module but no type hints or library stubs, https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968384005,https://api.github.com/repos/simonw/sqlite-utils/issues/331,968384005,IC_kwDOCGYnMM45uF4F,9599,simonw,2021-11-14T23:19:29Z,2021-11-14T23:20:32Z,OWNER,"Tested it like this, against a freshly built `.tar.gz` package from my development environment: ``` (w) w % mypy . hello.py:1: error: Skipping analyzing ""sqlite_utils"": found module but no type hints or library stubs hello.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 1 source file) (w) w % pip install ~/Dropbox/Development/sqlite-utils/dist/sqlite-utils-3.17.1.tar.gz Processing /Users/simon/Dropbox/Development/sqlite-utils/dist/sqlite-utils-3.17.1.tar.gz ... Successfully installed sqlite-utils-3.17.1 (w) w % mypy . Success: no issues found in 1 source file ``` I tested against the `.whl` too. My `hello.py` script contained this: ```python import sqlite_utils from typing import cast if __name__ == ""__main__"": db = sqlite_utils.Database(memory=True) table = cast(sqlite_utils.db.Table, db[""foo""]) table.insert({""id"": 5}) print(list(db.query(""select * from foo""))) ``` That `cast()` is necessary because without it you get this error: ``` (w) w % mypy . hello.py:7: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""insert"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026794056,Mypy error: found module but no type hints or library stubs, https://github.com/simonw/sqlite-utils/issues/33#issuecomment-507077631,https://api.github.com/repos/simonw/sqlite-utils/issues/33,507077631,MDEyOklzc3VlQ29tbWVudDUwNzA3NzYzMQ==,9599,simonw,2019-06-30T23:53:34Z,2019-06-30T23:53:34Z,OWNER,This was needed for https://github.com/simonw/db-to-sqlite/issues/12,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",462423839,index_foreign_keys / index-foreign-keys utilities, https://github.com/simonw/sqlite-utils/issues/33#issuecomment-507077672,https://api.github.com/repos/simonw/sqlite-utils/issues/33,507077672,MDEyOklzc3VlQ29tbWVudDUwNzA3NzY3Mg==,9599,simonw,2019-06-30T23:54:11Z,2019-06-30T23:54:11Z,OWNER,Released in 1.4: https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-4,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",462423839,index_foreign_keys / index-foreign-keys utilities, https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926203501,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926203501,IC_kwDOCGYnMM43NL5t,9599,simonw,2021-09-23T22:15:07Z,2021-09-23T22:15:30Z,OWNER,"Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full: > I'm not sure what to do about this one. > > I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again. > > So... I could do one of the following: > > * Throw an error if a column name includes those characters. That's my preferred option I think. > * Automatically replace `[` in column names with `(` and `]` with `)` > * Do the automatic replacement but show a user-visible warning when I do it > * Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix. > > > Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926204046,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926204046,IC_kwDOCGYnMM43NMCO,9599,simonw,2021-09-23T22:16:18Z,2021-09-23T22:16:18Z,OWNER,"So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926205047,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926205047,IC_kwDOCGYnMM43NMR3,9599,simonw,2021-09-23T22:18:34Z,2021-09-23T22:19:38Z,OWNER,"Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948 It's called from three different methods in `db.py`: `create_table_sql()`, `update()` and `insert_all()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206220,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926206220,IC_kwDOCGYnMM43NMkM,9599,simonw,2021-09-23T22:20:55Z,2021-09-23T22:21:19Z,OWNER,"If I add a new parameter for opting in and out of fixing these, what should it be called? A few options: - `fix_columns=False` - `rename_bad_columns=False` - `fix_column_names=False` - `error_on_invalid_column_names=True`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206705,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926206705,IC_kwDOCGYnMM43NMrx,9599,simonw,2021-09-23T22:21:58Z,2021-09-23T22:21:58Z,OWNER,"I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207246,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926207246,IC_kwDOCGYnMM43NM0O,9599,simonw,2021-09-23T22:23:09Z,2021-09-23T22:23:09Z,OWNER,"What are my options for replacing those characters? - `[` becomes `(` and `]` becomes `)` - `[` and `]` are removed entirely - `[` and `]` both become `_`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207719,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926207719,IC_kwDOCGYnMM43NM7n,9599,simonw,2021-09-23T22:24:09Z,2021-09-23T22:24:09Z,OWNER,"I think I like the underscore option best. I don't like the idea of injecting surprise `( )` parenthesis, and having them vanish entirely could result in things like `item[price]` becoming `itemprice` which feels confusing. `item_price_` is a little ugly but I think I can live with it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926208819,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926208819,IC_kwDOCGYnMM43NNMz,9599,simonw,2021-09-23T22:26:54Z,2021-09-23T22:26:54Z,OWNER,"I could even have those replacement characters be properties of the `Database` class, so uses can sub-class and change them. ```python class Database: left_brace_replace = ""_"" right_brace_replace = ""_"" ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968451954,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968451954,IC_kwDOCGYnMM45uWdy,9599,simonw,2021-11-15T02:05:29Z,2021-11-15T02:05:29Z,OWNER,"> I could even have those replacement characters be properties of the `Database` class, so uses can sub-class and change them. I'm not going to do this, it's unnecessary extra complexity and it means the function that fixes the column names needs to have access to the current `Database` instance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968453129,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968453129,IC_kwDOCGYnMM45uWwJ,9599,simonw,2021-11-15T02:07:46Z,2021-11-15T02:07:46Z,OWNER,"If I replace `validate_column_names(row.keys())` with `fix_column_names(row)` I need to decide what to do about things like `pk=` and `column_order=`. What should the following do? ```python table.insert({""foo[bar]"": 4}, pk=""foo[bar]"", column_order=[""foo[bar]""]) ``` Should it spot the old column names in the `pk=` and `column_order=` parameters and pretend that `foo_bar_` was passed instead? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968458837,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968458837,IC_kwDOCGYnMM45uYJV,9599,simonw,2021-11-15T02:21:15Z,2021-11-15T02:21:15Z,OWNER,"I'm not going to implement a fix that rewrites the `pk` and `column_order` and other parameters - at least not yet. The main thing I'm trying to fix here is what happens when you attempt to import a CSV file with `[ ]` in the column names, which should be unaffected by that second challenge.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968470212,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968470212,IC_kwDOCGYnMM45ua7E,9599,simonw,2021-11-15T02:49:28Z,2021-11-15T02:49:28Z,OWNER,"I was going to replace all of the `validate_column_names()` bits with something that fixed them instead, but I think I have a better idea: I'm only going to apply the fix for the various '.insert()` methods that create the initial tables. I'll keep the `validate_column_names()` where they are at the moment. Once you've inserted the data and created the tables it will be up to you to use the new, correct column names. This avoids the whole issue of needing to rewrite parameters, and solves the immediate problem which is consuming CSV files with bad column names.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error), https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925292384,https://api.github.com/repos/simonw/sqlite-utils/issues/328,925292384,IC_kwDOCGYnMM43Jtdg,9599,simonw,2021-09-22T20:09:53Z,2021-09-22T20:09:53Z,OWNER,"Good catch, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1004613267,Invalid JSON output when no rows, https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925296085,https://api.github.com/repos/simonw/sqlite-utils/issues/328,925296085,IC_kwDOCGYnMM43JuXV,9599,simonw,2021-09-22T20:14:53Z,2021-09-22T20:14:53Z,OWNER,The bug is in this code: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/cli.py#L2205-L2227,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1004613267,Invalid JSON output when no rows, https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925300637,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925300637,IC_kwDOCGYnMM43Jved,9599,simonw,2021-09-22T20:21:26Z,2021-09-22T20:21:26Z,OWNER,"The `t1` and `t2` aliases were meant to handle this case, but the are no good if the tool throws an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name, https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925301981,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925301981,IC_kwDOCGYnMM43Jvzd,9599,simonw,2021-09-22T20:23:25Z,2021-09-22T20:23:25Z,OWNER,"Oddly I can't replicate this on macOS: ``` (sqlite-utils) sqlite-utils % ls foo/*.csv foo/bug.csv (sqlite-utils) sqlite-utils % ls bar/*.csv bar/bug.csv (sqlite-utils) sqlite-utils % sqlite-utils memory foo/bug.csv bar/bug.csv --schema CREATE TABLE ""bug"" ( [col1] TEXT, [col2] TEXT ); CREATE VIEW t1 AS select * from [bug]; CREATE VIEW t AS select * from [bug]; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name, https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925303497,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925303497,IC_kwDOCGYnMM43JwLJ,9599,simonw,2021-09-22T20:25:44Z,2021-09-22T20:25:44Z,OWNER,"Here's the relevant code: https://github.com/simonw/sqlite-utils/blob/7427a9137f60de961b6331d0922a3f03da0d1890/sqlite_utils/cli.py#L1289-L1292 I can fix this by checking to see if `csv_table` is already in use and adding a suffix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name, https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925321439,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925321439,IC_kwDOCGYnMM43J0jf,9599,simonw,2021-09-22T20:52:56Z,2021-09-22T20:52:56Z,OWNER,"Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json > If two files have the same name they will be assigned a numeric suffix: > > $ sqlite-utils memory foo/data.csv bar/data.csv ""select * from data_2""","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name, https://github.com/simonw/sqlite-utils/pull/324#issuecomment-968384988,https://api.github.com/repos/simonw/sqlite-utils/issues/324,968384988,IC_kwDOCGYnMM45uGHc,9599,simonw,2021-11-14T23:25:16Z,2021-11-14T23:25:16Z,OWNER,"Yes this was absolutely the intention! Thanks, I wonder how often I've made that mistake in other projects?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988013247,Use python-dateutil package instead of dateutils, https://github.com/simonw/sqlite-utils/issues/323#issuecomment-905886797,https://api.github.com/repos/simonw/sqlite-utils/issues/323,905886797,IC_kwDOCGYnMM41_rxN,9599,simonw,2021-08-25T21:25:18Z,2021-08-25T21:25:18Z,OWNER,"As far as I can tell the Python `sqlite3` module doesn't actually have a mechanism for de-registering a custom SQL function. This means that if I implement a mechanism whereby each call to `.convert()` registers a new SQL function with a random suffix (`convert_value_23424()` for example) those functions will stay registered - and if `.convert()` is called a large number of times the number of obsolete custom function registrations will grow without bounds. For that reason, I'm going to `wontfix` this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",979627285,`table.convert()` method should clean up after itself, https://github.com/simonw/sqlite-utils/pull/322#issuecomment-968401459,https://api.github.com/repos/simonw/sqlite-utils/issues/322,968401459,IC_kwDOCGYnMM45uKIz,9599,simonw,2021-11-15T00:26:42Z,2021-11-15T00:26:42Z,OWNER,"This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947 So the `COLUMN_TYPE_MAPPING` should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",979612115,Add dict type to be mapped as TEXT in sqllite, https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905037323,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905037323,IC_kwDOCGYnMM418cYL,9599,simonw,2021-08-24T23:15:29Z,2021-08-24T23:15:29Z,OWNER,"Huh, tests are failing but only on Windows!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905039576,IC_kwDOCGYnMM418c7Y,9599,simonw,2021-08-24T23:21:29Z,2021-08-24T23:21:29Z,OWNER,"Hah, the error here is actually: ``` > assert result.exit_code == 1, result.output E AssertionError: E E assert 0 == 1 E + where 0 = .exit_code ``` So I was expecting an error, but instead the command worked. I suspect this is because on Windows the default character set may not be UTF-8?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905040307,IC_kwDOCGYnMM418dGz,9599,simonw,2021-08-24T23:23:36Z,2021-08-24T23:23:36Z,OWNER,"https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says: > Currently, `TextIOWrapper` uses `locale.getpreferredencoding(False)` (hereinafter called “locale encoding”) when encoding is not specified. > ... > Package authors using macOS or Linux may forget that the default encoding is not always UTF-8.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040902,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905040902,IC_kwDOCGYnMM418dQG,9599,simonw,2021-08-24T23:25:03Z,2021-08-24T23:25:03Z,OWNER,I'm going to skip this test on windows.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob", https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288430,https://api.github.com/repos/simonw/sqlite-utils/issues/320,903288430,IC_kwDOCGYnMM411xZu,9599,simonw,2021-08-22T15:44:55Z,2021-08-22T15:45:52Z,OWNER,"``` curl 'https://api.github.com/users/dogsheep/repos' | sqlite-utils memory - --analyze ``` ``` stdin.id: (1/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.node_id: (2/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.name: (3/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.full_name: (4/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.private: (5/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: 0 stdin.owner: (6/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: {""login"": ""dogsheep"", ""id"": 53015001, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjUzMDE1MD... stdin.html_url: (7/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.description: (8/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.fork: (9/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 1 Most common: 13: 0 stdin.url: (10/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.forks_url: (11/73) Total rows: 13 Null rows: 0 Blank rows: 0 Distinct values: 13 stdin.keys_url: (12/73) Total rows: 13 ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976405225,sqlite-utils memory --analyze option, https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288691,https://api.github.com/repos/simonw/sqlite-utils/issues/320,903288691,IC_kwDOCGYnMM411xdz,9599,simonw,2021-08-22T15:46:56Z,2021-08-22T15:46:56Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#schema-analyze-dump-and-save,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976405225,sqlite-utils memory --analyze option, https://github.com/simonw/sqlite-utils/pull/32#issuecomment-506784912,https://api.github.com/repos/simonw/sqlite-utils/issues/32,506784912,MDEyOklzc3VlQ29tbWVudDUwNjc4NDkxMg==,9599,simonw,2019-06-28T15:55:13Z,2019-06-28T15:55:13Z,OWNER,"Documentation, to be added as a subsection of https://sqlite-utils.readthedocs.io/en/latest/python-api.html#adding-foreign-key-constraints ### Adding multiple foreign key constraints at once The final step in adding a new foreign key to a SQLite database is to run VACUUM, to ensure the new foreign key is available in future introspection queries. VACUUM against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using table.add_foreign_key(...) these can quickly add up. Instead, you can use db.add_foreign_keys(...) to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a table, column, other_table and other_column. Here's an example adding two foreign keys at once: db.add_foreign_keys([ (""dogs"", ""breed_id"", ""breeds"", ""id""), (""dogs"", ""home_town_id"", ""towns"", ""id"") ]) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",462094937,db.add_foreign_keys() method, https://github.com/simonw/sqlite-utils/issues/319#issuecomment-904999850,https://api.github.com/repos/simonw/sqlite-utils/issues/319,904999850,IC_kwDOCGYnMM418TOq,9599,simonw,2021-08-24T21:49:08Z,2021-08-24T21:49:08Z,OWNER,This is a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905001586,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905001586,IC_kwDOCGYnMM418Tpy,9599,simonw,2021-08-24T21:52:50Z,2021-08-24T21:52:50Z,OWNER,"Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - ""Inserting binary data from files"" will become ""Inserting data from files"" I'm OK with keeping the default as `BLOB` but I could add a `--text` option which stores the content as text instead. If the text can't be stored as `utf-8` I'll probably raise an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013162,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905013162,IC_kwDOCGYnMM418Weq,9599,simonw,2021-08-24T22:15:31Z,2021-08-24T22:15:31Z,OWNER,"I'm going to assume utf-8 but allow `--encoding` to be used to specify something different, since that option is already supported by other commands.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013183,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905013183,IC_kwDOCGYnMM418We_,9599,simonw,2021-08-24T22:15:34Z,2021-08-24T22:15:34Z,OWNER,"Here's the error message I have working for invalid unicode: ``` sqlite-utils insert-files /tmp/text.db files *.txt --text [------------------------------------] 0% Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text 'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021047,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905021047,IC_kwDOCGYnMM418YZ3,9599,simonw,2021-08-24T22:33:48Z,2021-08-24T22:33:48Z,OWNER,"I had a few doubts about the design just now. Since `content_text` is supported as a special argument, an alternative way of handling the above would be: sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size This does exactly the same thing as just using `--text` and not specifying any columns, because the actual implementation of `--text` is as follows: https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855 But actually I think that's OK - ``--text`` is a useful shorthand that avoids you having to remember how to manually specify those columns with `-c`. So I'm going to leave the design as is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021933,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905021933,IC_kwDOCGYnMM418Ynt,9599,simonw,2021-08-24T22:36:04Z,2021-08-24T22:36:04Z,OWNER,"> Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode. I thought about supporting those different policies (with something like `--errors ignore`) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the `sqlite-utils insert` command (I added the same detailed error message): ``` Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text 'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a `BLOB` column instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905043974,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905043974,IC_kwDOCGYnMM418eAG,9599,simonw,2021-08-24T23:33:44Z,2021-08-24T23:33:44Z,OWNER,Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text., https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440207,https://api.github.com/repos/simonw/sqlite-utils/issues/318,901440207,IC_kwDOCGYnMM41uuLP,9599,simonw,2021-08-18T21:24:28Z,2021-08-18T21:24:49Z,OWNER,"Something like this then: sqlite-utils file.db ""select * from t"" --csv --gz > t.csv.gz Maybe add a `-o t.csv.gz` option too so you don't have to use a `>`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974067156,Research: handle gzipped CSV directly, https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440752,https://api.github.com/repos/simonw/sqlite-utils/issues/318,901440752,IC_kwDOCGYnMM41uuTw,9599,simonw,2021-08-18T21:25:30Z,2021-08-18T21:25:30Z,OWNER,"Some questions: - Should this support compression formats other than gzip? - Should `memory` learn to auto-detect gzipped data?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974067156,Research: handle gzipped CSV directly, https://github.com/simonw/sqlite-utils/issues/317#issuecomment-901337305,https://api.github.com/repos/simonw/sqlite-utils/issues/317,901337305,IC_kwDOCGYnMM41uVDZ,9599,simonw,2021-08-18T18:30:59Z,2021-08-18T18:30:59Z,OWNER,"I'm just going to remove this - I added it when the library was mostly undocumented, but it has comprehensive documentation now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972827346,Link to a better example on docs index, https://github.com/simonw/sqlite-utils/issues/316#issuecomment-898824020,https://api.github.com/repos/simonw/sqlite-utils/issues/316,898824020,IC_kwDOCGYnMM41kvdU,9599,simonw,2021-08-14T05:12:23Z,2021-08-14T05:12:23Z,OWNER,No visible backticks on https://sqlite-utils.datasette.io/en/latest/reference.html any more.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970320615,Fix visible backticks on reference page, https://github.com/simonw/sqlite-utils/issues/315#issuecomment-896339144,https://api.github.com/repos/simonw/sqlite-utils/issues/315,896339144,IC_kwDOCGYnMM41bQzI,9599,simonw,2021-08-10T21:55:41Z,2021-08-10T21:55:41Z,OWNER,Or should we raise an error if you attempt to call `.delete_where()` on a table that doesn't exist?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965440017,`.delete_where()` returns `[]` when it should return self, https://github.com/simonw/sqlite-utils/issues/314#issuecomment-896344833,https://api.github.com/repos/simonw/sqlite-utils/issues/314,896344833,IC_kwDOCGYnMM41bSMB,9599,simonw,2021-08-10T22:07:34Z,2021-08-10T22:07:34Z,OWNER,"Also the `.insert()` family of methods - they look pretty ugly in Sphinx right now: I should probably define reusable types for things like `pk=`, which have complex type signatures (a string or a list/tuple of strings) and show up in multiple places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965210966,Type signatures for `.create_table()` and `.create_table_sql()` and `.create()` and `Table.__init__`, https://github.com/simonw/sqlite-utils/issues/314#issuecomment-896369551,https://api.github.com/repos/simonw/sqlite-utils/issues/314,896369551,IC_kwDOCGYnMM41bYOP,9599,simonw,2021-08-10T23:06:41Z,2021-08-10T23:06:41Z,OWNER,"I took a big bite out of this when I annotated the ``.insert()`` method - but there are a bunch of other places that still need doing: https://github.com/simonw/sqlite-utils/blob/43bc06481783c3cfcee70c0cb541a686e8894adb/sqlite_utils/db.py#L2382-L2397 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965210966,Type signatures for `.create_table()` and `.create_table_sql()` and `.create()` and `Table.__init__`, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896154028,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896154028,IC_kwDOCGYnMM41ajms,9599,simonw,2021-08-10T17:01:06Z,2021-08-10T17:01:06Z,OWNER,"On Python 3.6: ``` sqlite_utils/db.py:366: in Database def tables(self) -> List[Table]: E NameError: name 'Table' is not defined ``` Python 3.7 can fix this with `from __future__ import annotations` but since we still support 3.6 I'll have to use a string instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896156971,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896156971,IC_kwDOCGYnMM41akUr,9599,simonw,2021-08-10T17:04:22Z,2021-08-10T17:05:59Z,OWNER,"I'm going to get Read The Docs to build the docs for this branch too - on https://readthedocs.org/projects/sqlite-utils/versions/ I am clicking this button: I then set it to ""active"" (so pushes to the branch will build it) and ""hidden"" (so it wouldn't show up in search or in the navigation menu). https://docs.readthedocs.io/en/stable/versions.html#version-states ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896174456,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896174456,IC_kwDOCGYnMM41aol4,9599,simonw,2021-08-10T17:27:01Z,2021-08-10T17:27:01Z,OWNER,"Docs are now building at https://sqlite-utils.datasette.io/en/autodoc/reference.html But there's a problem! The page is semi-blank: I need to teach Read The Docs how to ensure `sqlite_utils` is available for introspection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896175438,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896175438,IC_kwDOCGYnMM41ao1O,9599,simonw,2021-08-10T17:28:19Z,2021-08-10T17:28:19Z,OWNER,"https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration says do something like this at the top of `conf.py`: ```python import os import sys sys.path.insert(0, os.path.abspath('../../simpleble/')) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896180956,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896180956,IC_kwDOCGYnMM41aqLc,9599,simonw,2021-08-10T17:35:51Z,2021-08-10T17:35:51Z,OWNER,Reading the rest of https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration it suggests using a `requirements.txt` file to install dependencies - but I use `setup.py` for that so I need to figure out a different pattern here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896182934,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896182934,IC_kwDOCGYnMM41aqqW,9599,simonw,2021-08-10T17:38:44Z,2021-08-10T17:38:44Z,OWNER,"From https://docs.readthedocs.io/en/stable/config-file/v2.html#packages it looks like I can tell Read The Docs to run `pip install -e .` using a `.readthedocs.yaml` configuration: ```yaml version: 2 sphinx: configuration: docs/conf.py python: version: ""3.9"" install: - method: pip path: . extra_requirements: - docs ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896186025,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896186025,IC_kwDOCGYnMM41arap,9599,simonw,2021-08-10T17:42:51Z,2021-08-10T17:42:51Z,OWNER,That worked! https://sqlite-utils.datasette.io/en/autodoc/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896200682,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896200682,IC_kwDOCGYnMM41au_q,9599,simonw,2021-08-10T18:03:40Z,2021-08-10T18:03:40Z,OWNER,"Adding type signatures to `create_table()` and `.create_table_sql()` is a bit too involved, I'll do that in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896284722,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896284722,IC_kwDOCGYnMM41bDgy,9599,simonw,2021-08-10T20:08:03Z,2021-08-10T20:08:21Z,OWNER,"Spotted a rogue backtick: ![A0147E27-7506-49B0-BEFB-20D99BBFEBAD](https://user-images.githubusercontent.com/9599/128927930-b3333dee-a385-409b-a945-f108e6ea40df.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc, https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896131902,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896131902,IC_kwDOCGYnMM41aeM-,9599,simonw,2021-08-10T16:31:51Z,2021-08-10T16:31:51Z,OWNER,"`make livehtml` wasn't picking up changes I made to the docstrings `.py` files. Fix was to change it to this: ``` sphinx-autobuild -a -b html ""$(SOURCEDIR)"" ""$(BUILDDIR)"" $(SPHINXOPTS) $(0) --watch ../sqlite_utils ``` See https://github.com/executablebooks/sphinx-autobuild#relevant-sphinx-bugs - though that suggested `-a` but didn't suggest `--watch`, which is a tip I got from https://github.com/executablebooks/sphinx-autobuild#working-on-a-sphinx-html-theme ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings, https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896149590,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896149590,IC_kwDOCGYnMM41aihW,9599,simonw,2021-08-10T16:55:36Z,2021-08-10T16:55:36Z,OWNER,"I'm going to use this as an excuse to add a bunch more type signatures too, refs #266.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings, https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896152812,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896152812,IC_kwDOCGYnMM41ajTs,9599,simonw,2021-08-10T16:59:34Z,2021-08-10T16:59:34Z,OWNER,Work will continue in PR #312.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings, https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896381184,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896381184,IC_kwDOCGYnMM41bbEA,9599,simonw,2021-08-10T23:33:33Z,2021-08-10T23:33:33Z,OWNER,"Now live at https://sqlite-utils.datasette.io/en/latest/reference.html TIL from what I learned today here: https://til.simonwillison.net/sphinx/sphinx-autodoc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings,