id,node_id,number,title,user,user_label,state,locked,assignee,assignee_label,milestone,milestone_label,comments,created_at,updated_at,closed_at,author_association,pull_request,body,repo,repo_label,type,active_lock_reason,performed_via_github_app,reactions,draft,state_reason 1066474200,I_kwDOCGYnMM4_kRrY,344,Support STRICT tables,9599,simonw,closed,0,,,,,14,2021-11-29T20:32:23Z,2023-12-08T05:22:39Z,2023-12-08T05:22:39Z,OWNER,,"New in SQLite 3.37.0, released a few days ago: https://www.sqlite.org/stricttables.html",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/344/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 2029161033,I_kwDOCGYnMM548opJ,606,str and int as aliases for text and integer,9599,simonw,closed,0,,,,,2,2023-12-06T18:35:49Z,2023-12-06T19:44:04Z,2023-12-06T18:49:32Z,OWNER,,"I keep making this mistake: ```bash sqlite-utils add-column content.db assets _since int ``` ``` Usage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME [[integer|float|b lob|text|INTEGER|FLOAT|BLOB|TEXT]] Try 'sqlite-utils add-column -h' for help. Error: Invalid value for '[[integer|float|blob|text|INTEGER|FLOAT|BLOB|TEXT]]': 'int' is not one of 'integer', 'float', 'blob', 'text', 'INTEGER', 'FLOAT', 'BLOB', 'TEXT'. ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/606/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1977155641,I_kwDOCGYnMM512QA5,601,Move plugin directory into documentation,9599,simonw,open,0,,,,,0,2023-11-04T04:07:52Z,2023-11-04T04:07:52Z,,OWNER,,"https://github.com/simonw/sqlite-utils-plugins should be in the official documentation. I can use the same pattern as https://llm.datasette.io/en/stable/plugins/directory.html https://til.simonwillison.net/readthedocs/stable-docs",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/601/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1884335789,PR_kwDOCGYnMM5Zs0KB,591,Test against Python 3.12 preview,9599,simonw,closed,0,,,,,3,2023-09-06T16:10:00Z,2023-11-04T00:58:03Z,2023-11-04T00:58:02Z,OWNER,simonw/sqlite-utils/pulls/591,"https://dev.to/hugovk/help-test-python-312-beta-1508/ ---- :books: Documentation preview :books:: https://sqlite-utils--591.org.readthedocs.build/en/591/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/591/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",0, 944846776,MDU6SXNzdWU5NDQ4NDY3NzY=,297,Option for importing CSV data using the SQLite .import mechanism,9599,simonw,open,0,,,,,23,2021-07-14T22:36:41Z,2023-09-22T20:49:52Z,,OWNER,,"As seen in https://til.simonwillison.net/sqlite/import-csv - `.mode csv` and then `.import school.csv schools` is hugely faster than importing via `sqlite-utils insert` and doing the work in Python - but it can only be implemented by shelling out to the `sqlite3` CLI tool, it's not functionality that is exposed to the Python `sqlite3` module. An option to use this would be useful - maybe something like this: sqlite-utils insert blah.db blah blah.csv --fast",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/297/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1891614971,I_kwDOCGYnMM5wv8D7,594,Represent compound foreign keys in table.foreign_keys output,9599,simonw,open,0,,,,,2,2023-09-12T03:48:24Z,2023-09-12T03:51:13Z,,OWNER,,"Given this schema: ```sql CREATE TABLE departments ( campus_name TEXT NOT NULL, dept_code TEXT NOT NULL, dept_name TEXT, PRIMARY KEY (campus_name, dept_code) ); CREATE TABLE courses ( course_code TEXT PRIMARY KEY, course_name TEXT, campus_name TEXT NOT NULL, dept_code TEXT NOT NULL, FOREIGN KEY (campus_name, dept_code) REFERENCES departments(campus_name, dept_code) ); ``` The output of `db[""courses""].foreign_keys` right now is: ``` [ForeignKey(table='courses', column='campus_name', other_table='departments', other_column='campus_name'), ForeignKey(table='courses', column='dept_code', other_table='departments', other_column='dept_code')] ``` Which suggests two normal foreign keys, not one compound foreign key.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/594/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1886771493,I_kwDOCGYnMM5wddkl,592,`table.transform()` should preserve `rowid` values,9599,simonw,closed,0,,,,,6,2023-09-08T00:42:38Z,2023-09-10T17:46:41Z,2023-09-09T00:45:32Z,OWNER,,"I just spotted a bug when using https://datasette.io/plugins/datasette-configure-fts and https://datasette.io/plugins/datasette-edit-schema at the same time. Steps to reproduce: - Configure FTS for a table, then run a test search - Edit the schema for that table and change the order of columns - Run the test search again I got the wrong search results, which I think is because the `_fts` table pointed to the first table by `rowid` but those `rowid` values were entirely rewritten as a consequence of running `table.transform()` on the table. Reconfiguring FTS on the table fixed the problem. I think `table.transform()` should be able to preserve `rowid` values.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/592/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1886783150,PR_kwDOCGYnMM5Z1H1d,593,".transform() now preserves rowid values, refs #592",9599,simonw,closed,0,,,,,1,2023-09-08T01:02:28Z,2023-09-10T17:44:59Z,2023-09-09T00:45:30Z,OWNER,simonw/sqlite-utils/pulls/593,"Refs: - #592 - [x] Tests against weird shaped tables I need to test that this works against: - `rowid` tables - Tables that have a column called `rowid` even though they are not rowid tables ---- :books: Documentation preview :books:: https://sqlite-utils--593.org.readthedocs.build/en/593/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/593/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1879214365,I_kwDOCGYnMM5wAokd,590,Ability to tell if a Database is an in-memory one,9599,simonw,open,0,,,,,1,2023-09-03T19:50:15Z,2023-09-03T19:50:36Z,,OWNER,,"Currently the constructor accepts `memory=True` or `memory_name=...` and uses those to create a connection, but does not record what those values were: https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L307-L349 This makes it hard to tell if a database object is to an in-memory or a file-based database, which is sometimes useful to know.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/590/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1879209560,I_kwDOCGYnMM5wAnZY,589,Mechanism for de-registering registered SQL functions,9599,simonw,open,0,,,,,3,2023-09-03T19:32:39Z,2023-09-03T19:36:34Z,,OWNER,,I used a custom SQL function in a migration script and then realized that it should be de-registered before the end of the script to avoid leaking into the calling code.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/589/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1868713944,I_kwDOCGYnMM5vYk_Y,588,`table.get(column=value)` option for retrieving things not by their primary key,9599,simonw,open,0,,,,,1,2023-08-28T00:41:23Z,2023-08-28T00:41:54Z,,OWNER,,"This came up working on this feature: - https://github.com/simonw/llm/pull/186 I have a table with this schema: ```sql CREATE TABLE [collections] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [model] TEXT ); CREATE UNIQUE INDEX [idx_collections_name] ON [collections] ([name]); ``` So the primary key is an integer (because it's going to have a huge number of rows foreign key related to it, and I don't want to store a larger text value thousands of times), but there is a unique constraint on the `name` - that would be the primary key column if not for all of those foreign keys. Problem is, fetching the collection by name is actually pretty inconvenient. Fetch by numeric ID: ```python try: table[""collections""].get(1) except NotFoundError: # It doesn't exist ``` Fetching by name: ```python def get_collection(db, collection): rows = db[""collections""].rows_where(""name = ?"", [collection]) try: return next(rows) except StopIteration: raise NotFoundError(""Collection not found: {}"".format(collection)) ``` It would be neat if, for columns where we know that we should always get 0 or one result, we could do this instead: ```python try: collection = table[""collections""].get(name=""entries"") except NotFoundError: # It doesn't exist ``` The existing `.get()` method doesn't have any non-positional arguments, so using `**kwargs` like that should work: https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L1495",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/588/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1857851384,I_kwDOCGYnMM5uvI_4,587,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference,9599,simonw,closed,0,,,,,3,2023-08-19T20:01:26Z,2023-08-19T20:04:33Z,2023-08-19T20:04:32Z,OWNER,,"Extremely detailed story of how I got to this point: - https://github.com/simonw/llm/issues/162 Steps to reproduce (only if that pragma is on though): ```bash python -c ' import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute("""""" CREATE TABLE ""logs"" ( [id] INTEGER PRIMARY KEY, [model] TEXT, [prompt] TEXT, [system] TEXT, [prompt_json] TEXT, [options_json] TEXT, [response] TEXT, [response_json] TEXT, [reply_to_id] INTEGER, [chat_id] INTEGER REFERENCES [log]([id]), [duration_ms] INTEGER, [datetime_utc] TEXT ); """""") db[""logs""].add_foreign_key(""reply_to_id"", ""logs"", ""id"") ' ``` This succeeds in some environments, fails in others.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/587/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1817289521,I_kwDOCGYnMM5sUaMx,577,Get `add_foreign_keys()` to work without modifying `sqlite_master`,9599,simonw,closed,0,,,,,9,2023-07-23T20:40:18Z,2023-08-18T17:43:11Z,2023-08-18T00:48:10Z,OWNER,,"https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1165-L1174 This is the only place in the code that attempts to modify `sqlite_master` directly, which fails on some Python installations. Could this use the `.transform()` trick instead? Or automatically switch to that trick if it hits an error?",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/577/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1856075668,I_kwDOCGYnMM5uoXeU,586,.transform() fails to drop column if table is part of a view,9599,simonw,open,0,,,,,3,2023-08-18T05:25:22Z,2023-08-18T06:13:47Z,,OWNER,,"I got this error trying to drop a column from a table that was part of a SQL view: > error in view plugins: no such table: main.pypi_releases Upon further investigation I found that this pattern seemed to fix it: ```python def transform_the_table(conn): # Run this in a transaction: with conn: # We have to read all the views first, because we need to drop and recreate them db = sqlite_utils.Database(conn) views = {v.name: v.schema for v in db.views if table.lower() in v.schema.lower()} for view in views.keys(): db[view].drop() db[table].transform( types=types, rename=rename, drop=drop, column_order=[p[0] for p in order_pairs], ) # Now recreate the views for name, schema in views.items(): db.create_view(name, schema) ``` So grab a copy of any view that might reference this table, start a transaction, drop those views, run the transform, recreate the views again. > I wonder if this should become an option in `sqlite-utils`? Maybe a `recreate_views=True` argument for `table.tranform(...)`? Should it be opt-in or opt-out? _Originally posted by @simonw in https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683370548_ ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/586/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1855894222,I_kwDOCGYnMM5unrLO,585,CLI equivalents to `transform(add_foreign_keys=)`,9599,simonw,closed,0,,,,,7,2023-08-18T01:07:15Z,2023-08-18T01:51:16Z,2023-08-18T01:51:15Z,OWNER,,"The new options added in: - #577 Deserve consideration in the CLI as well. https://github.com/simonw/sqlite-utils/blob/d2bcdc00c6ecc01a6e8135e775ffdb87572b802b/sqlite_utils/db.py#L1706-L1708",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/585/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1855836914,I_kwDOCGYnMM5undLy,583,Get rid of test.utils.collapse_whitespace,9599,simonw,closed,0,,,,,1,2023-08-17T23:31:09Z,2023-08-18T00:59:19Z,2023-08-18T00:59:19Z,OWNER,,"I have a neater pattern for this now - instead of: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L472-L475 I now prefer: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L1163-L1171",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/583/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1855838223,PR_kwDOCGYnMM5YM-I3,584,.transform() instead of modifying sqlite_master for add_foreign_keys,9599,simonw,closed,0,,,,,13,2023-08-17T23:32:45Z,2023-08-18T00:48:13Z,2023-08-18T00:48:08Z,OWNER,simonw/sqlite-utils/pulls/584,"Refs: - #577 ---- :books: Documentation preview :books:: https://sqlite-utils--584.org.readthedocs.build/en/584/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/584/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1818838294,I_kwDOCGYnMM5saUUW,578,Plugin hook for adding new output formats,9599,simonw,open,0,,,,,5,2023-07-24T17:29:18Z,2023-08-07T15:41:49Z,,OWNER,,"> What would it take to add a format hook? I'm still thinking about my GIS workflow, and being able to do `sqlite-utils query ... --geojson` would be nice. It's the one place my Datasette workflow is messy, having to do `datasette . --get /path/to/query.geojson --setting max_rows_returned 10000 --load-extension spatialite`. > I know the current pattern is `--csv`, but maybe `--format geojson` is more future-proof. https://discord.com/channels/823971286308356157/997738192360964156/1133076679011602432",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/578/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1823160748,I_kwDOCGYnMM5sqzms,581,`sqlite-utils convert --pdb` option,9599,simonw,closed,0,,,,,1,2023-07-26T21:02:50Z,2023-07-26T21:07:45Z,2023-07-26T21:06:10Z,OWNER,,While using `sqlite-utils convert` I realized it would be handy if you could pass `--pdb` to have it open the debugger at the first instance of a failed conversion.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/581/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1816997390,I_kwDOCGYnMM5sTS4O,576,Backfill the release notes prior to 0.4,9599,simonw,closed,0,,,,,2,2023-07-23T05:41:42Z,2023-07-23T05:49:51Z,2023-07-23T05:48:21Z,OWNER,,"Currently the changelog starts at 0.4: https://sqlite-utils.datasette.io/en/3.34/changelog.html#id115 I want the other releases - according to https://pypi.org/project/sqlite-utils/#history there are three missing: ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/576/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1816919568,I_kwDOCGYnMM5sS_4Q,575,Python API ability to opt-out of connection plugins,9599,simonw,closed,0,,,,,2,2023-07-22T23:01:13Z,2023-07-22T23:17:22Z,2023-07-22T23:08:22Z,OWNER,,"Plugins affecting the CLI by default makes sense to me. I'm less confident about them _always_ affecting users of the Python API. I'm going to have them apply by default, but I'm going to add a mechanism to opt-out on an individual database basis. Basically this: ```python from sqlite_utils import Database db = Database(memory=True, execute_plugins=False) # Anything using db from here on will not execute plugins ``` cc @asg017 Refs: - #567 - #574 ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/575/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1816918185,I_kwDOCGYnMM5sS_ip,574,`prepare_connection()` plugin hook,9599,simonw,closed,0,,,,,3,2023-07-22T22:52:47Z,2023-07-22T23:13:14Z,2023-07-22T22:59:10Z,OWNER,,"> Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34! _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424_ ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/574/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1816876211,I_kwDOCGYnMM5sS1Sz,571,`.transform(keep_table=...)` option,9599,simonw,closed,0,,,,,1,2023-07-22T19:49:29Z,2023-07-22T22:32:18Z,2023-07-22T22:32:18Z,OWNER,,">> Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one. > > I think `keep_table=""name_of_table""` is good for this. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324_ ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/571/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1816877910,I_kwDOCGYnMM5sS1tW,572,Don't test Python 3.7 against textual,9599,simonw,closed,0,,,,,2,2023-07-22T19:57:03Z,2023-07-22T22:16:50Z,2023-07-22T22:16:50Z,OWNER,,"Spotted this in the GitHub Actions logs: ![IMG_5046](https://github.com/simonw/sqlite-utils/assets/9599/81fb1093-cd8a-4019-a612-2e49b500c933) ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/572/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1786243905,I_kwDOCGYnMM5qd-tB,564,Document that running `db.transform()` tidies up the schema indentation,9599,simonw,closed,0,,,,,0,2023-07-03T13:59:28Z,2023-07-22T22:15:34Z,2023-07-22T22:15:34Z,OWNER,,"> ... and it turns out running `.transform()` with no arguments still fixes the format of the schema! ```pycon >>> db[""log""].add_column(""foo"", str) >>> db[""log""].add_column(""bar"", str)
>>> db[""log""].add_column(""baz"", str)
>>> print(db[""log""].schema) CREATE TABLE ""log"" ( [id] INTEGER PRIMARY KEY, [name2] TEXT, [age] INTEGER, [weight] FLOAT , [foo] TEXT, [bar] TEXT, [baz] TEXT) >>> db[""log""].transform()
>>> print(db[""log""].schema) CREATE TABLE ""log"" ( [id] INTEGER PRIMARY KEY, [name2] TEXT, [age] INTEGER, [weight] FLOAT, [foo] TEXT, [bar] TEXT, [baz] TEXT ) ``` _Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618347727_ ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/564/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",,completed 1205687423,I_kwDOCGYnMM5H3VR_,426,CLI docs should link to Python docs and vice versa,9599,simonw,closed,0,9599,simonw,,,1,2022-04-15T16:05:15Z,2023-07-22T22:13:22Z,2023-07-22T22:13:22Z,OWNER,,"For every command/API method there should be a link to the equivalent in the other form factor. Maybe also link to the API and CLI reference pages too.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/426/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1786258502,I_kwDOCGYnMM5qeCRG,565,Table renaming: db.rename_table() and sqlite-utils rename-table,9599,simonw,closed,0,,,,,6,2023-07-03T14:07:42Z,2023-07-22T22:12:40Z,2023-07-22T22:12:40Z,OWNER,,"> I find myself wanting two new features in `sqlite-utils`: > - The ability to have the new transformed table set to a specific name, while keeping the old table around > - The ability to rename a table (`sqlite-utils` doesn't have a table rename function at all right now) _Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618375042_ ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/565/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1816851056,I_kwDOCGYnMM5sSvJw,568,"table.create(..., replace=True)",9599,simonw,closed,0,,,,,7,2023-07-22T18:12:22Z,2023-07-22T19:25:35Z,2023-07-22T19:15:44Z,OWNER,,"Found myself using this pattern to quickly prototype a schema: ```python import sqlite_utils db = sqlite_utils.Database(memory=True) print(db[""answers_chunks""].create({ ""id"": int, ""content"": str, ""embedding_type_id"": int, ""embedding"": bytes, ""embedding_content_md5"": str, ""source"": str, }, pk=""id"", transform=True).schema) ``` Using `replace=True` to drop and then recreate the table would be neat here, and would be consistent with other places that use `replace=True`.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/568/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1816852402,I_kwDOCGYnMM5sSvey,569,register_command plugin hook,9599,simonw,closed,0,,,,,3,2023-07-22T18:17:27Z,2023-07-22T19:19:35Z,2023-07-22T19:19:35Z,OWNER,,"> I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450_ ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/569/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1816857105,I_kwDOCGYnMM5sSwoR,570,`sqlite-utils install -e` option,9599,simonw,closed,0,,,,,0,2023-07-22T18:32:23Z,2023-07-22T18:55:59Z,2023-07-22T18:32:56Z,OWNER,,"As seen in LLM. Needed while working on: - #567",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/570/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1785360409,I_kwDOCGYnMM5qanAZ,563,`--empty-null` option when importing CSV,9599,simonw,closed,0,,,,,1,2023-07-03T05:23:36Z,2023-07-03T05:44:43Z,2023-07-03T05:42:30Z,OWNER,,"CSV files with empty cells in (which come through as the empty string) are common and a bit gross. Having an option that means ""and if it's an empty string store `null` instead) would be cool. I brainstormed name options here https://chat.openai.com/share/c947b738-ee7d-419c-af90-bc84e90987da",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/563/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1784794489,I_kwDOCGYnMM5qYc15,562,Explore the intersection between sqlite-utils and dataclasses,9599,simonw,open,0,,,,,1,2023-07-02T19:23:08Z,2023-07-02T19:26:39Z,,OWNER,,"> Aside: this makes me think it might be cool if `sqlite-utils` had a way of working with dataclasses rather than just dicts, and knew how to create a SQLite table to match a dataclass and maybe how to code-generate dataclasses for a specific table schema (dynamically or even using code-generation that can be written to disk, for better editor integrations). _Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1616742529_ ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/562/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1777548699,I_kwDOCGYnMM5p8z2b,561,`--stop-after` option for `insert` and `upsert` commands,9599,simonw,closed,0,,,,,1,2023-06-27T18:44:15Z,2023-06-27T18:50:09Z,2023-06-27T18:50:08Z,OWNER,,I found myself wanting to insert rows from a 849MB CSV file without processing the whole thing: https://huggingface.co/datasets/jerpint-org/HackAPrompt-Playground-Submissions/tree/main,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/561/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1773458985,PR_kwDOCGYnMM5T2mMb,560,Use sqlean if available in environment,9599,simonw,closed,0,,,,,10,2023-06-25T19:48:48Z,2023-06-26T08:21:00Z,2023-06-25T23:25:51Z,OWNER,simonw/sqlite-utils/pulls/560,"Refs: - #559 ---- :books: Documentation preview :books:: https://sqlite-utils--560.org.readthedocs.build/en/560/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/560/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1773450152,I_kwDOCGYnMM5ptLOo,559,sqlean support,9599,simonw,closed,0,,,,,0,2023-06-25T19:27:26Z,2023-06-25T23:25:53Z,2023-06-25T23:25:53Z,OWNER,,"If sqlean is available, use that. Refs: - https://github.com/nalgeon/sqlean.py/issues/1#issuecomment-1605707788 This will provide a good workaround for: - #235 ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/559/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1718612569,I_kwDOCGYnMM5mb_JZ,552,Document how to setup shell auto-completion,9599,simonw,closed,0,,,,,1,2023-05-21T19:20:41Z,2023-05-21T21:05:16Z,2023-05-21T21:03:40Z,OWNER,,"https://click.palletsprojects.com/en/8.1.x/shell-completion/ This works for `zsh`: eval ""$(_SQLITE_UTILS_COMPLETE=zsh_source sqlite-utils)"" This will probably work for `bash`: eval ""$(_SQLITE_UTILS_COMPLETE=bash_source sqlite-utils)"" Need to add this to the installation docs here: https://sqlite-utils.datasette.io/en/stable/installation.html - along with the pattern for adding that to `.zshrc` or whatever.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/552/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1718607907,I_kwDOCGYnMM5mb-Aj,551,Make as many examples in the CLI docs as possible copy-and-pastable,9599,simonw,closed,0,,,,,6,2023-05-21T19:04:10Z,2023-05-21T21:04:04Z,2023-05-21T20:57:24Z,OWNER,,"e.g. in this section: https://sqlite-utils.datasette.io/en/stable/cli.html#running-queries-directly-against-csv-or-json The little copy button will also copy the `$ ` which breaks the examples when copied.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/551/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1718635018,PR_kwDOCGYnMM5Q9lY4,553,Reformatted CLI examples in docs,9599,simonw,closed,0,,,,,2,2023-05-21T20:44:34Z,2023-05-21T20:57:27Z,2023-05-21T20:57:23Z,OWNER,simonw/sqlite-utils/pulls/553,"Refs: - #551 ---- :books: Documentation preview :books:: https://sqlite-utils--553.org.readthedocs.build/en/553/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/553/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1718517882,I_kwDOCGYnMM5mboB6,545,Try out Trogon for a tui interface,9599,simonw,closed,0,,,,,6,2023-05-21T14:08:25Z,2023-05-21T19:33:13Z,2023-05-21T18:41:58Z,OWNER,,https://github.com/Textualize/trogon,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/545/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1718595700,I_kwDOCGYnMM5mb7B0,550,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7,9599,simonw,closed,0,,,,,3,2023-05-21T18:24:39Z,2023-05-21T18:42:25Z,2023-05-21T18:41:58Z,OWNER,,"https://github.com/simonw/sqlite-utils/actions/runs/5039064797/jobs/9036965488 ``` Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.7.16/x64/bin/flake8"", line 8, in sys.exit(main()) File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/cli.py"", line 22, in main app.run(argv) File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py"", line 363, in run self._run(argv) File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py"", line 350, in _run self.initialize(argv) File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py"", line 330, in initialize self.find_plugins(config_finder) File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py"", line 153, in find_plugins self.check_plugins = plugin_manager.Checkers(local_plugins.extension) File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py"", line 357, in __init__ self.namespace, local_plugins=local_plugins File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py"", line 238, in __init__ self._load_entrypoint_plugins() File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py"", line 254, in _load_entrypoint_plugins eps = importlib_metadata.entry_points().get(self.namespace, ()) AttributeError: 'EntryPoints' object has no attribute 'get' Error: Process completed with exit code 1. ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/550/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1718586377,PR_kwDOCGYnMM5Q9cAv,549,TUI powered by Trogon,9599,simonw,closed,0,,,,,3,2023-05-21T17:55:42Z,2023-05-21T18:42:00Z,2023-05-21T18:41:56Z,OWNER,simonw/sqlite-utils/pulls/549,"Refs: - #545 ---- :books: Documentation preview :books:: https://sqlite-utils--549.org.readthedocs.build/en/549/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/549/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1718576761,I_kwDOCGYnMM5mb2Z5,548,analyze-tables should validate provide --column names,9599,simonw,closed,0,,,,,1,2023-05-21T17:20:24Z,2023-05-21T17:35:52Z,2023-05-21T17:35:52Z,OWNER,,"Noticed this while testing: - #547 If you pass a non-existent column to `-c/--column` you don't get an error message.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/548/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1718572201,I_kwDOCGYnMM5mb1Sp,547,No need to show common values if everything is null,9599,simonw,closed,0,,,,,1,2023-05-21T17:05:07Z,2023-05-21T17:19:21Z,2023-05-21T17:19:21Z,OWNER,,"Noticed this: ``` % sqlite-utils analyze-tables content.db repos -c delete_branch_on_merge --common-limit 20 --no-least repos.delete_branch_on_merge: (1/1) Total rows: 158 Null rows: 158 Blank rows: 0 Distinct values: 0 Most common: 158: None ``` The `158: None` there is duplicate information considering we already know there are 158/158 null rows.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/547/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1718515590,I_kwDOCGYnMM5mbneG,544,New options for analyze-tables --common-limit --no-most and --no-least,9599,simonw,closed,0,,,,,2,2023-05-21T14:03:19Z,2023-05-21T17:03:06Z,2023-05-21T16:19:31Z,OWNER,,"The ""least common"" section is frequently uninteresting, especially for huge tables with a large number of repeated-once values. sqlite-utils analyze-tables content.db repos --common-limit 20 --no-least",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/544/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1718550688,PR_kwDOCGYnMM5Q9VH0,546,"Analyze tables options: --common-limit, --no-most, --no-least",9599,simonw,closed,0,,,,,2,2023-05-21T15:54:39Z,2023-05-21T16:19:30Z,2023-05-21T16:19:30Z,OWNER,simonw/sqlite-utils/pulls/546,"Refs #544 - [x] Documentation for CLI options - [x] Documentation for new Python API parameters: `most_common: bool` and `least_common: bool` - [x] Tests for CLI - [x] Tests for Python API",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/546/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1124731464,I_kwDOCGYnMM5DCgpI,399,"Make it easier to insert geometries, with documentation and maybe code",9599,simonw,open,0,,,,,25,2022-02-05T00:11:26Z,2023-05-16T03:11:52Z,,OWNER,,"In playing with the new SpatiaLite helpers from #385 I noticed that actually populating geometry columns is still a little bit tricky. Here's what I ended up doing: ```python import httpx, sqlite_utils db = sqlite_utils.Database(""/tmp/spatial.db"") attractions = httpx.get(""https://latest.datasette.io/fixtures/roadside_attractions.json?_shape=array"").json() db[""attractions""].insert_all(attractions, pk=""pk"") # Schema of that table is now: # CREATE TABLE [attractions] ( # [pk] INTEGER PRIMARY KEY, # [name] TEXT, # [address] TEXT, # [latitude] FLOAT, # [longitude] FLOAT # ) db.init_spatialite() db[""attractions""].add_geometry_column(""point"", ""POINT"") db.execute("""""" update attractions set point = GeomFromText( 'POINT(' || longitude || ' ' || latitude || ')', 4326 ) """""") ``` That last line took some figuring out - especially the need for the SRID of `4326`, without which I got this error: > `IntegrityError: attractions.point violates Geometry constraint [geom-type or SRID not allowed]` It would be good to both document this in more detail, but ideally also to come up with a more obvious pattern for inserting common types of spatial data. Also related: - #398 - #79",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/399/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1701018909,I_kwDOCGYnMM5lY30d,543,Tests broken on Windows due to new convert() lambda names,9599,simonw,closed,0,,,,,0,2023-05-08T22:11:29Z,2023-05-08T22:19:04Z,2023-05-08T22:19:04Z,OWNER,,"https://github.com/simonw/sqlite-utils/actions/runs/4920084038/jobs/8788501314 ```python sql = 'update [example] set [dt] = lambda_-9223371942137158589([dt]);' ``` From: - #526",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/543/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1516644980,I_kwDOCGYnMM5aZip0,520,rows_from_file() raises confusing error if file-like object is not in binary mode,9599,simonw,closed,0,,,,,3,2023-01-02T19:00:14Z,2023-05-08T22:08:07Z,2023-05-08T22:08:07Z,OWNER,,"I got this error: ``` File ""/Users/simon/Dropbox/Development/openai-to-sqlite/openai_to_sqlite/cli.py"", line 27, in embeddings rows, _ = rows_from_file(input) ^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.local/share/virtualenvs/openai-to-sqlite-jt4obeb2/lib/python3.11/site-packages/sqlite_utils/utils.py"", line 305, in rows_from_file first_bytes = buffered.peek(2048).strip() ^^^^^^^^^^^^^^^^^^^ ``` From this code: ```python @cli.command() @click.argument( ""db_path"", type=click.Path(file_okay=True, dir_okay=False, allow_dash=False), ) @click.option( ""-i"", ""--input"", type=click.File(""r""), default=""-"", ) def embeddings(db_path, input): ""Store embeddings for one or more text documents"" click.echo(""Here is some output"") db = sqlite_utils.Database(db_path) rows, _ = rows_from_file(input) print(list(rows)) ``` The error went away when I changed it to `type=click.File(""rb"")`. This should either be called out in the documentation or `rows_from_file()` should be fixed to handle text-mode files in addition to binary files.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/520/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1700936245,I_kwDOCGYnMM5lYjo1,542,Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0,9599,simonw,open,0,,,9374594,4.0 backwards incomatible changes,1,2023-05-08T21:04:28Z,2023-05-08T21:07:41Z,,OWNER,,"Following: - #527 The only reason I didn't remove fix this mis-feature entirely is that it represents a backwards incompatible change. I'll make that change in 4.0.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/542/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1700840265,I_kwDOCGYnMM5lYMNJ,541,Get tests to pass with `pytest -Werror`,9599,simonw,open,0,,,,,1,2023-05-08T19:57:23Z,2023-05-08T19:59:35Z,,OWNER,,"Inspired by: - #534",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/541/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1699184583,I_kwDOCGYnMM5lR3_H,540,sphinx.builders.linkcheck build error,9599,simonw,closed,0,,,,,4,2023-05-07T18:37:09Z,2023-05-08T04:56:13Z,2023-05-07T18:42:36Z,OWNER,,"https://readthedocs.org/projects/sqlite-utils/builds/20512693/ ``` Running Sphinx v6.2.1 Traceback (most recent call last): File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py"", line 442, in load_extension mod = import_module(extname) File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/importlib/__init__.py"", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File """", line 1014, in _gcd_import File """", line 991, in _find_and_load File """", line 975, in _find_and_load_unlocked File """", line 671, in _load_unlocked File """", line 783, in exec_module File """", line 219, in _call_with_frames_removed File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/builders/linkcheck.py"", line 20, in from requests import Response File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/requests/__init__.py"", line 43, in import urllib3 File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/urllib3/__init__.py"", line 38, in raise ImportError( ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168 The above exception was the direct cause of the following exception: Traceback (most recent call last): File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py"", line 280, in build_main app = Sphinx(args.sourcedir, args.confdir, args.outputdir, File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py"", line 225, in __init__ self.setup_extension(extension) File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py"", line 404, in setup_extension self.registry.load_extension(self, extname) File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py"", line 445, in load_extension raise ExtensionError(__('Could not import extension %s') % extname, sphinx.errors.ExtensionError: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168) Extension error: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168) ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/540/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1699174055,I_kwDOCGYnMM5lR1an,539,"`--raw-lines` option, like `--raw` for multiple lines",9599,simonw,closed,0,,,,,4,2023-05-07T18:07:46Z,2023-05-07T18:43:24Z,2023-05-07T18:26:18Z,OWNER,,I wanted to output newline-separated output of the first column of every row in the results - like `--row` but for more than one line.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/539/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1203842656,I_kwDOCGYnMM5HwS5g,425,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher,9599,simonw,closed,0,,,,,5,2022-04-13T22:16:53Z,2023-04-15T20:14:58Z,2022-04-13T22:48:57Z,OWNER,,"Got this error while investigating: - #421 Even though I was using the `LD_PRELOAD` trick from https://til.simonwillison.net/sqlite/ld-preload to use a newer version of SQLite. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/425/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1114543475,I_kwDOCGYnMM5CbpVz,388,Link to stable docs from older versions,9599,simonw,closed,0,,,,,7,2022-01-26T01:55:46Z,2023-03-26T23:43:12Z,2022-01-26T02:00:22Z,OWNER,,"https://sqlite-utils.datasette.io/en/2.14.1/ isn't showing a link to the stable release right now. I should also apply the same fix I used for Datasette in: - https://github.com/simonw/datasette/issues/1608 TIL: https://til.simonwillison.net/readthedocs/link-from-latest-to-stable",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/388/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1620516340,I_kwDOCGYnMM5glx30,533,ReadTheDocs error: not all arguments converted during string formatting,9599,simonw,closed,0,,,,,2,2023-03-12T21:21:05Z,2023-03-12T21:25:33Z,2023-03-12T21:25:33Z,OWNER,,"This came up as a failure running tests for: - #531 Traceback on https://readthedocs.org/projects/sqlite-utils/builds/19749348/ ``` File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/docutils/parsers/rst/states.py"", line 889, in interpreted nodes, messages2 = role_fn(role, rawsource, text, lineno, self) File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py"", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting Exception occurred: File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py"", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/533/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1373224657,I_kwDOCGYnMM5R2b7R,488,`sqlite-utils transform` should set empty strings to null when converting text columns to integer/float,9599,simonw,open,0,,,,,5,2022-09-14T15:51:30Z,2022-12-23T17:38:55Z,,OWNER,,"``` /tmp % echo ""id,age,weight\n1,3,2.5\n2,,"" | sqlite-utils insert test.db test - --csv /tmp % sqlite-utils schema test.db CREATE TABLE [test] ( [id] TEXT, [age] TEXT, [weight] TEXT ); /tmp % sqlite-utils transform test.db test --type age integer --type weight float /tmp % sqlite-utils schema test.db CREATE TABLE ""test"" ( [id] TEXT, [age] INTEGER, [weight] FLOAT ); /tmp % sqlite-utils rows test.db test [{""id"": ""1"", ""age"": 3, ""weight"": 2.5}, {""id"": ""2"", ""age"": """", ""weight"": """"}] ``` It would be neat if this resulted in the following instead: ``` {""id"": ""2"", ""age"": null, ""weight"": null} ``` Related Discord discussion: https://discord.com/channels/823971286308356157/823971286941302908/1019635490833567794",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/488/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1487764628,I_kwDOCGYnMM5YrXyU,518,flake8 ValueError: Error code '#' supplied to 'extend-ignore' option...,9599,simonw,closed,0,,,,,0,2022-12-10T01:30:24Z,2022-12-10T01:36:46Z,2022-12-10T01:36:46Z,OWNER,,"> `Error code '#' supplied to 'extend-ignore' option does not match '^[A-Z]{1,3}[0-9]{0,3}$'` https://github.com/simonw/sqlite-utils/actions/runs/3662011265/jobs/6190770361 I think from this: https://github.com/simonw/sqlite-utils/blob/e660635cea6c32f4022818380b1e1ee88e7c93a6/setup.cfg#L1-L3 ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/518/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1487757143,I_kwDOCGYnMM5YrV9X,517,Drop support for Python 3.6,9599,simonw,closed,0,,,,,1,2022-12-10T01:23:31Z,2022-12-10T01:36:36Z,2022-12-10T01:36:36Z,OWNER,,"CI has started failing for Python 3.6: https://github.com/simonw/sqlite-utils/actions/runs/3576322798 It's fixable by swiching away from `ubuntu-latest` according to: - https://github.com/actions/setup-python/issues/355#issuecomment-1335042510 But https://endoflife.date/python says that 3.6 end of life was almost 6 years ago, and end of security support nearly 1 year ago. So I'm OK dropping support entirely - Python 3.6 users will still be able to install version 3.30, just not any releases that come next.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/517/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1479914599,I_kwDOCGYnMM5YNbRn,516,Feature request: output number of ignored/replaced rows for insert command,9599,simonw,open,0,,,,,4,2022-12-06T18:59:21Z,2022-12-06T19:08:14Z,,OWNER,,"https://hachyderm.io/@briandorsey/109468185742876820 > I'm fiddling with piping json to `insert -ignore` I'd love to see the count of records inserted & ignored, but didn't see a way to do that in the help/docs. > > Example: `xh ""https://hachyderm.io/api/v1/timelines/tag/rust?max_id=109443380308326328"" | sqlite-utils insert aoc.db aoc - --pk=id --ignore`",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/516/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1450952393,I_kwDOCGYnMM5We8bJ,512,mypy failures in CI,9599,simonw,closed,0,,,,,3,2022-11-16T06:22:48Z,2022-11-16T07:49:51Z,2022-11-16T07:49:50Z,OWNER,,"https://github.com/simonw/sqlite-utils/actions/runs/3472012235 failed on Python 3.11: Truncated output: ``` sqlite_utils/db.py:2467: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True sqlite_utils/db.py:2467: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase sqlite_utils/db.py:2530: error: Incompatible default for argument ""where"" (default has type ""None"", argument has type ""str"") [assignment] sqlite_utils/db.py:2530: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True sqlite_utils/db.py:2530: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase sqlite_utils/db.py:2658: error: Argument 1 to ""count_where"" of ""Queryable"" has incompatible type ""Optional[str]""; expected ""str"" [arg-type] Found 23 errors in 1 file (checked 51 source files) ``` Best look at https://github.com/hauntsaninja/no_implicit_optional",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/512/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 473083260,MDU6SXNzdWU0NzMwODMyNjA=,50,"""Too many SQL variables"" on large inserts",9599,simonw,closed,0,,,,,4,2019-07-25T21:43:31Z,2022-11-04T14:38:36Z,2019-07-28T11:59:33Z,OWNER,,"Reported here: https://github.com/dogsheep/healthkit-to-sqlite/issues/9 It looks like there's a default limit of 999 variables - we need to be smart about that, maybe dynamically lower the batch size based on the number of columns.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/50/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1429029604,I_kwDOCGYnMM5VLULk,506,Make `cursor.rowcount` accessible (wontfix),9599,simonw,closed,0,,,,,3,2022-10-30T21:51:55Z,2022-11-01T17:37:47Z,2022-11-01T17:37:13Z,OWNER,,"In building this Datasette feature on top of `sqlite-utils` I thought it might be useful to expose the number of rows that had been affected by a bulk insert or update - the `cursor.rowcount`: - https://github.com/simonw/datasette/issues/1866 This isn't currently exposed by `sqlite-utils`.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/506/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1423182778,I_kwDOCGYnMM5U1Au6,505,Release sqlite-utils 3.30,9599,simonw,closed,0,,,,,2,2022-10-25T22:20:05Z,2022-10-25T22:41:26Z,2022-10-25T22:41:16Z,OWNER,,https://github.com/simonw/sqlite-utils/compare/3.29...defa2974c6d3abc19be28d6b319649b8028dc966,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/505/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1386562662,I_kwDOCGYnMM5SpURm,493,Tiny typographical error in install/uninstall docs,9599,simonw,open,0,,,,,3,2022-09-26T19:00:42Z,2022-10-25T21:31:15Z,,OWNER,,"Added in: - #483 I don't know how to fix this in Sphinx: I'm getting this: https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install > The [insert –convert](https://sqlite-utils.datasette.io/en/latest/cli.html#cli-insert-convert) and [query –functions](https://sqlite-utils.datasette.io/en/latest/cli.html#cli-query-functions) options But I want it to display `insert --convert` and not `insert –convert` there. Here's the code: https://github.com/simonw/sqlite-utils/blob/85247038f70d7eb2f3e272cfeaa4c44459cafba8/docs/cli.rst#L2125",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/493/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1423069384,I_kwDOCGYnMM5U0lDI,504,"db.close() method, calling db.conn.close()",9599,simonw,closed,0,,,,,1,2022-10-25T20:50:50Z,2022-10-25T21:00:29Z,2022-10-25T20:57:47Z,OWNER,,"I ended up needing to use `db.conn.close()` to fix this issue: - #503 I think `.close()` should be a method on `Database` itself.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/504/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1423000702,I_kwDOCGYnMM5U0UR-,503,test_recreate failing on Windows Python 3.11,9599,simonw,closed,0,,,,,10,2022-10-25T20:01:41Z,2022-10-25T20:47:34Z,2022-10-25T20:45:43Z,OWNER,,"https://github.com/simonw/sqlite-utils/actions/runs/3323672128/jobs/5494726927 Related: - #502 ``` FAILED tests/test_recreate.py::test_recreate[True-True] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_True_True_0\\data.db' FAILED tests/test_recreate.py::test_recreate[False-True] - PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_recreate_False_True_0\\data.db' ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/503/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1422954582,I_kwDOCGYnMM5U0JBW,502,Fix tests for Python 3.11,9599,simonw,closed,0,,,,,1,2022-10-25T19:20:31Z,2022-10-25T19:23:47Z,2022-10-25T19:23:47Z,OWNER,,"The way errors are represented has changed: https://github.com/simonw/sqlite-utils/actions/runs/3323588047/jobs/5494127154 ``` _________________________ test_query_invalid_function __________________________ db_path = '/tmp/pytest-of-runner/pytest-0/test_query_invalid_function0/test.db' def test_query_invalid_function(db_path): result = CliRunner().invoke( cli.cli, [db_path, ""select bad()"", ""--functions"", ""def invalid_python""] ) assert result.exit_code == 1 > assert ( result.output.strip() == ""Error: Error in functions definition: invalid syntax (, line 1)"" ) E AssertionError: assert 'Error: Error...ing>, line 1)' == 'Error: Error...ing>, line 1)' E - Error: Error in functions definition: invalid syntax (, line 1) E ? ^^^^^^ ^^^^^^ E + Error: Error in functions definition: expected '(' (, line 1) E ? ^^^^^^^ ^^^ ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/502/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1413641049,I_kwDOCGYnMM5UQnNZ,501,Tests failing due to updated tabulate library,9599,simonw,closed,0,,,,,4,2022-10-18T18:07:52Z,2022-10-18T18:23:40Z,2022-10-18T18:23:40Z,OWNER,,"Failure here: https://github.com/simonw/sqlite-utils/actions/runs/3275786702/jobs/5391063221 I figured out the problem: ```diff diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index b88e38a..82b4b6c 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -112,11 +112,15 @@ See :ref:`cli_query`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, - simple, textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, + latex, latex_booktabs, latex_longtable, latex_raw, + mediawiki, mixed_grid, mixed_outline, moinmoin, + orgtbl, outline, pipe, plain, presto, pretty, + psql, rounded_grid, rounded_outline, rst, simple, + simple_grid, simple_outline, textile, tsv, + unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row @@ -176,11 +180,15 @@ See :ref:`cli_memory`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, - simple, textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, + latex, latex_booktabs, latex_longtable, latex_raw, + mediawiki, mixed_grid, mixed_outline, moinmoin, + orgtbl, outline, pipe, plain, presto, pretty, + psql, rounded_grid, rounded_outline, rst, simple, + simple_grid, simple_outline, textile, tsv, + unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row @@ -401,11 +409,14 @@ See :ref:`cli_search`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --load-extension TEXT Path to SQLite extension, with optional :entrypoint @@ -651,11 +662,14 @@ See :ref:`cli_tables`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --columns Include list of columns for each table @@ -689,11 +703,14 @@ See :ref:`cli_views`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --columns Include list of columns for each view @@ -732,11 +749,15 @@ See :ref:`cli_rows`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, - simple, textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, + latex, latex_booktabs, latex_longtable, latex_raw, + mediawiki, mixed_grid, mixed_outline, moinmoin, + orgtbl, outline, pipe, plain, presto, pretty, + psql, rounded_grid, rounded_outline, rst, simple, + simple_grid, simple_outline, textile, tsv, + unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --load-extension TEXT Path to SQLite extension, with optional @@ -768,11 +789,14 @@ See :ref:`cli_triggers`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --load-extension TEXT Path to SQLite extension, with optional :entrypoint @@ -804,11 +828,14 @@ See :ref:`cli_indexes`. --tsv Output TSV --no-headers Omit CSV headers -t, --table Output as a formatted table - --fmt TEXT Table format - one of fancy_grid, fancy_outline, - github, grid, html, jira, latex, latex_booktabs, - latex_longtable, latex_raw, mediawiki, moinmoin, - orgtbl, pipe, plain, presto, pretty, psql, rst, simple, - textile, tsv, unsafehtml, youtrack + --fmt TEXT Table format - one of asciidoc, double_grid, + double_outline, fancy_grid, fancy_outline, github, + grid, heavy_grid, heavy_outline, html, jira, latex, + latex_booktabs, latex_longtable, latex_raw, mediawiki, + mixed_grid, mixed_outline, moinmoin, orgtbl, outline, + pipe, plain, presto, pretty, psql, rounded_grid, + rounded_outline, rst, simple, simple_grid, + simple_outline, textile, tsv, unsafehtml, youtrack --json-cols Detect JSON cols and output them as JSON, not escaped strings --load-extension TEXT Path to SQLite extension, with optional :entrypoint diff --git a/docs/cli.rst b/docs/cli.rst index 8bc4176..1d67e88 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -187,10 +187,15 @@ Available ``--fmt`` options are: cog.out(""\n"" + ""\n"".join('- ``{}``'.format(t) for t in tabulate.tabulate_formats) + ""\n\n"") .. ]]] +- ``asciidoc`` +- ``double_grid`` +- ``double_outline`` - ``fancy_grid`` - ``fancy_outline`` - ``github`` - ``grid`` +- ``heavy_grid`` +- ``heavy_outline`` - ``html`` - ``jira`` - ``latex`` @@ -198,15 +203,22 @@ Available ``--fmt`` options are: - ``latex_longtable`` - ``latex_raw`` - ``mediawiki`` +- ``mixed_grid`` +- ``mixed_outline`` - ``moinmoin`` - ``orgtbl`` +- ``outline`` - ``pipe`` - ``plain`` - ``presto`` - ``pretty`` - ``psql`` +- ``rounded_grid`` +- ``rounded_outline`` - ``rst`` - ``simple`` +- ``simple_grid`` +- ``simple_outline`` - ``textile`` - ``tsv`` - ``unsafehtml`` ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/501/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1413610718,I_kwDOCGYnMM5UQfze,500,Turn --flatten into a documented utility function,9599,simonw,closed,0,,,,,4,2022-10-18T17:43:36Z,2022-10-18T18:02:10Z,2022-10-18T18:00:40Z,OWNER,,The `--flatten` implementation isn't currently available to Python code - people have to roll their own implementation. Feedback from a conversation at DjangoCon.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/500/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1149661489,I_kwDOCGYnMM5EhnEx,409,`with db:` for transactions,9599,simonw,open,0,,,,,3,2022-02-24T19:22:06Z,2022-10-01T03:42:50Z,,OWNER,,This can be a documented wrapper around `with db.conn:`.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/409/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1386593843,I_kwDOCGYnMM5Spb4z,494,Document how to use Just,9599,simonw,closed,0,,,,,2,2022-09-26T19:25:12Z,2022-09-26T19:32:36Z,2022-09-26T19:26:39Z,OWNER,,"I'm using `just` a lot know, based on this file - I should add that to https://sqlite-utils.datasette.io/en/latest/contributing.html https://github.com/simonw/sqlite-utils/blob/afbd2b2cba45cccb305c3d4638d18db4dd3d4bbd/Justfile#L1-L24",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/494/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1363765916,I_kwDOCGYnMM5RSWqc,483,`sqlite-utils install` command,9599,simonw,closed,0,,,,,2,2022-09-06T20:13:55Z,2022-09-26T19:04:43Z,2022-09-26T18:57:15Z,OWNER,,"With the addition of `--functions` in: - #471 In addition to the existing `convert` command, there are now very good reasons to want to install additional packages into the same virtual environment as `sqlite-utils` itself, to allow them to be used with those features. This isn't easy if you installed the tool with `pipx` or `brew install sqlite-utils`. Datasette solved this problem with the `datasette install` command: - https://github.com/simonw/datasette/issues/925 `sqlite-utils` could benefit from the same idea.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/483/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1386530156,I_kwDOCGYnMM5SpMVs,492,Idea: ability to pass extra variables to `--convert` scripts,9599,simonw,open,0,,,,,1,2022-09-26T18:30:45Z,2022-09-26T18:33:19Z,,OWNER,,"Got this idea from this example in https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/ ```bash sqlite-utils insert /tmp/kafka-logs.db logs server.log.2022-09-24-21 --text --convert "" import re r = re.compile(r'^\[(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P\w+) (?P(.+(\n(?\!\[).+|)+))', re.MULTILINE) def convert(text): rows = [m.groupdict() for m in r.finditer(text)] for row in rows: row.update({'server': 'localhost'}) row.update({'component': 'broker'}) return rows "" ``` And the accompanying note: > The `row.update` allows to label rows as I’m planning to ingest logs from different hosts and potentially different components. This made me think: it might be neat if you could inject additional variable values into that script with extra command-line options, to make this kind of reuse easier. Something like this: ```bash sqlite-utils insert /tmp/kafka-logs.db logs server.log.2022-09-24-21 --text --convert "" import re r = re.compile(r'^\[(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P\w+) (?P(.+(\n(?\!\[).+|)+))', re.MULTILINE) def convert(text): rows = [m.groupdict() for m in r.finditer(text)] for row in rows: row.update({'server': server}) row.update({'component': component}) return rows "" --var server ""localhost"" --var component ""broker"" ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/492/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1374939463,I_kwDOCGYnMM5R8-lH,489,Ability to load JSON records held in a file with a single top level key that is a list of objects,9599,simonw,open,0,,,,,9,2022-09-15T18:46:03Z,2022-09-15T20:56:10Z,,OWNER,,"It's very common for JSON to look like this: ```json { ""Version"": ""5.5.52.6"", ""List"": [ { ""Description"": ""Nonpartisan"", ""Id"": 1, ""ExternalId"": """" }, { ""Description"": ""Undeclared"", ""Id"": 2, ""ExternalId"": """" } ] } ``` This example taken from the records downloaded from https://www.elections.alaska.gov/election-results/e/ Right now you can't import this into `sqlite-utils` - you need to run it through `jq .List` first. But since this is so common, it would be neat if `sqlite-utils` could have a rule of thumb that says ""if it's an object, but it has a single key that is is a list of objects, use that instead"".",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/489/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1363766973,I_kwDOCGYnMM5RSW69,484,Expose convert recipes to `sqlite-utils --functions`,9599,simonw,open,0,,,,,11,2022-09-06T20:15:08Z,2022-09-07T19:09:52Z,,OWNER,,"`--functions` was added in: - #471 It would be useful if the `r.jsonsplit()` and similar recipes for `sqlite-utils convert` could be used in these blocks of code too: https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/484/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1352932716,I_kwDOCGYnMM5QpB1s,471,sqlite-utils query --functions mechanism for registering extra functions,9599,simonw,closed,0,,,8355157,3.29,12,2022-08-27T03:57:53Z,2022-09-07T03:46:26Z,2022-08-27T05:10:57Z,OWNER,,"It would be really cool if you could register additional custom SQL functions for use with the `sqlite-utils query` command - something like this: ``` sqlite-utils data.db 'update images set domain = extract_domain(url)' --functions ' from urllib.parse import urlparse def extract_domain(url): return urlparse(url).netloc ' ``` Every function defined in that code block would be registered with the connection, unless the name began with an underscore.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/471/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 816526538,MDU6SXNzdWU4MTY1MjY1Mzg=,239,sqlite-utils extract could handle nested objects,9599,simonw,open,0,,,,,16,2021-02-25T15:10:28Z,2022-09-03T23:46:02Z,,OWNER,,"Imagine a table (imported from a nested JSON file) where one of the columns contains values that look like this: {""email"": ""anonymous@noreply.airtable.com"", ""id"": ""usrROSHARE0000000"", ""name"": ""Anonymous""} The `sqlite-utils extract` command already uses single text values in a column to populate a new table. It would not be much of a stretch for it to be able to use JSON instead, including specifying which of those values should be used as the primary key in the new table.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/239/reactions"", ""total_count"": 6, ""+1"": 5, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1359604075,I_kwDOCGYnMM5RCelr,481,"Idea: `sqlite-utils create-table tablename --sql ""select ...""`",9599,simonw,open,0,,,,,0,2022-09-02T01:41:24Z,2022-09-02T01:42:08Z,,OWNER,,"Could offer syntactic sugar for: ```sql create table foo as select * from bar ``` ``` sqlite-utils create-table data.db foo --sql ""select * from bar"" ``` https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/481/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1353481513,I_kwDOCGYnMM5QrH0p,478,`sqlite-utils tables data.db table1 table2`,9599,simonw,open,0,,,,,1,2022-08-28T22:05:53Z,2022-08-28T22:22:35Z,,OWNER,,"The `sqlite-utils tables` command currently lists all tables. If you have a huge table in there then running it with `--counts` can get expensive, because of the huge table. Would be useful if it could accept an optional list of tables that it should execute against, as an alternative to the default of all of them. This should be a backwards compatible change. Current design is: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#tables ``` Usage: sqlite-utils tables [OPTIONS] PATH List the tables in the database Example: sqlite-utils tables trees.db ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/478/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1353196970,I_kwDOCGYnMM5QqCWq,476,Release notes for 3.29,9599,simonw,closed,0,,,8355157,3.29,2,2022-08-27T23:21:21Z,2022-08-28T04:07:15Z,2022-08-28T04:07:03Z,OWNER,,https://github.com/simonw/sqlite-utils/compare/3.28...104f37fa4d2e7e5999c1d829267b62c737f74d3e,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/476/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1348169997,I_kwDOCGYnMM5QW3EN,467,Mechanism for ensuring a table has all the columns,9599,simonw,closed,0,,,8355157,3.29,13,2022-08-23T15:50:23Z,2022-08-27T23:19:41Z,2022-08-27T23:17:56Z,OWNER,,Suggested by @jefftriplett on Discord: https://discord.com/channels/823971286308356157/997738192360964156/1011655389063958600,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/467/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1348294436,PR_kwDOCGYnMM49qP2V,468,"db[table].create(..., transform=True) and create-table --transform",9599,simonw,closed,0,,,8355157,3.29,6,2022-08-23T17:27:58Z,2022-08-27T23:17:55Z,2022-08-27T23:17:55Z,OWNER,simonw/sqlite-utils/pulls/468,"Work in progress. Still needs documentation and tests (and to cover more cases of things that might have changed). Refs: - #467 ---- :books: Documentation preview :books:: https://sqlite-utils--468.org.readthedocs.build/en/468/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/468/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1353189941,I_kwDOCGYnMM5QqAo1,475,table.default_values introspection property,9599,simonw,closed,0,,,8355157,3.29,1,2022-08-27T22:33:31Z,2022-08-27T22:44:46Z,2022-08-27T22:43:02Z,OWNER,,"> Interesting challenge with `default_value`: I need to be able to tell if the default values passed to `.create()` differ from those in the database already. > > Introspecting that is a bit tricky: > > ```pycon > >>> import sqlite_utils > >>> db = sqlite_utils.Database(memory=True) > >>> db[""blah""].create({""id"": int, ""name"": str}, not_null=(""name"",), defaults={""name"": ""bob""}) >
> >>> db[""blah""].columns > [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=""'bob'"", is_pk=0)] > ``` > Note how a default value of the Python string `bob` is represented in the results of `PRAGMA table_info()` as `default_value=""'bob'""` - it's got single quotes added to it! > > So comparing default values from introspecting the database needs me to first parse that syntax. This may require a new table introspection method. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/468#issuecomment-1229279539_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/475/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1352953535,PR_kwDOCGYnMM4950Az,473,Support entrypoints for `--load-extension`,9599,simonw,closed,0,,,,,1,2022-08-27T05:53:59Z,2022-08-27T05:55:52Z,2022-08-27T05:55:47Z,OWNER,simonw/sqlite-utils/pulls/473,"Refs #470 ---- :books: Documentation preview :books:: https://sqlite-utils--473.org.readthedocs.build/en/473/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/473/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1352932038,I_kwDOCGYnMM5QpBrG,470,Upgrade `--load-extension` to accept entrypoints like Datasette,9599,simonw,closed,0,,,8355157,3.29,6,2022-08-27T03:53:20Z,2022-08-27T05:55:49Z,2022-08-27T05:55:48Z,OWNER,,"Imitate: - https://github.com/simonw/datasette/pull/1789 ``` # would load default entrypoint like before datasette data.db --load-extension ext # loads the extensions with the ""sqlite3_foo_init"" entrpoint datasette data.db --load-extension ext:sqlite3_foo_init # loads the extensions with the ""sqlite3_bar_init"" entrpoint datasette data.db --load-extension ext:sqlite3_bar_init ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/470/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1352946135,I_kwDOCGYnMM5QpFHX,472,Reuse the locals/globals fix from --functions for other code accepting options,9599,simonw,closed,0,,,8355157,3.29,2,2022-08-27T05:12:05Z,2022-08-27T05:20:12Z,2022-08-27T05:20:12Z,OWNER,,"I figured out a workaround for the ugly `global x` hack here: - https://github.com/simonw/sqlite-utils/issues/471#issuecomment-1229120653",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/472/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1352931464,I_kwDOCGYnMM5QpBiI,469,sqlite-utils rows --order option,9599,simonw,closed,0,,,8355157,3.29,1,2022-08-27T03:49:51Z,2022-08-27T04:30:49Z,2022-08-27T04:10:32Z,OWNER,,"For consistency with `search`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search ``` -o, --order TEXT Order by ('column' or 'column desc') ``` I wanted to run `sqlite-utils rows db.db mytable --order 'rowid desc'` to see the most recently imported rows.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/469/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1320243134,I_kwDOCGYnMM5OsU--,458,Support custom names for registered functions,9599,simonw,closed,0,,,8355157,3.29,1,2022-07-28T00:13:00Z,2022-08-27T03:56:01Z,2022-07-28T00:13:57Z,OWNER,,"In this example: ```python @db.register_function def reverse_string(s): return """".join(reversed(list(s))) print(db.execute('select reverse_string(""hello"")').fetchone()[0]) ``` There's currently no way to over-ride the automatically selected name for the SQL function.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/458/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1319881016,PR_kwDOCGYnMM48Mmde,457,Link to installation instructions,9599,simonw,closed,0,,,8355157,3.29,2,2022-07-27T17:38:36Z,2022-08-27T03:55:52Z,2022-07-27T17:57:50Z,OWNER,simonw/sqlite-utils/pulls/457,Also testing https://docs.readthedocs.io/en/stable/pull-requests.html,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/457/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1326087800,PR_kwDOCGYnMM48hI-_,460,Cross-link CLI to Python docs,9599,simonw,closed,0,,,,,4,2022-08-02T16:18:28Z,2022-08-18T21:58:10Z,2022-08-18T21:58:07Z,OWNER,simonw/sqlite-utils/pulls/460,"Work in progress, partly to test the ReadTheDocs preview link action. Refs: - #426 ---- :books: Documentation preview :books:: https://readthedocs-preview--460.org.readthedocs.build/en/460/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/460/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1342357149,PR_kwDOCGYnMM49Wsnq,465,beanbag-docutils>=2.0,9599,simonw,closed,0,,,,,2,2022-08-17T22:41:39Z,2022-08-17T23:38:07Z,2022-08-17T23:38:02Z,OWNER,simonw/sqlite-utils/pulls/465,Refs #464,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/465/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1338001039,I_kwDOCGYnMM5PwEaP,464,Link from documentation to source code,9599,simonw,closed,0,,,,,5,2022-08-13T16:19:57Z,2022-08-17T23:38:03Z,2022-08-17T23:38:03Z,OWNER,,Twitter conversation asking for ways to automate this here: https://twitter.com/simonw/status/1558260492015046656,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/464/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1342374388,PR_kwDOCGYnMM49Wv9T,466,Use Read the Docs action v1 (#463),9599,simonw,closed,0,,,,,0,2022-08-17T23:11:50Z,2022-08-17T23:11:54Z,2022-08-17T23:11:54Z,OWNER,simonw/sqlite-utils/pulls/466,"Read the Docs repository was renamed from `readthedocs/readthedocs-preview` to `readthedocs/actions/`. Now, the `preview` action is under `readthedocs/actions/preview` and is tagged as `v1`",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/466/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1326391841,PR_kwDOCGYnMM48iLGF,462,Discord badge,9599,simonw,closed,0,,,,,2,2022-08-02T20:56:04Z,2022-08-02T21:15:57Z,2022-08-02T21:15:52Z,OWNER,simonw/sqlite-utils/pulls/462,"Also testing fix for: - https://github.com/readthedocs/readthedocs-preview/issues/10 ---- :books: Documentation preview :books:: https://sqlite-utils--462.org.readthedocs.build/en/462/ ",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/462/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1326349129,I_kwDOCGYnMM5PDntJ,461,Consider including animated SVG console demos,9599,simonw,open,0,,,,,1,2022-08-02T20:10:04Z,2022-08-02T20:12:14Z,,OWNER,,"I recorded this one using https://github.com/nbedos/termtosvg - with `pipx install termtosvg` and then `termtosvg` - execute demo - `exit` to save. ![sqlite-utils-insert-json](https://user-images.githubusercontent.com/9599/182464206-f4976af4-eda8-4020-8257-4ada1867fb44.svg) ```json [ { ""id"": 1, ""name"": ""Catimus"" }, { ""id"": 2, ""name"": ""Feliopia"" } ] ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/461/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1292060682,I_kwDOCGYnMM5NA0gK,450,Add --ignore option to more commands,9599,simonw,closed,0,,,,,9,2022-07-02T13:52:02Z,2022-07-15T22:39:09Z,2022-07-15T22:37:45Z,OWNER,,"As seen in https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key Could make this TIL trick unnecessary: https://til.simonwillison.net/bash/ignore-errors",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/450/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1298531653,I_kwDOCGYnMM5NZgVF,451,Make sqlite_utils.utils.chunks a documented function,9599,simonw,closed,0,,,,,2,2022-07-08T06:01:04Z,2022-07-15T22:09:34Z,2022-07-15T21:59:33Z,OWNER,,I want to use it in another project: https://github.com/simonw/sqlite-utils/blob/8a9fe6498faf783a1fdeb1793e661ad194a05267/sqlite_utils/utils.py#L471-L474,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/451/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1306548397,I_kwDOCGYnMM5N4Fit,454,CLI command for duplicating tables,9599,simonw,closed,0,,,,,1,2022-07-15T21:31:27Z,2022-07-15T21:48:23Z,2022-07-15T21:45:51Z,OWNER,,"CLI equivalent of: - #449",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/454/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1271426387,I_kwDOCGYnMM5LyG1T,444,CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool,9599,simonw,open,0,,,,,5,2022-06-14T22:22:47Z,2022-07-07T16:39:18Z,,OWNER,,"> I forgot to add equivalents of `extras_key=` and `ignore_extras=` to the CLI tool - will do that in a separate issue. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/444/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 455486286,MDU6SXNzdWU0NTU0ODYyODY=,26,Mechanism for turning nested JSON into foreign keys / many-to-many,9599,simonw,open,0,,,,,14,2019-06-13T00:52:06Z,2022-06-29T23:35:29Z,,OWNER,,"The GitHub JSON APIs have a really interesting convention with respect to related objects. Consider https://api.github.com/repos/simonw/sqlite-utils/issues - here's a truncated subset: ```json { ""id"": 449818897, ""node_id"": ""MDU6SXNzdWU0NDk4MTg4OTc="", ""number"": 24, ""title"": ""Additional Column Constraints?"", ""user"": { ""login"": ""IgnoredAmbience"", ""id"": 98555, ""node_id"": ""MDQ6VXNlcjk4NTU1"", ""avatar_url"": ""https://avatars0.githubusercontent.com/u/98555?v=4"", ""gravatar_id"": """" }, ""labels"": [ { ""id"": 993377884, ""node_id"": ""MDU6TGFiZWw5OTMzNzc4ODQ="", ""url"": ""https://api.github.com/repos/simonw/sqlite-utils/labels/enhancement"", ""name"": ""enhancement"", ""color"": ""a2eeef"", ""default"": true } ], ""state"": ""open"" } ``` The `user` column lists a complete user. The `labels` column has a list of labels. Since both user and label have populated `id` field this is actually enough information for us to create records for them AND set up the corresponding foreign key (for user) and m2m relationships (for labels). It would be really neat if `sqlite-utils` had some kind of mechanism for correctly processing these kind of patterns. Thanks to `jq` there's not much need for extra customization of the shape here - if we support a narrowly defined structure users can use `jq` to reshape arbitrary JSON to match.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/26/reactions"", ""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1277328147,I_kwDOCGYnMM5MInsT,446,Use Just to automate running tests and linters locally,9599,simonw,closed,0,,,,,2,2022-06-20T19:51:09Z,2022-06-21T19:28:35Z,2022-06-20T19:54:50Z,OWNER,,I keep committing code that fails additional tests like `mypy` and `flake8` and `black`. Automate those using Just.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/446/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1278571700,I_kwDOCGYnMM5MNXS0,447,Incorrect syntax highlighting in docs CLI reference,9599,simonw,closed,0,,,,,3,2022-06-21T14:53:10Z,2022-06-21T18:48:47Z,2022-06-21T18:48:46Z,OWNER,,"https://sqlite-utils.datasette.io/en/stable/cli-reference.html#insert ![CE020DDA-27FB-49C3-9EA6-37457DC4C321](https://user-images.githubusercontent.com/9599/174830380-06530537-b870-41c0-a8af-03c7fa720c6f.jpeg) It looks like Python keywords are being incorrectly highlighted here.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/447/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1269998342,I_kwDOCGYnMM5LsqMG,443,Make `utils.rows_from_file()` a documented API,9599,simonw,closed,0,,,,,2,2022-06-13T21:53:24Z,2022-06-20T19:49:37Z,2022-06-14T20:12:46Z,OWNER,,"> `rows_from_file()` isn't part of the documented API but maybe it should be! _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/443/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1277295119,I_kwDOCGYnMM5MIfoP,445,`sqlite_utils.utils.TypeTracker` should be a documented API,9599,simonw,closed,0,,,,,3,2022-06-20T19:08:28Z,2022-06-20T19:49:02Z,2022-06-20T19:46:58Z,OWNER,,"I've used it in a couple of external places now: - https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/__init__.py#L264-L280 - https://github.com/simonw/datasette-lite/blob/caa8eade10f0321c64f9f65c4561186f02d57c5b/webworker.js#L55-L64 Refs: - https://github.com/simonw/datasette-lite/issues/32",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/445/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1269886084,I_kwDOCGYnMM5LsOyE,442,`maximize_csv_field_size_limit()` utility function,9599,simonw,closed,0,,,,,2,2022-06-13T19:54:54Z,2022-06-14T21:55:15Z,2022-06-14T21:31:49Z,OWNER,,"This code here runs only if `cli.py` is imported: https://github.com/simonw/sqlite-utils/blob/7ddf5300886a32d6daf60cf1d71efe492b65c87e/sqlite_utils/cli.py#L50-L59 I found myself needing the same fix in another library: - https://github.com/simonw/datasette-socrata/issues/13 It should be a documented utility function.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/442/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1160182768,I_kwDOCGYnMM5FJvvw,412,Optional Pandas integration,9599,simonw,open,0,,,,,13,2022-03-05T01:49:27Z,2022-06-14T15:36:29Z,,OWNER,,"It would be neat if there was a way to use this more seamlessly with Pandas, in particular Pandas dataframes - but without making Pandas a required dependency.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/412/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1243715381,I_kwDOCGYnMM5KIZc1,436,"Add ""copy to clipboard"" button to code examples in documentation",9599,simonw,closed,0,,,,,0,2022-05-20T21:53:23Z,2022-05-20T21:57:53Z,2022-05-20T21:57:53Z,OWNER,,"Follows: - #435 Imitates: - https://github.com/simonw/datasette/issues/1748 I'll use https://github.com/executablebooks/sphinx-copybutton - here's the Datasette commit: https://github.com/simonw/datasette/commit/1465fea4798599eccfe7e8f012bd8d9adfac3039",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/436/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1243704847,I_kwDOCGYnMM5KIW4P,435,Switch to Furo documentation theme,9599,simonw,closed,0,,,,,2,2022-05-20T21:46:39Z,2022-05-20T21:56:10Z,2022-05-20T21:54:43Z,OWNER,,"As seen in: - https://github.com/simonw/datasette/issues/1746 - https://github.com/simonw/shot-scraper/issues/77",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/435/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1223177069,PR_kwDOCGYnMM43LrKB,429,Depend on click-default-group-wheel,9599,simonw,closed,0,,,,,2,2022-05-02T18:03:10Z,2022-05-02T18:52:42Z,2022-05-02T18:05:00Z,OWNER,simonw/sqlite-utils/pulls/429,"Trying to get this to work with Pyodide. Refs: https://github.com/simonw/click-default-group-wheel/issues/3",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/429/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1215216249,I_kwDOCGYnMM5Ibrp5,428,Research adding support for savepoints,9599,simonw,open,0,,,,,1,2022-04-26T01:04:01Z,2022-04-26T01:05:29Z,,OWNER,,"https://www.sqlite.org/lang_savepoint.html Savepoints are like regular transactions except they have names and can be nested. Would there be any value in adding support to them to `sqlite-utils`, potentially as some kind of context manager? Something like this: ```python with db.savepoint(""name""): # do stuff with db.savepoint(""name2""): # do more stuff raise Release # Rolls back to before ""name2"" savepoint ``` I've never used this feature so I'm not comfortable adding anything like this without a bunch of extra research.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/428/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1200866134,I_kwDOCGYnMM5Hk8NW,424,Better error message if you try to create a table with no columns,9599,simonw,closed,0,,,,,1,2022-04-12T02:43:20Z,2022-04-13T22:40:15Z,2022-04-13T22:40:10Z,OWNER,,"Seen here: - https://github.com/simonw/geojson-to-sqlite/issues/30 Attempting to create a table with no columns produced this confusing error: ``` File ""/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/geojson_to_sqlite/utils.py"", line 69, in import_features db[table].create(column_types, pk=pk) File ""/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py"", line 863, in create self.db.create_table( File ""/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py"", line 517, in create_table self.execute(sql) File ""/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py"", line 236, in execute return self.conn.execute(sql) sqlite3.OperationalError: near "")"": syntax error ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/424/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1181236173,I_kwDOCGYnMM5GaDvN,422,Reconsider not running convert functions against null values,9599,simonw,open,0,,,,,1,2022-03-25T20:22:40Z,2022-03-25T20:23:21Z,,OWNER,,"I just got caught out by the fact that `None` values are not processed by the `.convert()` mechanism https://github.com/simonw/sqlite-utils/blob/0b7b80bd40fe86e4d66a04c9f607d94991c45c0b/sqlite_utils/db.py#L2504-L2510 I had run this code while working on #420 and I wasn't sure why it didn't work: ``` $ sqlite-utils add-column content.db articles score float $ sqlite-utils convert content.db articles score ' import random random.seed(10) def convert(value): global random return random.random() ' ``` The reason it didn't work is that the newly added `score` column was full of `null` values. I fixed it by doing this instead: $ sqlite-utils add-column content.db articles score float --not-null-default 1.0 But this indicates to me that the design of `convert()` here may be incorrect.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/422/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 688351054,MDU6SXNzdWU2ODgzNTEwNTQ=,140,Idea: insert-files mechanism for adding extra columns with fixed values,9599,simonw,open,0,,,,,1,2020-08-28T20:57:36Z,2022-03-20T19:45:45Z,,OWNER,,"Say for example you want to populate a `file_type` column with the value `gif`. That could work like this: ``` sqlite-utils insert-files gifs.db images *.gif \ -c path -c md5 -c last_modified:mtime \ -c file_type:text:gif --pk=path ``` So a column defined as a `text` column with a value that follows a second colon.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 675753042,MDU6SXNzdWU2NzU3NTMwNDI=,131,sqlite-utils insert: options for column types,9599,simonw,open,0,,,,,5,2020-08-09T18:59:11Z,2022-03-15T13:21:42Z,,OWNER,,"The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs. It would be useful if you could do the following: - automatically detects the column types based on eg the first 1000 records - explicitly state the rule for specific columns `--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types` For specific columns maybe this: sqlite-utils insert db.db images images.tsv \ --tsv \ -c id int \ -c score float",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1166731361,I_kwDOCGYnMM5Fiuhh,414,I forgot to include the changelog in the 3.25.1 release,9599,simonw,closed,0,,,,,7,2022-03-11T18:32:36Z,2022-03-11T18:40:39Z,2022-03-11T18:40:39Z,OWNER,,"I pushed a release for https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 but forgot to include the release notes in `docs/changelog.rst` This means https://sqlite-utils.datasette.io/en/stable/changelog.html isn't showing them.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/414/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1166587040,I_kwDOCGYnMM5FiLSg,413,Display autodoc type information more legibly,9599,simonw,closed,0,,,,,5,2022-03-11T15:58:20Z,2022-03-11T18:07:10Z,2022-03-11T18:07:10Z,OWNER,,"https://sqlite-utils.datasette.io/en/3.25/reference.html#sqlite_utils.db.Table.insert looks like this at the moment: ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/413/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1125297737,I_kwDOCGYnMM5DEq5J,402,Advanced class-based `conversions=` mechanism,9599,simonw,open,0,,,,,14,2022-02-06T19:47:41Z,2022-02-16T10:18:55Z,,OWNER,,"The `conversions=` parameter works like this at the moment: https://sqlite-utils.datasette.io/en/3.23/python-api.html#converting-column-values-using-sql-functions ```python db[""places""].insert( {""name"": ""Wales"", ""geometry"": wkt}, conversions={""geometry"": ""GeomFromText(?, 4326)""}, ) ``` This proposal is to support values in that dictionary that are objects, not strings, which can represent more complex conversions - spun out from #399. New proposed mechanism: ```python from sqlite_utils.utils import LongitudeLatitude db[""places""].insert( { ""name"": ""London"", ""point"": (-0.118092, 51.509865) }, conversions={""point"": LongitudeLatitude}, ) ``` Here `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string. This would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function. Best of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer! _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/402/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1128139375,I_kwDOCGYnMM5DPgpv,405,"`Database(memory_name=""name"")` constructor argument",9599,simonw,closed,0,,,,,2,2022-02-09T07:15:03Z,2022-02-16T01:23:16Z,2022-02-16T01:23:16Z,OWNER,,"SQLite in-memory databases can be named, in which case multiple connections can be opened to a shared in-memory database running within the same process. Datasette supports this - SQLite could support it too. https://docs.datasette.io/en/0.60.2/internals.html#database-ds-path-none-is-mutable-false-is-memory-false-memory-name-none",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/405/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1128120451,I_kwDOCGYnMM5DPcCD,404,Add example of `--convert` to the help for `sqlite-utils insert`,9599,simonw,closed,0,,,,,2,2022-02-09T06:49:09Z,2022-02-09T06:56:35Z,2022-02-09T06:55:16Z,OWNER,,"https://sqlite-utils.datasette.io/en/3.23/cli-reference.html#insert would be more useful if it included an example of `--convert` in action. I can maybe use an example from https://simonwillison.net/2022/Jan/11/sqlite-utils/",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/404/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1125081640,I_kwDOCGYnMM5DD2Io,401,Update SpatiaLite example in the documentation,9599,simonw,closed,0,,,,,2,2022-02-06T02:02:07Z,2022-02-06T02:05:03Z,2022-02-06T02:03:24Z,OWNER,,"This one here: https://sqlite-utils.datasette.io/en/3.23/python-api.html#converting-column-values-using-sql-functions It should take advantage of the new methods from: - #79",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/401/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1125077063,I_kwDOCGYnMM5DD1BH,400,`sqlite-utils create-table` ... `--if-not-exists`,9599,simonw,closed,0,,,,,1,2022-02-06T01:32:53Z,2022-02-06T01:34:53Z,2022-02-06T01:34:46Z,OWNER,,"Inspired by: - #397 To match the option on `create-index`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-index ``` --if-not-exists Ignore if index already exists ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/400/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 683805434,MDU6SXNzdWU2ODM4MDU0MzQ=,135,Code for finding SpatiaLite in the usual locations,9599,simonw,closed,0,,,,,3,2020-08-21T20:15:34Z,2022-02-05T00:04:26Z,2020-08-21T20:30:13Z,OWNER,,"I built this for `shapefile-to-sqlite` but it would be useful in `sqlite-utils` too: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19 ```python SPATIALITE_PATHS = ( ""/usr/lib/x86_64-linux-gnu/mod_spatialite.so"", ""/usr/local/lib/mod_spatialite.dylib"", ) ``` https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109 ```python def find_spatialite(): for path in SPATIALITE_PATHS: if os.path.exists(path): return path return None ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 683812642,MDU6SXNzdWU2ODM4MTI2NDI=,136,--load-extension=spatialite shortcut option,9599,simonw,closed,0,,,,,3,2020-08-21T20:31:25Z,2022-02-05T00:04:26Z,2020-10-16T19:14:32Z,OWNER,,In conjunction with #135 - this would do the same thing as `--load-extension=path-to-spatialite` (see #134),140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 723460107,MDU6SXNzdWU3MjM0NjAxMDc=,187,Maybe: Utility method / CLI tool for initializing SpatiaLite,9599,simonw,closed,0,,,,,2,2020-10-16T19:04:03Z,2022-02-05T00:04:26Z,2020-10-16T19:15:13Z,OWNER,,"> I think this should initialize SpatiaLite against the current database if it has not been initialized already. > > Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/187/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 557825032,MDU6SXNzdWU1NTc4MjUwMzI=,77,Ability to insert data that is transformed by a SQL function,9599,simonw,closed,0,,,,,2,2020-01-30T23:45:55Z,2022-02-05T00:04:25Z,2020-01-31T00:24:32Z,OWNER,,"I want to be able to run the equivalent of this SQL insert: ```python # Convert to ""Well Known Text"" format wkt = shape(geojson['geometry']).wkt # Insert and commit the record conn.execute(""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", ( ""Wales"", wkt )) conn.commit() ``` From the Datasette SpatiaLite docs: https://datasette.readthedocs.io/en/stable/spatialite.html To do this, I need a way of telling `sqlite-utils` that a specific column should be wrapped in `GeomFromText(?, 4326)`.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/77/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 557842245,MDU6SXNzdWU1NTc4NDIyNDU=,79,Helper methods for working with SpatiaLite,9599,simonw,closed,0,,,,,8,2020-01-31T00:39:19Z,2022-02-05T00:04:25Z,2022-02-04T05:55:11Z,OWNER,,"As demonstrated by this piece of documentation, using SpatiaLite with sqlite-utils requires a fair bit of boilerplate: https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/docs/python-api.rst#L880-L909",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/79/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1123849278,I_kwDOCGYnMM5C_JQ-,395,"""apt-get: command not found"" error on macOS",9599,simonw,closed,0,,,,,1,2022-02-04T06:03:42Z,2022-02-04T06:10:58Z,2022-02-04T06:10:58Z,OWNER,,"Yeah, `apt-get` isn't a thing on macOS so 4a2a3e2fd0d5534f446b3f1fee34cb165e4d86d2 (to test #79 against real SpatiaLite) broke.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/395/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1123851690,I_kwDOCGYnMM5C_J2q,396,"mypy failure, sqlite_utils/utils.py:56",9599,simonw,closed,0,,,,,0,2022-02-04T06:08:09Z,2022-02-04T06:10:33Z,2022-02-04T06:10:33Z,OWNER,,"https://github.com/simonw/sqlite-utils/runs/5062725880?check_suite_focus=true > `sqlite_utils/utils.py:56: error: Incompatible return value type (got ""None"", expected ""str"")`",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/396/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1072792507,I_kwDOCGYnMM4_8YO7,352,`sqlite-utils insert --extract colname`,9599,simonw,open,0,,,,,4,2021-12-07T00:55:44Z,2022-02-03T22:59:36Z,,OWNER,,"Is there a reason I've not added `--extract` as an option for `sqlite-utils insert` next? There's a `extracts=` option for the various `table.insert()` etc methods - last line in this code block: https://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L2483-L2495",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/352/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1094981339,I_kwDOCGYnMM5BRBbb,363,Better error message if `--convert` code fails to return a dict,9599,simonw,closed,0,,,,,4,2022-01-06T05:26:28Z,2022-02-03T22:52:30Z,2022-02-03T22:51:30Z,OWNER,,"Here's the traceback if your `--convert` function doesn't return a dict right now: ``` % sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1062, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 763, in invoke return __callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 949, in insert insert_upsert_implementation( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 834, in insert_upsert_implementation db[table].insert_all( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2602, in insert_all first_record = next(records) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 3044, in fix_square_braces for record in records: File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 831, in docs = (decode_base64_values(doc) for doc in docs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 86, in decode_base64_values to_fix = [ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py"", line 89, in if isinstance(doc[k], dict) TypeError: string indices must be integers ``` It would be nicer if that returned a more useful error message. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/361#issuecomment-1006295276_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/363/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1118585417,I_kwDOCGYnMM5CrEJJ,393,Better documentation for insert-replace,9599,simonw,closed,0,,,,,1,2022-01-30T15:40:23Z,2022-02-03T22:13:24Z,2022-02-03T22:13:24Z,OWNER,,"Currently: https://sqlite-utils.datasette.io/en/stable/python-api.html#insert-replacing-data > If you want to insert a record or replace an existing record with the same primary key, using the replace=True argument to .insert() or .insert_all(): Should describe the exception you get first, then how to use replace to avoid it.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/393/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097091527,I_kwDOCGYnMM5BZEnH,369,Research how much of a difference analyze / sqlite_stat1 makes,9599,simonw,closed,0,,,,,11,2022-01-09T03:03:36Z,2022-02-03T21:07:41Z,2022-02-03T21:07:35Z,OWNER,,"> Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it? _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050_ More generally: how much of a difference does the `sqlite_stat1` table created by `ANALYZE` make to queries? I'm particularly interested in `group by` / `count *` queries since Datasette uses those for faceting.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/369/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1122446693,I_kwDOCGYnMM5C5y1l,394,Test against Python 3.11-dev,9599,simonw,open,0,,,,,1,2022-02-02T22:21:03Z,2022-02-03T21:06:35Z,,OWNER,,"Same as: - https://github.com/simonw/datasette/issues/1621",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/394/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1114640101,I_kwDOCGYnMM5CcA7l,392,`sqlite-utils bulk --batch-size` option,9599,simonw,closed,0,,,,,4,2022-01-26T05:17:11Z,2022-01-26T18:17:59Z,2022-01-26T18:17:59Z,OWNER,,"> Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/392/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1114638930,I_kwDOCGYnMM5CcApS,391,`sqlite-utils bulk` progress bar,9599,simonw,closed,0,,,,,2,2022-01-26T05:14:49Z,2022-01-26T05:17:20Z,2022-01-26T05:16:51Z,OWNER,,"It can easily have a progress bar because it works by looping through an iterator: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018 Should also support the `--silent` option if I add this.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/391/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1114557284,I_kwDOCGYnMM5Cbstk,390,`sqlite-utils upsert` should require `--pk` more elegantly,9599,simonw,closed,0,,,,,1,2022-01-26T02:20:31Z,2022-01-26T03:20:25Z,2022-01-26T03:19:43Z,OWNER,,"Currently throws an ugly traceback: ``` % echo '[ {""id"": 1, ""name"": ""Lila""}, {""id"": 1, ""name"": ""Lila""} ]' | sqlite-utils upsert data.db chickens - Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1137, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1062, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py"", line 763, in invoke return __callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 1104, in upsert insert_upsert_implementation( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 906, in insert_upsert_implementation db[table].insert_all( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2615, in insert_all raise PrimaryKeyRequired(""upsert() requires a pk"") sqlite_utils.db.PrimaryKeyRequired: upsert() requires a pk ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/390/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1099897648,I_kwDOCGYnMM5Bjxsw,384,Add examples to every `--help`,9599,simonw,closed,0,,,,,0,2022-01-12T05:31:25Z,2022-01-26T03:15:02Z,2022-01-26T03:15:02Z,OWNER,,Everything on https://sqlite-utils.datasette.io/en/stable/cli-reference.html would benefit from an example.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/384/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 471818939,MDU6SXNzdWU0NzE4MTg5Mzk=,48,"Jupyter notebook demo of the library, launchable on Binder",9599,simonw,closed,0,,,,,2,2019-07-23T17:05:05Z,2022-01-26T02:08:46Z,2022-01-26T02:08:39Z,OWNER,,,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/48/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1114544727,I_kwDOCGYnMM5CbppX,389,Plausible analytics for documentation,9599,simonw,closed,0,,,,,2,2022-01-26T01:58:35Z,2022-01-26T02:07:41Z,2022-01-26T02:07:41Z,OWNER,,"```html ``` _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/389/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1111293050,I_kwDOCGYnMM5CPPx6,387,Python library docs should start with a self contained example,9599,simonw,closed,0,,,,,1,2022-01-22T06:23:56Z,2022-01-26T01:37:17Z,2022-01-26T01:35:30Z,OWNER,,You have to read a lot of stuff in a lot of different places to get started with the Python library. Add a getting started introduction to https://sqlite-utils.datasette.io/en/stable/python-api.html,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/387/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1107557831,I_kwDOCGYnMM5CA_3H,386,"Better ""contributing"" documentation",9599,simonw,closed,0,,,,,0,2022-01-19T02:11:48Z,2022-01-19T02:15:21Z,2022-01-19T02:15:21Z,OWNER,,"This page jumps straight into running the tests: https://sqlite-utils.datasette.io/en/latest/contributing.html It should add a little more about expected collaboration styles - opening an issue before filing a pull request - and probably link to https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/386/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1067771698,I_kwDOCGYnMM4_pOcy,348,Command for creating an empty database,9599,simonw,closed,0,,,7558727,3.21,6,2021-11-30T23:24:27Z,2022-01-13T07:06:59Z,2022-01-09T20:33:20Z,OWNER,,"I sometimes find the need to create an empty SQLite database file - for example if I want to enable WAL on it before using it with another script. I currently do that like this: sqlite3 my.db vacuum sqlite-utils enable-wal my.db It would be nice if `sqlite-utils` had a convenience command for doing this.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/348/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1099584685,I_kwDOCGYnMM5BilSt,381,`sqlite-utils rows` options `--limit` and `--offset`,9599,simonw,closed,0,,,,,2,2022-01-11T20:23:12Z,2022-01-11T23:33:37Z,2022-01-11T23:19:36Z,OWNER,,Because I often want to use it just to preview a few rows from the database. Piping through `| head -n 20` works for JSON and CSV (they stream) but not for `--table`.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/381/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1099585611,I_kwDOCGYnMM5BilhL,382,`--where` option for `sqlite-rows`,9599,simonw,closed,0,,,,,1,2022-01-11T20:24:23Z,2022-01-11T23:33:14Z,2022-01-11T23:32:47Z,OWNER,,CLI equivalent of `table.rows_where()` - should accept parameters too. Work on this at the same time as #381.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/382/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1099586786,I_kwDOCGYnMM5Bilzi,383,Add documentation page with the output of `--help`,9599,simonw,closed,0,,,,,4,2022-01-11T20:25:58Z,2022-01-11T22:55:05Z,2022-01-11T21:44:05Z,OWNER,,"Can be maintained using `cog` from #373. Similar in purpose to the API reference page, but this is for the CLI.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/383/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1098574572,I_kwDOCGYnMM5Beurs,380,Release notes for 3.21,9599,simonw,closed,0,,,7558727,3.21,1,2022-01-11T02:12:30Z,2022-01-11T02:34:26Z,2022-01-11T02:34:26Z,OWNER,,For these commits: https://github.com/simonw/sqlite-utils/compare/3.20...129141572f249ea290e2a075437e2ebaad215859,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/380/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097251014,I_kwDOCGYnMM5BZrjG,375,`sqlite-utils bulk` command,9599,simonw,closed,0,,,7558727,3.21,3,2022-01-09T17:12:38Z,2022-01-11T02:12:58Z,2022-01-11T02:10:55Z,OWNER,,"The `.executemany()` method is a very efficient way to execute the same SQL query against a huge list of parameters. `sqlite-utils insert` supports a bunch of ways of loading a list of dictionaries - from CSV, TSV, JSON, newline JSON and more thanks to: - #361 What if you could load a list of dictionaries and provide a SQL query with `:named` parameters that correspond to keys in those dictionaries instead? This would need to be a new command - I thought about adding a `--sql` option to `insert` but that doesn't make sense as that command already requires a table name.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/375/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097477582,PR_kwDOCGYnMM4wtl17,377,`sqlite-utils bulk` command,9599,simonw,closed,0,,,7558727,3.21,3,2022-01-10T05:34:24Z,2022-01-11T02:10:57Z,2022-01-11T02:10:54Z,OWNER,simonw/sqlite-utils/pulls/377,"Refs #375 Still needs: - [x] Refactor `@insert_upsert_options` so that it doesn't duplicate `@import_options` - [x] Tests - [x] Documentation - [x] Try it against a really big file",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/377/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1098544628,I_kwDOCGYnMM5BenX0,379,CLI options for running ANALYZE,9599,simonw,closed,0,,,7558727,3.21,0,2022-01-11T01:09:16Z,2022-01-11T01:38:01Z,2022-01-11T01:36:48Z,OWNER,,"> The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865_ - [x] `sqlite-utils analyze` command - [x] `sqlite-utils create-index --analyze` option (see #365) - [x] `sqlite-utils insert --analyze` option - [x] `sqlite-utils upsert --analyze` option In #378 I also added `.delete_where(..., analyze=True)` but there isn't currently a `sqlite-utils delete-where` CLI command - deletions via CLI are expected to be handled using SQL queries.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/379/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1096563265,I_kwDOCGYnMM5BXDpB,366,Python library methods for calling ANALYZE,9599,simonw,closed,0,,,7558727,3.21,10,2022-01-07T18:28:01Z,2022-01-11T01:09:33Z,2022-01-11T01:09:33Z,OWNER,,"> Relevant documentation: https://www.sqlite.org/lang_analyze.html _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/366/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1098309897,I_kwDOCGYnMM5BduEJ,378,analyze=True parameter for some methods,9599,simonw,closed,0,,,7558727,3.21,0,2022-01-10T19:54:52Z,2022-01-11T01:08:11Z,2022-01-11T01:08:09Z,OWNER,,"This would cause `ANALYZE` to be run against the relevant table at the end of executing the method. > Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are: - [x] `table.create_index` - [x] `table.insert_all` - [x] `table.upsert_all` - [x] `table.delete_where` _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/378/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097041471,PR_kwDOCGYnMM4wsVM6,367,Initial prototype of .analyze() methods,9599,simonw,closed,0,,,7558727,3.21,2,2022-01-08T21:35:12Z,2022-01-10T19:31:08Z,2022-01-10T19:31:08Z,OWNER,simonw/sqlite-utils/pulls/367,Refs #366,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/367/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1, 1097436959,I_kwDOCGYnMM5BaY8f,376,`--nl` mode should ignore blank lines,9599,simonw,closed,0,,,7558727,3.21,0,2022-01-10T04:10:54Z,2022-01-10T19:27:41Z,2022-01-10T04:12:46Z,OWNER,,Spotted this while manually testing #364 - there's no reason `--nl` should crash if you feed it an empty line in between JSON objects.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/376/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097129710,I_kwDOCGYnMM5BZN7u,372,Idea: `suffix` and `stem` file columns,9599,simonw,closed,0,,,7558727,3.21,1,2022-01-09T07:48:53Z,2022-01-10T19:27:34Z,2022-01-09T20:17:00Z,OWNER,,"For https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files Given a file called `dogs.jpg` stem would be `dogs` and ext would be `jpg`. Need to decide what happens for `dogs.and.cats.jpg.gz`.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/372/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097128334,I_kwDOCGYnMM5BZNmO,371,Support mutating row in `--convert` without returning it,9599,simonw,closed,0,,,7558727,3.21,6,2022-01-09T07:38:44Z,2022-01-10T19:27:30Z,2022-01-09T20:06:15Z,OWNER,,"Currently you have to do this: ``` $ sqlite-utils insert dogs.db dogs dogs.json --convert ' row[""is_good""] = 1 return row' ``` Would be neat if this worked too: ``` $ sqlite-utils insert dogs.db dogs dogs.json \ --convert 'row[""is_good""] = 1' ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/371/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097135860,I_kwDOCGYnMM5BZPb0,374,`--fmt` should imply `-t`,9599,simonw,closed,0,,,7558727,3.21,4,2022-01-09T08:23:07Z,2022-01-10T19:27:26Z,2022-01-09T18:07:59Z,OWNER,,Not sure why I didn't implement this.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/374/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097135732,I_kwDOCGYnMM5BZPZ0,373,List `--fmt` options in the docs ,9599,simonw,closed,0,,,7558727,3.21,3,2022-01-09T08:22:11Z,2022-01-10T19:27:24Z,2022-01-09T17:49:00Z,OWNER,,https://sqlite-utils.datasette.io/en/stable/cli.html#table-formatted-output currently cheats and tells the user to run `--help` - can fix this using `cog`. ,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/373/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1097087280,I_kwDOCGYnMM5BZDkw,368,Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`,9599,simonw,closed,0,,,7558727,3.21,3,2022-01-09T02:29:30Z,2022-01-10T19:27:20Z,2022-01-09T02:40:50Z,OWNER,,"> Add this to `sqlite_utils/cli.py`: > > ```python > if __name__ == ""__main__"": > cli() > ``` > Now the tool can be run using `python -m sqlite_utils.cli --help` _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/368/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1095570074,I_kwDOCGYnMM5BTRKa,364,`--batch-size 1` doesn't seem to commit for every item,9599,simonw,closed,0,,,7558727,3.21,16,2022-01-06T18:18:50Z,2022-01-10T19:27:17Z,2022-01-10T05:36:19Z,OWNER,,"I'm trying this, but it doesn't seem to write anything to the database file until I hit `CTRL+C`: ``` heroku logs --app=simonwillisonblog --tail | grep 'measure#nginx.service' | \ sqlite-utils insert /tmp/herokutail.db log - --import re --convert ""$(cat < ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/362/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1094890366,PR_kwDOCGYnMM4wlm3B,361,--lines and --text and --convert and --import,9599,simonw,closed,0,,,,,15,2022-01-06T01:49:44Z,2022-01-06T06:37:03Z,2022-01-06T06:24:54Z,OWNER,simonw/sqlite-utils/pulls/361,"Refs #356 Still TODO: - [x] Get `--lines` working, with tests - [x] Get `--text` working, with tests - [x] Get regular JSON import working with `--convert` with tests - [x] Get `--lines` working with `--convert` with tests - [x] Get `--text` working with `--convert` with tests - [x] Get `--csv` and `--tsv` import working with `--convert` with tests - [x] Get `--nl` working with `--convert` with tests - [x] Documentation for all of the above",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/361/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1077431957,I_kwDOCGYnMM5AOE6V,356,`sqlite-utils insert --convert` option,9599,simonw,closed,0,,,,,11,2021-12-11T07:24:48Z,2022-01-06T06:30:13Z,2022-01-06T06:28:53Z,OWNER,,"Idea come to me while re-reading this: https://simonwillison.net/2021/Aug/6/sqlite-utils-convert/ This is a bit of a hack: ``` cat /tmp/log.txt | \ jq --raw-input '{line: .}' --compact-output | \ sqlite-utils insert /tmp/logs.db log - --nl ``` Would be great if you could pipe lines to `insert` and transform them on the way in. A `--convert python-code` option, modeled after `sqlite-utils convert`, could do this.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/356/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1090798237,I_kwDOCGYnMM5BBEKd,359,Use RETURNING if available to populate last_pk,9599,simonw,open,0,,,,,0,2021-12-29T23:43:23Z,2021-12-29T23:43:23Z,,OWNER,,"Inspired by this: https://news.ycombinator.com/item?id=29729283 > Because SQLite is effectively serializing all the writes for us, we have zero locking in our code. We used to have to lock when inserting new items (to get the LastInsertRowId), but the newer version of SQLite supports the RETURNING keyword, so we don't even have to lock on inserts now.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/359/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 807437089,MDU6SXNzdWU4MDc0MzcwODk=,228,--no-headers option for CSV and TSV,9599,simonw,closed,0,,,,,10,2021-02-12T17:56:51Z,2021-12-26T07:01:31Z,2021-02-14T22:25:17Z,OWNER,,"https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c has a fascinating CSV file that doesn't have a header row - it starts like this: ```csv Computation and measurement of turbulent flow through idealized turbine blade passages,,""Loizou, Panos A."",https://isni.org/isni/0000000136122593,,University of Manchester,https://isni.org/isni/0000000121662407,1989,Thesis (Ph.D.),,Physical Sciences,,,https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.232781, ""Prolactin and growth hormone secretion in normal, hyperprolactinaemic and acromegalic man"",,""Prescott, R. W. G."",https://isni.org/isni/0000000134992122,,University of Newcastle upon Tyne,https://isni.org/isni/0000000104627212,1983,Thesis (Ph.D.),,Biological Sciences,,,https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.232784, ``` It would be useful if `sqlite-utils insert ... --csv` had a mechanism for importing files like this one.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/228/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 706001517,MDU6SXNzdWU3MDYwMDE1MTc=,163,Idea: conversions= could take Python functions,9599,simonw,open,0,,,,,4,2020-09-22T00:37:12Z,2021-12-20T00:56:52Z,,OWNER,,"Right now you use `conversions=` like this: ```python db[""example""].insert({ ""name"": ""The Bigfoot Discovery Museum"" }, conversions={""name"": ""upper(?)""}) ``` How about if you could optionally provide a Python function (or a lambda) like this? ```python db[""example""].insert({ ""name"": ""The Bigfoot Discovery Museum"" }, conversions={""name"": lambda s: s.upper()}) ``` This would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1077243232,I_kwDOCGYnMM5ANW1g,354,Test failure in test_rebuild_fts,9599,simonw,closed,0,,,,,7,2021-12-10T21:27:55Z,2021-12-11T01:08:46Z,2021-12-11T01:08:46Z,OWNER,,"Not sure why this has only just started failing, but I'm getting this: https://github.com/simonw/sqlite-utils/runs/4488687639 ``` E sqlite3.DatabaseError: database disk image is malformed sqlite_utils/db.py:425: DatabaseError _______________________ test_rebuild_fts[searchable_fts] _______________________ fresh_db = > table_to_fix = 'searchable_fts' @pytest.mark.parametrize(""table_to_fix"", [""searchable"", ""searchable_fts""]) def test_rebuild_fts(fresh_db, table_to_fix): table = fresh_db[""searchable""] table.insert(search_records[0]) table.enable_fts([""text"", ""country""]) # Run a search rows = list(table.search(""tanuki"")) assert len(rows) == 1 assert { ""rowid"": 1, ""text"": ""tanuki are running tricksters"", ""country"": ""Japan"", ""not_searchable"": ""foo"", }.items() <= rows[0].items() # Delete from searchable_fts_data fresh_db[""searchable_fts_data""].delete_where() # This should have broken the index with pytest.raises(sqlite3.DatabaseError): list(table.search(""tanuki"")) # Running rebuild_fts() should fix it > fresh_db[table_to_fix].rebuild_fts() ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/354/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1066603133,PR_kwDOCGYnMM4vKAzW,347,Test against pysqlite3 running SQLite 3.37,9599,simonw,open,0,,,,,9,2021-11-29T23:17:57Z,2021-12-11T01:02:19Z,,OWNER,simonw/sqlite-utils/pulls/347,Refs #346 and #344.,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/347/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1077322009,I_kwDOCGYnMM5ANqEZ,355,Allow users to pass a full convert() function definition,9599,simonw,closed,0,,,,,4,2021-12-10T23:59:58Z,2021-12-11T00:51:15Z,2021-12-11T00:49:31Z,OWNER,,"> I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too: > ```sh > sqlite-utils convert my.db mytable col1 ' > def convert(value): > return value.upper() > ' > ``` _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/355/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1072780607,I_kwDOCGYnMM4_8VU_,351,Support `--import xml.etree.ElementTree` in `sqlite-utils convert`,9599,simonw,closed,0,,,,,1,2021-12-07T00:40:29Z,2021-12-11T00:11:25Z,2021-12-11T00:11:25Z,OWNER,,"It's not possible to use a module that requires a nested import, such as `xml.etree.ElementTree`, at the moment. I found and fixed this bug in `git-history`, I should replicate that fix (and accompanying documentation) here: https://github.com/simonw/git-history/issues/39",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/351/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1071531082,I_kwDOCGYnMM4_3kRK,349,A way of creating indexes on newly created tables,9599,simonw,open,0,,,,,3,2021-12-05T18:56:12Z,2021-12-07T01:04:37Z,,OWNER,,"I'm writing code for https://github.com/simonw/git-history/issues/33 that creates a table inside a loop: ```python item_pk = db[item_table].lookup( {""_item_id"": item_id}, item_to_insert, column_order=(""_id"", ""_item_id""), pk=""_id"", ) ``` I need to look things up by `_item_id` on this table, which means I need an index on that column (the table can get very big). But there's no mechanism in SQLite utils to detect if the table was created for the first time and add an index to it. And I don't want to run `CREATE INDEX IF NOT EXISTS` every time through the loop. This should work like the `foreign_keys=` mechanism. ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/349/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1072435124,I_kwDOCGYnMM4_7A-0,350,Optional caching mechanism for table.lookup(),9599,simonw,open,0,,,,,3,2021-12-06T17:54:25Z,2021-12-06T17:56:57Z,,OWNER,,"Inspired by work on `git-history` where I used this pattern: ```python column_name_to_id = {} def column_id(column): if column not in column_name_to_id: id = db[""columns""].lookup( {""namespace"": namespace_id, ""name"": column}, foreign_keys=((""namespace"", ""namespaces"", ""id""),), ) column_name_to_id[column] = id return column_name_to_id[column] ``` If you're going to be doing a large number of `table.lookup(...)` calls and you know that no other script will be modifying the database at the same time you can presumably get a big speedup using a Python in-memory cache - maybe even a LRU one to avoid memory bloat.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/350/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1066563554,I_kwDOCGYnMM4_knfi,346,Way to test SQLite 3.37 (and potentially other versions) in CI,9599,simonw,open,0,,,,,5,2021-11-29T22:21:06Z,2021-11-29T23:12:49Z,,OWNER,,"> Need to figure out a good pattern for testing this in CI too - it will currently skip the new tests if it doesn't have SQLite 3.37 or higher. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076924_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/346/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 1066501534,I_kwDOCGYnMM4_kYWe,345,`table.strict` introspection boolean for identifying STRICT mode tables,9599,simonw,closed,0,,,,,2,2021-11-29T21:05:10Z,2021-11-29T22:45:26Z,2021-11-29T22:44:36Z,OWNER,,"> From the STRICT docs: >> The SQLite parser accepts a comma-separated list of table options after the final close parenthesis in a CREATE TABLE statement. As of this writing (2021-08-23) only two options are recognized: >> >> - STRICT >> - [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) > > So I think I need to read the `CREATE TABLE` statement from the `sqlite_master` table, split on the last `)`, split those tokens on `,` and see if `create` is in there (case insensitive). _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982020757_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/345/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1058196641,I_kwDOCGYnMM4_Esyh,342,Extra options to `lookup()` which get passed to `insert()`,9599,simonw,closed,0,,,,,7,2021-11-19T06:53:03Z,2021-11-19T07:26:54Z,2021-11-19T07:26:54Z,OWNER,,"For https://github.com/simonw/git-history/issues/12 I found myself wanting to pass extra options to `lookup()` to set the column order, primary key etc.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/342/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1025726600,PR_kwDOCGYnMM4tKxHD,330,Test against Python 3.10,9599,simonw,closed,0,,,,,1,2021-10-13T21:50:22Z,2021-11-15T02:59:29Z,2021-10-13T22:25:05Z,OWNER,simonw/sqlite-utils/pulls/330,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/330/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 1005891028,I_kwDOCGYnMM479K3U,329,Rethink approach to [ and ] in column names (currently throws error),9599,simonw,closed,0,,,,,12,2021-09-23T22:14:24Z,2021-11-15T02:57:51Z,2021-11-15T02:57:51Z,OWNER,,"> I think it's best to still keep `[` and `]` out of column names though. Transforming them into `(` and `)` seems reasonable - but should that happen here or in `sqlite-utils`? I think in `sqlite-utils`. _Originally posted by @simonw in https://github.com/simonw/datasette-app/issues/121#issuecomment-926200398_ This is a rethinking of the solution to: - https://github.com/simonw/sqlite-utils/issues/86",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/329/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1053136495,I_kwDOCGYnMM4-xZZv,341,`hash_id: Optional[Any]` should be `hash_id: Optional[str]`,9599,simonw,closed,0,,,,,0,2021-11-15T02:12:39Z,2021-11-15T02:19:31Z,2021-11-15T02:19:31Z,OWNER,,"In a few places: https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L642 https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L751 https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L1049 https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L1230 But it's correct here: https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L2470",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/341/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1053122092,I_kwDOCGYnMM4-xV4s,339,`table.lookup()` option to populate additional columns when creating a record,9599,simonw,closed,0,,,,,4,2021-11-15T01:41:17Z,2021-11-15T02:02:34Z,2021-11-15T02:02:00Z,OWNER,,"> For the commits table I feel like I want a version of `table.lookup()` that can be passed additional columns to populate only if the record does not exist yet. _Originally posted by @simonw in https://github.com/simonw/git-history/issues/12#issuecomment-967455017_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/339/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 1053087862,I_kwDOCGYnMM4-xNh2,338,"dict, list, tuple should all map to TEXT",9599,simonw,closed,0,,,,,0,2021-11-15T00:28:01Z,2021-11-15T00:36:03Z,2021-11-15T00:36:03Z,OWNER,,"> This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947 > > So the `COLUMN_TYPE_MAPPING` should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188 _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/322#issuecomment-968401459_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/338/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 707478649,MDU6SXNzdWU3MDc0Nzg2NDk=,173,Progress bar for sqlite-utils insert,9599,simonw,closed,0,,,,,6,2020-09-23T15:43:56Z,2021-11-01T08:42:24Z,2020-10-27T18:16:04Z,OWNER,,"It would be nice if `sqlite-utils insert` had a progress bar, for when it's churning through huge CSV files.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/173/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 761915790,MDU6SXNzdWU3NjE5MTU3OTA=,206,sqlite-utils should suggest --csv if JSON parsing fails,9599,simonw,closed,0,,,,,4,2020-12-11T05:17:56Z,2021-10-30T15:52:17Z,2021-01-03T18:42:22Z,OWNER,,"``` ~ % gsutil cat gs://ossf-criticality-score/python_top_200.csv | sqlite-utils insert /tmp/crit.db crit - ... File ""/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py"", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File ""/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py"", line 355, in raw_decode raise JSONDecodeError(""Expecting value"", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) ``` A nicer error message here would be one that says the JSON is invalid but suggests that maybe you could try `--csv`.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/206/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 964400482,MDU6SXNzdWU5NjQ0MDA0ODI=,310,`sqlite-utils insert --flatten` option to flatten nested JSON,9599,simonw,closed,0,,,,,3,2021-08-09T21:23:08Z,2021-10-16T13:54:56Z,2021-08-09T21:44:06Z,OWNER,,"I had to do this with a `jq` recipe today: https://til.simonwillison.net/cloudrun/tailing-cloud-run-request-logs ``` cat log.json | jq -c '[leaf_paths as $path | { ""key"": $path | join(""_""), ""value"": getpath($path) }] | from_entries' \ | sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1 ``` That was to turn something like this: ```json { ""httpRequest"": { ""latency"": ""0.112114537s"", ""requestMethod"": ""GET"", ""requestSize"": ""534"", ""status"": 200, }, ""insertId"": ""6111722f000b5b4c4d4071e2"", ""labels"": { ""service"": ""datasette-io"" } } ``` Into this instead: ```json { ""httpRequest_latency"": ""0.112114537s"", ""httpRequest_requestMethod"": ""GET"", ""httpRequest_requestSize"": ""534"", ""httpRequest_status"": 200, ""insertId"": ""6111722f000b5b4c4d4071e2"", ""labels_service"": ""datasette-io"" } ``` I have to do this often enough that I think it should be an option, `--flatten` - so I can do this instead: ``` cat log.json | sqlite-utils insert /tmp/logs.db logs - --flatten ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/310/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 929748885,MDExOlB1bGxSZXF1ZXN0Njc3NTU0OTI5,293,Test against Python 3.10-dev,9599,simonw,closed,0,,,,,3,2021-06-25T01:40:39Z,2021-10-13T21:49:33Z,2021-10-13T21:49:33Z,OWNER,simonw/sqlite-utils/pulls/293,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/293/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 991237645,MDExOlB1bGxSZXF1ZXN0NzI5NzMxNDQx,326,Test against 3.10-dev,9599,simonw,closed,0,,,,,2,2021-09-08T15:01:15Z,2021-10-13T21:49:28Z,2021-10-13T21:49:28Z,OWNER,simonw/sqlite-utils/pulls/326,"This tests against the latest 3.10 RC, https://www.python.org/downloads/release/python-3100rc2/",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/326/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 597671518,MDU6SXNzdWU1OTc2NzE1MTg=,98,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records",9599,simonw,closed,0,,,,,7,2020-04-10T03:19:40Z,2021-09-28T04:38:44Z,2020-04-13T03:29:15Z,OWNER,,,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/98/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 979627285,MDU6SXNzdWU5Nzk2MjcyODU=,323,`table.convert()` method should clean up after itself,9599,simonw,closed,0,,,,,1,2021-08-25T21:15:39Z,2021-08-25T21:25:26Z,2021-08-25T21:25:18Z,OWNER,,"It currently works like this: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/db.py#L2177-L2195 It's registering a function called `convert_value()` and then failing to de-register that function once it has finished. It might even be possible for two queries running against the same connection to clobber each other's `convert_value()` functions, leading to incorrect behaviour. So two fixes: firstly it should register the function with a unique name (maybe add a random suffix). Secondly, it should de-register that function once it has finished.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/323/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 978537855,MDExOlB1bGxSZXF1ZXN0NzE5MTA5NzA5,321,"Ability to insert file contents as text, in addition to blob",9599,simonw,closed,0,,,,,5,2021-08-24T22:37:18Z,2021-08-24T23:31:17Z,2021-08-24T23:31:13Z,OWNER,simonw/sqlite-utils/pulls/321,Refs #319.,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/321/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 976405225,MDU6SXNzdWU5NzY0MDUyMjU=,320,sqlite-utils memory --analyze option,9599,simonw,closed,0,,,,,2,2021-08-22T15:37:10Z,2021-08-22T15:46:56Z,2021-08-22T15:44:29Z,OWNER,,To provide a way of running [analyze-tables](https://sqlite-utils.datasette.io/en/stable/cli.html#analyzing-tables) directly against JSON or CSV data.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/320/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 913135723,MDU6SXNzdWU5MTMxMzU3MjM=,266,"Add some types, enforce with mypy",9599,simonw,closed,0,,,,,3,2021-06-07T06:05:56Z,2021-08-18T22:25:38Z,2021-08-18T22:25:38Z,OWNER,,"A good starting point would be adding type information to the members of these named tuples and the introspection methods that return them: https://github.com/simonw/sqlite-utils/blob/9dff7a38831d471b1dff16d40d89eb5c3b4e84d6/sqlite_utils/db.py#L51-L75",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/266/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 965210966,MDU6SXNzdWU5NjUyMTA5NjY=,314,Type signatures for `.create_table()` and `.create_table_sql()` and `.create()` and `Table.__init__`,9599,simonw,closed,0,,,,,2,2021-08-10T18:03:59Z,2021-08-18T22:25:21Z,2021-08-18T22:25:21Z,OWNER,,"> Adding type signatures to `create_table()` and `.create_table_sql()` is a bit too involved, I'll do that in a separate issue. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/312#issuecomment-896200682_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/314/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 465815372,MDU6SXNzdWU0NjU4MTUzNzI=,37,Experiment with type hints,9599,simonw,closed,0,,,,,6,2019-07-09T14:30:34Z,2021-08-18T21:48:57Z,2021-08-18T21:48:57Z,OWNER,,"Since it's designed to be used in Jupyter or for rapid prototyping in an IDE (and it's still pretty small) `sqlite-utils` feels like a great candidate for me to finally try out Python type hints. https://veekaybee.github.io/2019/07/08/python-type-hints/ is good. It suggests the mypy docs for getting started: https://mypy.readthedocs.io/en/latest/existing_code.html plus this tutorial: https://pymbook.readthedocs.io/en/latest/typehinting.html",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/37/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 931752773,MDU6SXNzdWU5MzE3NTI3NzM=,294,Add a `sqlite-utils memory` example to the README,9599,simonw,closed,0,,,,,0,2021-06-28T16:35:59Z,2021-08-18T21:40:03Z,2021-08-18T21:40:03Z,OWNER,,,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/294/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 974067156,MDU6SXNzdWU5NzQwNjcxNTY=,318,Research: handle gzipped CSV directly,9599,simonw,open,0,,,,,2,2021-08-18T21:23:04Z,2021-08-18T21:25:30Z,,OWNER,,"Would it be worthwhile for the `sqlite-utils` command-line tool to grow features to efficiently directly interact with gzipped CSV data? Maybe add `--gz` options to both `insert` and to the various commands that output query results.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/318/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 972827346,MDU6SXNzdWU5NzI4MjczNDY=,317,Link to a better example on docs index,9599,simonw,closed,0,,,,,1,2021-08-17T15:43:40Z,2021-08-18T18:31:43Z,2021-08-18T18:31:43Z,OWNER,,https://github.com/simonw/sqlite-utils/blob/7a19822ac9ee24be2fbb4c2326a0bf2f3d2d9c4d/docs/index.rst#L39 Is a very old example,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/317/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 970320615,MDU6SXNzdWU5NzAzMjA2MTU=,316,Fix visible backticks on reference page,9599,simonw,closed,0,,,,,1,2021-08-13T11:37:46Z,2021-08-14T05:12:23Z,2021-08-14T05:10:48Z,OWNER,,"https://sqlite-utils.datasette.io/en/latest/reference.html Search for backtick to reveal various minor markup bugs.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/316/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 722816436,MDU6SXNzdWU3MjI4MTY0MzY=,186,.extract() shouldn't extract null values,9599,simonw,open,0,,,,,7,2020-10-16T02:41:08Z,2021-08-12T12:32:14Z,,OWNER,,"This almost works, but it creates a rogue `type` record with a value of None. ``` In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database(memory=True) In [5]: db[""creatures""].insert_all([ {""id"": 1, ""name"": ""Simon"", ""type"": None}, {""id"": 2, ""name"": ""Natalie"", ""type"": None}, {""id"": 3, ""name"": ""Cleo"", ""type"": ""dog""}], pk=""id"") Out[5]:
In [7]: db[""creatures""].extract(""type"") Out[7]:
In [8]: list(db[""creatures""].rows) Out[8]: [{'id': 1, 'name': 'Simon', 'type_id': None}, {'id': 2, 'name': 'Natalie', 'type_id': None}, {'id': 3, 'name': 'Cleo', 'type_id': 2}] In [9]: db[""type""] Out[9]:
In [10]: list(db[""type""].rows) Out[10]: [{'id': 1, 'type': None}, {'id': 2, 'type': 'dog'}] ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/186/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 965102534,MDU6SXNzdWU5NjUxMDI1MzQ=,311,Add reference documentation generated from docstrings,9599,simonw,closed,0,,,,,4,2021-08-10T16:04:00Z,2021-08-11T12:03:50Z,2021-08-11T12:03:50Z,OWNER,,"Using https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html I'm not a big fan of this kind of documentation because it so often comes in place of narrative documentation - but the library has great narrative documentation now, so the reference documentation can link to it in places. This will also encourage me to add good docstrings everywhere, useful for IDEs and suchlike.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/311/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 965143346,MDExOlB1bGxSZXF1ZXN0NzA3NDkwNzg5,312,Add reference page to documentation using Sphinx autodoc,9599,simonw,closed,0,,,,,10,2021-08-10T16:59:17Z,2021-08-10T23:09:32Z,2021-08-10T23:09:28Z,OWNER,simonw/sqlite-utils/pulls/312,Refs #311.,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/312/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 965440017,MDU6SXNzdWU5NjU0NDAwMTc=,315,`.delete_where()` returns `[]` when it should return self,9599,simonw,closed,0,,,,,1,2021-08-10T21:54:55Z,2021-08-10T23:09:29Z,2021-08-10T23:09:29Z,OWNER,,"If the table doesn't exist it should still return `self`, not `[]`: https://github.com/simonw/sqlite-utils/blob/ee469e3122d6f5973ec2584c1580d930daca2e7c/sqlite_utils/db.py#L1676-L1683 Spotted with `mypy` while working on #312.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/315/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 965166058,MDU6SXNzdWU5NjUxNjYwNTg=,313,`.add_foreign_keys()` doesn't reject being called with a View,9599,simonw,closed,0,,,,,0,2021-08-10T17:22:17Z,2021-08-10T17:25:34Z,2021-08-10T17:25:34Z,OWNER,,"Spotted this bug using `mypy` while working on #311 / #312! ``` % mypy sqlite_utils sqlite_utils/db.py:725: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""foreign_keys"" Found 1 error in 1 file (checked 5 source files) ``` Refers to this code: https://github.com/simonw/sqlite-utils/blob/c11ff89894727270d4a9eb554d3a006f5b0d8d9d/sqlite_utils/db.py#L710-L720 It's a bug! We run some checks earlier but none of them ensure that it's a view: https://github.com/simonw/sqlite-utils/blob/c11ff89894727270d4a9eb554d3a006f5b0d8d9d/sqlite_utils/db.py#L697-L709",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/313/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 961008507,MDU6SXNzdWU5NjEwMDg1MDc=,308,Add an interactive tutorial as a Jupyter notebook,9599,simonw,open,0,,,,,2,2021-08-04T20:34:22Z,2021-08-04T21:30:59Z,,OWNER,,Can show people how to open this up in Binder.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/308/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 959305209,MDU6SXNzdWU5NTkzMDUyMDk=,307,codespell to spell check documentation,9599,simonw,closed,0,,,,,0,2021-08-03T16:48:19Z,2021-08-03T16:48:53Z,2021-08-03T16:48:53Z,OWNER,,As seen in https://github.com/simonw/datasette/issues/1417 and https://til.simonwillison.net/python/codespell,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/307/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 841377702,MDU6SXNzdWU4NDEzNzc3MDI=,251,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",9599,simonw,closed,0,,,,,15,2021-03-25T22:36:36Z,2021-08-02T22:39:46Z,2021-08-02T04:47:40Z,OWNER,,"See https://github.com/simonw/sqlite-transform/issues/11 - I built a separate `sqlite-transform` tool a while ago that uses the word ""transform"" to means something entirely different from `sqlite-utils transform` - I'd like to resolve this by merging the two tools.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/251/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 958516743,MDU6SXNzdWU5NTg1MTY3NDM=,306,Configure sphinx.ext.extlinks for issues,9599,simonw,closed,0,,,,,2,2021-08-02T21:19:19Z,2021-08-02T21:39:34Z,2021-08-02T21:29:22Z,OWNER,,As seen in Datasette: https://github.com/simonw/datasette/issues/1227,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/306/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 957383814,MDU6SXNzdWU5NTczODM4MTQ=,301,insert-files should get a --silent option,9599,simonw,closed,0,,,,,0,2021-08-01T04:11:03Z,2021-08-02T19:12:21Z,2021-08-02T19:12:21Z,OWNER,,"The new `sqlite-utils convert` command I'm adding in #251 will have a `--silent` option for turning off the progress bars. The only other command that has progress bars right now is `insert-files` so it should get this option too, for consistency.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/301/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 957731178,MDU6SXNzdWU5NTc3MzExNzg=,304,"`table.convert(..., where=)` and `sqlite-utils convert ... --where=`",9599,simonw,closed,0,,,,,3,2021-08-02T04:27:23Z,2021-08-02T19:00:00Z,2021-08-02T18:58:10Z,OWNER,,"For applying the conversion to a subset of rows selected using the where clause. Should also take optional arguments, as seen in `db[""dogs""].delete_where(""age < ?"", [3])`. Follows #302 and #251. This was originally https://github.com/simonw/sqlite-transform/issues/9",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/304/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 957741820,MDU6SXNzdWU5NTc3NDE4MjA=,305,Python: need a way to execute a count with an extra where clause,9599,simonw,closed,0,,,,,1,2021-08-02T04:52:02Z,2021-08-02T05:08:22Z,2021-08-02T05:08:22Z,OWNER,,I need this for #304. I'll probably add this to the `.execute_count()` method as `where=` and `where_args=`.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/305/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 957536983,MDExOlB1bGxSZXF1ZXN0NzAwOTQ0NjQ0,303,sqlite-utils convert command and db[table].convert(...) method,9599,simonw,closed,0,,,,,1,2021-08-01T16:52:42Z,2021-08-02T04:47:42Z,2021-08-02T04:47:39Z,OWNER,simonw/sqlite-utils/pulls/303,"Refs #251, #302. - [x] Get recipes working - [x] Document recipes - [x] Implement `db[table].convert(...)` method - [x] Add tests for recipes that use the new Python method - [x] Implement `db[table].convert(..., multi=True)` mechanism - [x] Documentation for `db[table].convert(...)` - [x] Refactor `sqlite-utils convert` to use the new method",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/303/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 957529248,MDU6SXNzdWU5NTc1MjkyNDg=,302,Python library version of `sqlite-utils convert`,9599,simonw,closed,0,9599,simonw,,,1,2021-08-01T16:11:02Z,2021-08-02T04:47:40Z,2021-08-02T04:47:40Z,OWNER,,"Spin off from #251. The ability to execute Python functions to convert and split columns should be part of the library too, not just the CLI.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/302/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 952154468,MDU6SXNzdWU5NTIxNTQ0Njg=,299,Ability to see just specific table schemas with `sqlite-utils schema`,9599,simonw,closed,0,,,,,1,2021-07-24T22:00:05Z,2021-07-24T22:12:01Z,2021-07-24T22:08:46Z,OWNER,,"It currently accepts no arguments. Allowing for optional arguments specifying tables would be useful: sqlite-utils schema fixtures.db facetable searchable ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/299/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 927789811,MDU6SXNzdWU5Mjc3ODk4MTE=,292,Add contributing documentation,9599,simonw,closed,0,,,,,0,2021-06-23T02:13:05Z,2021-06-25T17:53:51Z,2021-06-25T17:53:51Z,OWNER,,Like https://docs.datasette.io/en/latest/contributing.html (but simpler) - should cover how to run `black` and `flake8` and `mypy` and how to run the tests.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/292/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 926777310,MDU6SXNzdWU5MjY3NzczMTA=,290,`db.query()` method (renamed `db.execute_returning_dicts()`),9599,simonw,closed,0,,,,,6,2021-06-22T03:03:54Z,2021-06-24T23:17:38Z,2021-06-24T22:54:43Z,OWNER,,"Most of this library deals with lists of Python dictionaries - `.insert_all()`, `.rows`, `.rows_where()`, `.search()`. The `db.execute()` method is the only thing that returns a `sqlite3` cursor. There is a clumsily named `db.execute_returning_dicts(sql)` method but it's not currently mentioned in the documentation. It needs a better name, and needs to be properly documented.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/290/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 927766296,MDU6SXNzdWU5Mjc3NjYyOTY=,291,Adopt flake8,9599,simonw,closed,0,,,,,2,2021-06-23T01:19:37Z,2021-06-24T17:50:27Z,2021-06-24T17:50:27Z,OWNER,,,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/291/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 925487946,MDU6SXNzdWU5MjU0ODc5NDY=,286,Add installation instructions,9599,simonw,closed,0,,,,,1,2021-06-19T23:55:36Z,2021-06-20T18:47:13Z,2021-06-20T18:47:13Z,OWNER,,"`pip install sqlite-utils`, `pipx install sqlite-utils` and `brew install sqlite-utils`",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/286/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 925544070,MDU6SXNzdWU5MjU1NDQwNzA=,287,Update rowid examples in the docs,9599,simonw,closed,0,,,,,0,2021-06-20T08:03:00Z,2021-06-20T18:26:21Z,2021-06-20T18:26:21Z,OWNER,,Changed in #284 - a couple of examples need updating on https://github.com/simonw/sqlite-utils/blob/3.10/docs/cli.rst.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/287/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 925545468,MDU6SXNzdWU5MjU1NDU0Njg=,288,sqlite-utils memory blah.json --schema,9599,simonw,closed,0,,,,,0,2021-06-20T08:10:40Z,2021-06-20T18:26:21Z,2021-06-20T18:26:21Z,OWNER,,Like `--dump` but only outputs the schema - useful for understanding what you are about to run queries against.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/288/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 921878733,MDU6SXNzdWU5MjE4Nzg3MzM=,272,"Idea: import CSV to memory, run SQL, export in a single command",9599,simonw,closed,0,,,,,22,2021-06-15T23:02:48Z,2021-06-19T23:36:48Z,2021-06-18T15:05:03Z,OWNER,,"I quite often load a CSV file into a SQLite DB, then do stuff with it (like export results back out again as a new CSV) without any intention of keeping the CSV file around afterwards. What if `sqlite-utils` could do this for me? Something like this: sqlite-utils --csv blah.csv --csv baz.csv ""select * from blah join baz ..."" ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/272/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 925320167,MDU6SXNzdWU5MjUzMjAxNjc=,284,.transform(types=) turns rowid into a concrete column,9599,simonw,closed,0,,,,,5,2021-06-19T05:25:27Z,2021-06-19T15:28:30Z,2021-06-19T15:28:30Z,OWNER,,"Noticed this in the tests for `sqlite-utils memory` in #282 - is it possible to fix this? https://github.com/simonw/sqlite-utils/commit/ec5174ed40fa283cb06f25ee0c0136297ec313ae",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/284/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 925410305,MDU6SXNzdWU5MjU0MTAzMDU=,285,Introspection property for telling if a table is a rowid table,9599,simonw,closed,0,,,,,7,2021-06-19T14:56:16Z,2021-06-19T15:12:33Z,2021-06-19T15:12:33Z,OWNER,,_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785_,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/285/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 925319214,MDU6SXNzdWU5MjUzMTkyMTQ=,283,memory: Shouldn't detect types for JSON,9599,simonw,closed,0,,,,,1,2021-06-19T05:17:35Z,2021-06-19T14:52:48Z,2021-06-19T14:52:48Z,OWNER,,"https://github.com/simonw/sqlite-utils/blob/ec5174ed40fa283cb06f25ee0c0136297ec313ae/sqlite_utils/cli.py#L1244-L1251 This runs against JSON as well as CSV/TSV - which isn't necessary and In fact throws errors if there is any nested data.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/283/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 925305186,MDU6SXNzdWU5MjUzMDUxODY=,282,Automatic type detection for CSV data,9599,simonw,closed,0,,,,,4,2021-06-19T03:33:21Z,2021-06-19T04:42:03Z,2021-06-19T04:38:00Z,OWNER,,"I've touched on this before in #179 - but now that I've added `sqlite-utils memory` this is much more important - because unlike with `sqlite-utils insert` the in-memory command doesn't give you the opportunity to fix any types you imported from CSV, so queries like `select * from stdin where age > 3` are never going to work correctly against these temporary in-memory tables. Teaching `sqlite-utils insert` to detect types for columns in a CSV file would be a backwards-compatibility breaking change. Teaching `sqlite-utils memory` that trick would not be, since it hasn't been included in a release yet. It's a little inconsistent, but I'm going to have `sqlite-utils memory` default to detecting types while `sqlite-utils insert` does not. In each case this can be controlled by a new command-line option: cat file.csv | sqlite-utils memory - --no-detect-types To opt-in for `sqlite-utils insert`: cat file.csv | sqlite-utils insert blah.db blah - --detect-types I'll have short options for these too: `-n` for `--no-detect-types` and `-d` for `--detect-types`.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/282/reactions"", ""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,completed 709577625,MDU6SXNzdWU3MDk1Nzc2MjU=,179,sqlite-utils transform/insert --detect-types,9599,simonw,closed,0,,,,,4,2020-09-26T17:28:55Z,2021-06-19T03:36:16Z,2021-06-19T03:36:05Z,OWNER,,"Idea from https://github.com/simonw/datasette-edit-tables/issues/13 - provide Python utility methods and accompanying CLI options for detecting the likely types of TEXT columns. So if you have a text column that actually contained exclusively integer string values, it can let you know and let you run transform against it.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/179/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 924990677,MDU6SXNzdWU5MjQ5OTA2Nzc=,279,sqlite-utils memory should handle TSV and JSON in addition to CSV,9599,simonw,closed,0,,,,,7,2021-06-18T15:02:54Z,2021-06-19T03:11:59Z,2021-06-19T03:11:59Z,OWNER,,"- Use sniff to detect CSV or TSV (if `:tsv` or `:csv` was not specified) and delimiters Follow-on from #272",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/279/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 924992318,MDU6SXNzdWU5MjQ5OTIzMTg=,281,Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory,9599,simonw,closed,0,,,,,1,2021-06-18T15:04:53Z,2021-06-19T03:11:59Z,2021-06-19T03:11:59Z,OWNER,,"- Implement `filename.json:json` and `-:nl` and suchlike options for specifying the format rather than guessing it - see https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944 Follows #272",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/281/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 924991194,MDU6SXNzdWU5MjQ5OTExOTQ=,280,Add --encoding option to sqlite-utils memory,9599,simonw,closed,0,,,,,0,2021-06-18T15:03:32Z,2021-06-18T15:29:46Z,2021-06-18T15:29:46Z,OWNER,,Follow-on from #272 - this will work like `--encoding` on `sqlite-utils insert` and will affect all CSV files processed by `sqlite-utils memory`.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/280/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 922099793,MDExOlB1bGxSZXF1ZXN0NjcxMDE0NzUx,273,sqlite-utils memory command for directly querying CSV/JSON data,9599,simonw,closed,0,,,,,8,2021-06-16T05:04:58Z,2021-06-18T15:01:17Z,2021-06-18T15:00:52Z,OWNER,simonw/sqlite-utils/pulls/273,"Refs #272. Initial implementation only does CSV data, still needs: - [x] Implement `--save` - [x] Add `--dump` to the documentation - [x] Add `--attach` example to the documentation - [x] Replace `:memory:` in documentation",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/273/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 922955697,MDU6SXNzdWU5MjI5NTU2OTc=,275,Enable code coverage,9599,simonw,closed,0,,,,,1,2021-06-16T18:33:49Z,2021-06-17T00:12:12Z,2021-06-17T00:12:12Z,OWNER,,"https://app.codecov.io/gh/simonw/sqlite-utils Same mechanism as Datasette. Need to copy across the token from that page and add an equivalent of this workflow: https://github.com/simonw/datasette/blob/main/.github/workflows/test-coverage.yml",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/275/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 922832113,MDU6SXNzdWU5MjI4MzIxMTM=,274,sqlite-utils dump my.db command,9599,simonw,closed,0,,,,,0,2021-06-16T16:30:14Z,2021-06-16T23:51:54Z,2021-06-16T23:51:54Z,OWNER,,"Inspired by the `--dump` mechanism I added to `sqlite-utils memory` here: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937 > Can use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump > > Maybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/274/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 919702451,MDU6SXNzdWU5MTk3MDI0NTE=,271,table.upsert_all() fails if input has a single column that should be a primary key,9599,simonw,closed,0,,,,,1,2021-06-13T02:50:27Z,2021-06-13T02:57:29Z,2021-06-13T02:57:29Z,OWNER,,"This works: ```pycon >>> db['foo'].insert_all([{""name"": ""hello""}], pk=""name"")
``` But this fails: ``` >>> db['foo3'].upsert_all([{""name"": ""hello""}], pk=""name"") Traceback (most recent call last): File """", line 1, in File ""/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1837, in upsert_all return self.insert_all( File ""/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1778, in insert_all self.insert_chunk( File ""/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py"", line 1588, in insert_chunk result = self.db.execute(query, params) File ""/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py"", line 213, in execute return self.conn.execute(sql, parameters) sqlite3.OperationalError: near ""WHERE"": syntax error ``` With the debugger: ``` >>> import pdb; pdb.pm() > /Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py(213)execute() -> return self.conn.execute(sql, parameters) (Pdb) print(sql, parameters) UPDATE [foo3] SET WHERE [name] = ? ['hello'] ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/271/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 919181559,MDU6SXNzdWU5MTkxODE1NTk=,268,db.schema property and sqlite-utils schema command,9599,simonw,closed,0,,,,,4,2021-06-11T20:25:47Z,2021-06-11T20:51:56Z,2021-06-11T20:51:56Z,OWNER,,"`table.schema` returns the schema for a table. `db.schema` should return the schema for the whole databes. Can do this using `select sql from sqlite_master where sql is not null`: https://latest.datasette.io/fixtures?sql=select+sql+from+sqlite_master+where+sql+is+not+null",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/268/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 906356331,MDU6SXNzdWU5MDYzNTYzMzE=,263,`sqlite-utils indexes` command,9599,simonw,closed,0,,,,,6,2021-05-29T04:52:34Z,2021-06-03T04:34:38Z,2021-06-03T04:34:38Z,OWNER,,"While working on #260 I realized there's no command to show indexes in a database, even though there is one for showing tables and one for triggers. I should implement #261 first.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/263/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 906345899,MDU6SXNzdWU5MDYzNDU4OTk=,261,`table.xindexes` using `PRAGMA index_xinfo(table)`,9599,simonw,closed,0,,,,,5,2021-05-29T04:23:48Z,2021-06-03T03:54:14Z,2021-06-03T03:51:32Z,OWNER,,"> `PRAGMA index_xinfo(table)` DOES return that data: > ``` > (Pdb) [c[0] for c in fresh_db.execute(""PRAGMA > index_xinfo('idx_dogs_age_name')"").description] > ['seqno', 'cid', 'name', 'desc', 'coll', 'key'] > (Pdb) fresh_db.execute(""PRAGMA index_xinfo('idx_dogs_age_name')"").fetchall() > [(0, 2, 'age', 1, 'BINARY', 1), (1, 0, 'name', 0, 'BINARY', 1), (2, -1, None, 0, 'BINARY', 0)] > ``` > See https://sqlite.org/pragma.html#pragma_index_xinfo > > Example output: https://covid-19.datasettes.com/covid?sql=select+*+from+pragma_index_xinfo%28%27idx_ny_times_us_counties_date%27%29 _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766552_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/261/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 838148087,MDU6SXNzdWU4MzgxNDgwODc=,250,Handle byte order marks (BOMs) in CSV files,9599,simonw,closed,0,,,,,3,2021-03-22T22:13:18Z,2021-05-29T05:34:21Z,2021-05-29T05:34:21Z,OWNER,,I often find `sqlite-utils insert ... --csv` creates a first column with a weird character at the start of it - which it turns out is the UTF-8 BOM. Fix that.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/250/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 906355849,MDExOlB1bGxSZXF1ZXN0NjU3MzczNzI2,262,Ability to add descending order indexes,9599,simonw,closed,0,,,,,0,2021-05-29T04:51:04Z,2021-05-29T05:01:42Z,2021-05-29T05:01:39Z,OWNER,simonw/sqlite-utils/pulls/262,Refs #260,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/262/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 906330187,MDU6SXNzdWU5MDYzMzAxODc=,260,Support creating descending order indexes,9599,simonw,closed,0,,,,,12,2021-05-29T03:42:59Z,2021-05-29T05:01:39Z,2021-05-29T05:01:39Z,OWNER,,"SQLite lets you create indexes in reverse order, which can have a surprisingly big impact on performance, see https://github.com/simonw/covid-19-datasette/issues/27 I tried doing this using `sqlite-utils` like so, but it's didn't work: ```python db[""ny_times_us_counties""].create_index([""date desc""]) ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/260/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 858501079,MDU6SXNzdWU4NTg1MDEwNzk=,255,transform --help should tell you the available types,9599,simonw,closed,0,,,,,0,2021-04-15T05:24:48Z,2021-05-29T03:55:52Z,2021-05-29T03:55:52Z,OWNER,,"``` Usage: sqlite-utils transform [OPTIONS] PATH TABLE Transform a table beyond the capabilities of ALTER TABLE Options: --type ... Change column type to X ``` This should specify that the possible types are 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/255/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 894948100,MDU6SXNzdWU4OTQ5NDgxMDA=,259,Suggest the --alter option if a new column cannot be added,9599,simonw,closed,0,,,,,1,2021-05-19T03:17:38Z,2021-05-19T03:27:33Z,2021-05-19T03:26:26Z,OWNER,,Refs #256.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/259/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 816523763,MDU6SXNzdWU4MTY1MjM3NjM=,238,.add_foreign_key() corrupts database if column contains a space,9599,simonw,closed,0,,,,,1,2021-02-25T15:07:20Z,2021-02-25T16:54:02Z,2021-02-25T16:54:02Z,OWNER,,"I ran this: db[""Reports""].add_foreign_key(""Reported by ID"", ""Reporters"", ""id"") And got this: ``` ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in add_foreign_keys(self, foreign_keys) 616 # Have to VACUUM outside the transaction to ensure .foreign_keys property 617 # can see the newly created foreign key. --> 618 self.vacuum() 619 620 def index_foreign_keys(self): ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in vacuum(self) 629 630 def vacuum(self): --> 631 self.execute(""VACUUM;"") 632 633 ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in execute(self, sql, parameters) 234 return self.conn.execute(sql, parameters) 235 else: --> 236 return self.conn.execute(sql) 237 238 def executescript(self, sql): DatabaseError: database disk image is malformed ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/238/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 816560819,MDU6SXNzdWU4MTY1NjA4MTk=,240,table.pks_and_rows_where() method returning primary keys along with the rows,9599,simonw,closed,0,,,,,7,2021-02-25T15:49:28Z,2021-02-25T16:39:23Z,2021-02-25T16:28:23Z,OWNER,,"*Original title: Easier way to update a row returned from .rows* Here's a surprisingly hard problem I ran into while trying to implement #239 - given a row returned by `db[table].rows` how can you update that row? The problem is that the `db[table].update(...)` method requires a primary key. But if you have a row from the `db[table].rows` iterator it might not even contain the primary key - provided the table is a `rowid` table. Instead, currently, you need to introspect the table and, if `rowid` is a primary key, explicitly include that in the `select=` argument to `table.rows_where(...)` - otherwise it will not be returned. A utility mechanism to make this easier would be very welcome.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/240/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 816601354,MDExOlB1bGxSZXF1ZXN0NTgwMjM1NDI3,241,Extract expand - work in progress,9599,simonw,open,0,,,,,0,2021-02-25T16:36:38Z,2021-02-25T16:36:38Z,,OWNER,simonw/sqlite-utils/pulls/241,Refs #239. Still needs documentation and CLI implementation.,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/241/reactions"", ""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1, 811680502,MDU6SXNzdWU4MTE2ODA1MDI=,236,--attach command line option for attaching extra databases,9599,simonw,closed,0,,,,,1,2021-02-19T04:38:30Z,2021-02-19T05:10:41Z,2021-02-19T05:08:43Z,OWNER,,"This will enable cross-database joins, as seen in https://github.com/simonw/datasette/issues/283 Also refs #113",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/236/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 621286870,MDU6SXNzdWU2MjEyODY4NzA=,113,Syntactic sugar for ATTACH DATABASE,9599,simonw,closed,0,,,,,2,2020-05-19T21:10:00Z,2021-02-19T05:09:12Z,2021-02-19T04:56:36Z,OWNER,,"https://www.sqlite.org/lang_attach.html Maybe something like this: ```python db.attach(""other_db"", ""other_db.db"") ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/113/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 808008305,MDU6SXNzdWU4MDgwMDgzMDU=,230,--sniff option for sniffing delimiters,9599,simonw,closed,0,,,,,8,2021-02-14T17:43:54Z,2021-02-14T21:15:33Z,2021-02-14T19:24:32Z,OWNER,,"> I just spotted that `csv.Sniffer` in the Python standard library has a `.has_header(sample)` method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/230/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 808046597,MDU6SXNzdWU4MDgwNDY1OTc=,234,.insert_all() fails if subsequent chunks contain additional columns,9599,simonw,closed,0,,,,,1,2021-02-14T21:01:51Z,2021-02-14T21:03:40Z,2021-02-14T21:03:40Z,OWNER,,Reported by @nieuwenhoven in #225 along with a proposed fix.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/234/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 808036774,MDU6SXNzdWU4MDgwMzY3NzQ=,232,Run tests against Windows in GitHub Actions,9599,simonw,closed,0,,,,,0,2021-02-14T20:09:45Z,2021-02-14T20:39:55Z,2021-02-14T20:39:55Z,OWNER,,"> I'm going to try and get the test suite to run in Windows on GitHub Actions. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/225#issuecomment-778834504_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/232/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 808037010,MDExOlB1bGxSZXF1ZXN0NTczMTQ3MTY4,233,"Run tests against Ubuntu, macOS and Windows",9599,simonw,closed,0,,,,,0,2021-02-14T20:11:02Z,2021-02-14T20:39:54Z,2021-02-14T20:39:54Z,OWNER,simonw/sqlite-utils/pulls/233,Refs #232,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/233/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 808028757,MDU6SXNzdWU4MDgwMjg3NTc=,231,"limit=X, offset=Y parameters for more Python methods",9599,simonw,closed,0,,,,,2,2021-02-14T19:31:23Z,2021-02-14T20:03:08Z,2021-02-14T20:03:08Z,OWNER,,"> I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion! _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/224#issuecomment-778828495_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/231/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 802583450,MDU6SXNzdWU4MDI1ODM0NTA=,226,3.4 release is broken - includes a rogue line,9599,simonw,closed,0,,,,,0,2021-02-06T02:08:01Z,2021-02-06T02:10:26Z,2021-02-06T02:10:26Z,OWNER,,"I started seeing weird errors, caused by this line: https://github.com/simonw/sqlite-utils/blob/f8010ca78fed8c5fca6cde19658ec09fdd468420/sqlite_utils/cli.py#L1-L3 That was added by accident in 1b666f9315d4ea6bb332b2e75e48480c26100199 I'm surprised the tests didn't catch this!",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/226/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 788527932,MDU6SXNzdWU3ODg1Mjc5MzI=,223,--delimiter option for CSV import,9599,simonw,closed,0,,,,,2,2021-01-18T20:25:03Z,2021-02-06T01:39:47Z,2021-02-06T01:34:54Z,OWNER,,"https://bruxellesdata.opendatasoft.com/explore/dataset/dog-toilets/export/?location=12,50.85802,4.38054 says: > CSV uses semicolon (;) as a separator. Would be useful to be able to do this: sqlite-utils insert places.db places places.csv --delimiter ';' `--delimiter` could imply `--csv`",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/223/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 787900412,MDU6SXNzdWU3ODc5MDA0MTI=,222,.m2m() should accept alter=True parameter,9599,simonw,closed,0,,,,,0,2021-01-18T04:15:43Z,2021-01-18T04:26:10Z,2021-01-18T04:26:10Z,OWNER,,Needed by https://github.com/dogsheep/swarm-to-sqlite/issues/11,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/222/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 783910901,MDU6SXNzdWU3ODM5MTA5MDE=,221,.add_missing_columns() does not take case insensitivity into account,9599,simonw,closed,0,,,,,0,2021-01-12T05:01:00Z,2021-01-12T23:17:33Z,2021-01-12T23:17:33Z,OWNER,,SQLite columns are case insensitive - but the `.add_missing_columns()` method doesn't know that. This means that it can crash if it identifies a column that is a case-insensitive duplicate of an existing column. https://github.com/simonw/sqlite-utils/blob/4cc82fd0bccc9d2eeb3510beb4e691d7da099f84/sqlite_utils/db.py#L1974-L1980,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/221/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777707544,MDU6SXNzdWU3Nzc3MDc1NDQ=,219,reset_counts() method and command,9599,simonw,closed,0,,,,,4,2021-01-03T20:08:28Z,2021-01-03T20:59:37Z,2021-01-03T20:59:37Z,OWNER,,"> Thought: maybe there should be a `.reset_counts()` method too, for if the table gets out of date with the triggers. > > One way that could happen is if a table is dropped and recreated - the counts in the `_counts` table would likely no longer match the number of rows in that table. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545757_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/219/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777535402,MDU6SXNzdWU3Nzc1MzU0MDI=,215,Use _counts to speed up counts,9599,simonw,closed,0,,,,,9,2021-01-02T22:30:17Z,2021-01-03T20:19:40Z,2021-01-03T20:19:40Z,OWNER,,"Utility mechanism for taking advantage of the new `_counts` table from #212 would be nice. These can trigger automatically if the `_counts` table exists, but since `sqlite-utils` needs to work against any existing database there should be a way of opting out of this optimization.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/215/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 738514367,MDU6SXNzdWU3Mzg1MTQzNjc=,202,sqlite-utils insert -f colname - for configuring full-text search,9599,simonw,closed,0,,,,,2,2020-11-08T17:30:09Z,2021-01-03T05:00:36Z,2021-01-03T05:00:27Z,OWNER,,"A mechanism for specifying columns that should be configured for full-text search as part of the initial data import: sqlite-utils insert mydb.db articles articles.csv --csv -f title -f body",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/202/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777543336,MDU6SXNzdWU3Nzc1NDMzMzY=,217,Rename .escape() to .quote(),9599,simonw,closed,0,,,,,2,2021-01-02T23:40:52Z,2021-01-03T04:27:38Z,2021-01-03T04:15:23Z,OWNER,,"`.quote()` is a better name because it reflects that the method adds quotes around the value. This method has never been documented so I'm going to rename it without a major version bump. ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/217/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777540352,MDU6SXNzdWU3Nzc1NDAzNTI=,216,database.triggers_dict introspection property,9599,simonw,closed,0,,,,,1,2021-01-02T23:13:00Z,2021-01-03T04:27:14Z,2021-01-03T04:25:36Z,OWNER,,Following #211,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/216/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777530107,MDU6SXNzdWU3Nzc1MzAxMDc=,214,sqlite-utils enable-counts command,9599,simonw,closed,0,,,,,0,2021-01-02T21:45:48Z,2021-01-03T04:26:44Z,2021-01-03T04:26:44Z,OWNER,,"The CLI version of #212 and #213. # Enable counts for all tables: sqlite-utils enable-counts data.db # Enable counts for specific tables: sqlite-utils enable-counts data.db table1 table2",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/214/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777560474,MDU6SXNzdWU3Nzc1NjA0NzQ=,218,"""sqlite-utils triggers"" command",9599,simonw,closed,0,,,,,1,2021-01-03T02:34:50Z,2021-01-03T03:49:51Z,2021-01-03T03:03:35Z,OWNER,,"A command to list the triggers in the database. sqlite-utils triggers my.db Can optionally take one or more tables: sqlite-utils triggers my.db table1 table2",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/218/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777529979,MDU6SXNzdWU3Nzc1Mjk5Nzk=,213,db.enable_counts() method,9599,simonw,closed,0,,,,,2,2021-01-02T21:44:55Z,2021-01-02T22:04:02Z,2021-01-02T22:04:02Z,OWNER,,"Following #212 it would be useful if there was a utility method for enabling counts for ALL tables in a database: ```python db.enable_counts() ``` Open question: should this setup triggers for virtual tables such as FTS tables? Could that break things? ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/213/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777392020,MDU6SXNzdWU3NzczOTIwMjA=,212,Mechanism for maintaining cache of table counts using triggers,9599,simonw,closed,0,,,,,1,2021-01-02T02:58:53Z,2021-01-02T21:40:27Z,2021-01-02T21:40:27Z,OWNER,,"Counting all of the rows in a large table is expensive - this is one of the main causes of performance problems in Datasette when running against large databases. Carefully constructed SQL triggers could be used to maintain accurate cached counts for a table, by incrementing and decrementing a counter every time a row is inserted or deleted. `sqlite-utils` already has a mechanism for creating triggers for FTS - the `table.enable_fts(..., create_triggers=True)` method. How about a similar mechanism for setting up triggers to maintain a count of table rows?",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/212/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 777386465,MDU6SXNzdWU3NzczODY0NjU=,211,table.triggers_dict introspection property,9599,simonw,closed,0,,,,,0,2021-01-02T02:04:00Z,2021-01-02T02:10:10Z,2021-01-02T02:10:10Z,OWNER,,"`table.triggers` currently returns a list of `Trigger` values. A `table.triggers_dict` property could behave like `columns_dict`, returning a dictionary mapping trigger names to their SQL definitions for that table.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/211/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 763283616,MDU6SXNzdWU3NjMyODM2MTY=,207,sqlite-utils analyze-tables command,9599,simonw,closed,0,,,,,4,2020-12-12T04:33:12Z,2020-12-13T07:25:23Z,2020-12-13T07:20:13Z,OWNER,,"A command which analyzes a table (potentially taking quite a while if the table is large) and outputs information for each column - things like: - How many unique values does this column have? - How many null rows? - How many blank rows? (defined as empty string) - What are the 10 most common values? - What are the 10 least common values? The command can output this information to the terminal, but it should also provide an option for writing the information to a database table so it can be explored later.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/207/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 763320133,MDExOlB1bGxSZXF1ZXN0NTM3NzkxNjc1,208,sqlite-utils analyze-tables command and table.analyze_column() method,9599,simonw,closed,0,,,,,6,2020-12-12T05:27:49Z,2020-12-13T07:20:16Z,2020-12-13T07:20:12Z,OWNER,simonw/sqlite-utils/pulls/208,"Refs #207 - [x] Improve design of CLI output - [x] Truncate long values in least/most common - [x] Add a `-c` column selection option - [x] Tests - [x] Documentation",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/208/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 706167456,MDU6SXNzdWU3MDYxNjc0NTY=,168,Automate (as much as possible) updates published to Homebrew,9599,simonw,closed,0,,,,,2,2020-09-22T08:08:37Z,2020-11-09T07:43:30Z,2020-11-09T07:43:30Z,OWNER,,I'd like to get new `sqlite-utils` (and Datasette) releases submitted to Homebrew as painlessly as possible.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/168/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 735650864,MDU6SXNzdWU3MzU2NTA4NjQ=,194,3.0 release with some minor breaking changes,9599,simonw,closed,0,,,6079500,3.0,3,2020-11-03T21:36:31Z,2020-11-08T17:19:35Z,2020-11-08T17:19:34Z,OWNER,,"While working on search (#192) I've spotted a few small changes I would like to make that would break backwards compatibility in minor ways, hence requiring a 3.x release. `db[table].search()` - I would like this to default to sorting by rank Also I'd like to free up the `-c` and `-f` options for other purposes from the standard output formats here: https://github.com/simonw/sqlite-utils/blob/43eae8b193d362f2b292df73e087ed6f10838144/sqlite_utils/cli.py#L48-L58 I'd like `-f` to be used to indicate a full-text search column during an insert and `-c` to indicate a column (so you can specify which columns you want to output).",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/194/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 737153927,MDU6SXNzdWU3MzcxNTM5Mjc=,197,Rethink how table.search() method works,9599,simonw,closed,0,,,6079500,3.0,5,2020-11-05T18:04:34Z,2020-11-08T17:07:37Z,2020-11-08T17:07:37Z,OWNER,,"I need to improve this method to help build `sqlite-utils search` in #192 (PR is #195). The challenge is deciding how it should handle sorting by relevance - especially since that is easy in FTS5 but not at all easy in FTS4. > Latest test failure: > ``` > 114 -> assert [(""racoons are biting trash pandas"", ""USA"", ""bar"")] == table.search( > 115 ""bite"", order=""rowid"" > 116 ) > 117 > 118 > 119 def test_optimize_fts(fresh_db): > (Pdb) table.search(""bite"") > [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)] > ``` > The problem here is that the `table.search()` method now behaves differently for FTS4 v.s. FTS5 tables. > > With FTS4 you get back just the table columns. > > With FTS5 you also get back the `rowid` as the first column and the `rank` score as the last column. > > This is weird. It also makes me question whether having `.search()` return a list of tuples is the right API design. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/197/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 735532751,MDU6SXNzdWU3MzU1MzI3NTE=,192,sqlite-utils search command,9599,simonw,closed,0,,,6079500,3.0,9,2020-11-03T18:07:59Z,2020-11-08T17:07:01Z,2020-11-08T17:07:01Z,OWNER,,A command that knows how to run a search against a FTS enabled table and return results ranked by relevance.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/192/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 738128913,MDU6SXNzdWU3MzgxMjg5MTM=,201,.search(columns=) and sqlite-utils search -c ... bug,9599,simonw,closed,0,,,6079500,3.0,1,2020-11-07T01:27:26Z,2020-11-08T16:54:15Z,2020-11-08T16:54:15Z,OWNER,,"Both `table.search(columns=)` and the `sqlite-utils search -c` option do not work as expected - they always return both the `rowid` and the `rank` columns even if those have not been requested. This should be fixed before the 3.0 non-alpha release.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/201/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 738115165,MDU6SXNzdWU3MzgxMTUxNjU=,200,sqlite-utils rows -c option,9599,simonw,closed,0,,,6079500,3.0,1,2020-11-07T00:22:12Z,2020-11-07T00:28:48Z,2020-11-07T00:28:47Z,OWNER,,To let you specify the exact columns you want. Based on the `-c` option to `sqlite-utils search` in #192.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/200/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 735648209,MDU6SXNzdWU3MzU2NDgyMDk=,193,--tsv output format option,9599,simonw,closed,0,,,6079500,3.0,0,2020-11-03T21:31:18Z,2020-11-07T00:09:52Z,2020-11-07T00:09:52Z,OWNER,,"We already support `--csv` for output, and the `insert` command accepts `--tsv`. The output format options should accept `--tsv` too.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/193/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 735663855,MDExOlB1bGxSZXF1ZXN0NTE1MDE0ODgz,195,table.search() improvements plus sqlite-utils search command,9599,simonw,closed,0,,,,,3,2020-11-03T22:02:08Z,2020-11-06T18:30:49Z,2020-11-06T18:30:42Z,OWNER,simonw/sqlite-utils/pulls/195,Refs #192. Still needs tests.,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/195/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 737476423,MDU6SXNzdWU3Mzc0NzY0MjM=,198,Support order by relevance against FTS4,9599,simonw,closed,0,,,,,6,2020-11-06T05:36:31Z,2020-11-06T18:30:44Z,2020-11-06T18:30:44Z,OWNER,,"For #192 and #197 I've decided I want to be able to order by relevance in FTS4 as well as FTS5. This means I need to port over my work on bm25() from https://github.com/simonw/sqlite-fts4 (since I don't want to add a full dependency).",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/198/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 737855731,MDU6SXNzdWU3Mzc4NTU3MzE=,199,"@db.register_function(..., replace=False) to avoid double-registering custom functions",9599,simonw,closed,0,,,,,1,2020-11-06T15:39:21Z,2020-11-06T18:30:44Z,2020-11-06T18:30:44Z,OWNER,,"I'd like a mechanism to optionally avoid registering a custom function if it has already been registered. SQLite doesn't seem to offer a way to introspect registered custom functions so I'll need to track what has already been registered in `sqlite-utils` instead. > Should I register the custom `rank_bm25` SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/199/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 736520310,MDU6SXNzdWU3MzY1MjAzMTA=,196,Introspect if table is FTS4 or FTS5,9599,simonw,closed,0,,,,,19,2020-11-05T00:45:50Z,2020-11-05T03:54:07Z,2020-11-05T03:54:07Z,OWNER,,"> I want `.search()` to work against both FTS5 and FTS4 tables - but sort by rank should only work for FTS5. > > This means I need to be able to introspect and tell if a table is FTS4 or FTS5. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/192#issuecomment-722054264_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/196/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 731740458,MDU6SXNzdWU3MzE3NDA0NTg=,191,Idea: @db.register_function(deterministic=True),9599,simonw,closed,0,,,,,2,2020-10-28T19:45:18Z,2020-10-28T21:31:06Z,2020-10-28T21:31:06Z,OWNER,,"Python 3.8 added a `deterministic` parameter to `db.create_function()`: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function `sqlite-utils` could expose this in the decorator, only actually applying it if the Python version supports it (using feature detection) - since nothing will break if it's not applied. https://sqlite-utils.readthedocs.io/en/stable/python-api.html#registering-custom-sql-functions ```python db = Database(memory=True) @db.register_function(deterministic=True) def reverse_string(s): return """".join(reversed(list(s))) ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/191/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 730693696,MDExOlB1bGxSZXF1ZXN0NTEwOTU2MTM0,190,Progress bar for sqlite-utils insert command,9599,simonw,closed,0,,,,,0,2020-10-27T18:08:53Z,2020-10-27T18:16:03Z,2020-10-27T18:16:03Z,OWNER,simonw/sqlite-utils/pulls/190,Refs #173,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/190/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 709861194,MDU6SXNzdWU3MDk4NjExOTQ=,180,Try running some tests using Hypothesis,9599,simonw,closed,0,,,,,1,2020-09-28T01:11:30Z,2020-10-19T04:51:55Z,2020-10-19T04:51:55Z,OWNER,,Inspired by this Twitter conversation: https://twitter.com/simonw/status/1310386009465479168,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/180/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 472115381,MDU6SXNzdWU0NzIxMTUzODE=,49,extracts= should support multiple-column extracts,9599,simonw,open,0,,,,,10,2019-07-24T07:06:41Z,2020-10-16T19:18:19Z,,OWNER,,"Lookup tables can be constructed on compound columns, but the `extracts=` option doesn't currently support that. Right now extracts can be defined in two ways: ```python # Extract these columns into tables with the same name: dogs = db.table(""dogs"", extracts=[""breed"", ""most_recent_trophy""]) # Same as above but with custom table names: dogs = db.table(""dogs"", extracts={""breed"": ""Breeds"", ""most_recent_trophy"": ""Trophies""}) ``` Need some kind of syntax for much more complicated extractions, like when two columns (say ""source"" and ""source_version"") are extracted into a single table.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/49/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 573578548,MDU6SXNzdWU1NzM1Nzg1NDg=,89,Ability to customize columns used by extracts= feature,9599,simonw,open,0,,,,,3,2020-03-01T16:54:48Z,2020-10-16T19:17:50Z,,OWNER,,"@simonw any thoughts on allow extracts to specify the lookup column name? If I'm understanding the documentation right, `.lookup()` allows you to define the ""value"" column (the documentation uses name), but when you use `extracts` keyword as part of `.insert()`, `.upsert()` etc. the lookup must be done against a column named ""value"". I have an existing lookup table that I've populated with columns ""id"" and ""name"" as opposed to ""id"" and ""value"", and seems I can't use `extracts=`, unless I'm missing something... Initial thought on how to do this would be to allow the dictionary value to be a tuple of table name column pair... so: ``` table = db.table(""trees"", extracts={""species_id"": (""Species"", ""name""}) ``` I haven't dug too much into the existing code yet, but does this make sense? Worth doing? _Originally posted by @chrishas35 in https://github.com/simonw/sqlite-utils/issues/46#issuecomment-592999503_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/89/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 683830416,MDU6SXNzdWU2ODM4MzA0MTY=,137,--load-extension for other sqlite-utils commands,9599,simonw,closed,0,,,,,1,2020-08-21T21:12:56Z,2020-10-16T19:14:32Z,2020-10-16T19:14:32Z,OWNER,,"e.g. for this: ``` calands-datasette % sqlite-utils tables calands.db --counts [{""table"": ""spatial_ref_sys"", ""count"": 4924}, {""table"": ""spatialite_history"", ""count"": 14}, {""table"": ""sqlite_sequence"", ""count"": 1}, {""table"": ""geometry_columns"", ""count"": 2}, {""table"": ""spatial_ref_sys_aux"", ""count"": 4873}, {""table"": ""views_geometry_columns"", ""count"": 0}, {""table"": ""virts_geometry_columns"", ""count"": 0}, {""table"": ""geometry_columns_statistics"", ""count"": 2}, {""table"": ""views_geometry_columns_statistics"", ""count"": 0}, {""table"": ""virts_geometry_columns_statistics"", ""count"": 0}, {""table"": ""geometry_columns_field_infos"", ""count"": 0}, {""table"": ""views_geometry_columns_field_infos"", ""count"": 0}, {""table"": ""virts_geometry_columns_field_infos"", ""count"": 0}, {""table"": ""geometry_columns_time"", ""count"": 2}, {""table"": ""geometry_columns_auth"", ""count"": 2}, {""table"": ""views_geometry_columns_auth"", ""count"": 0}, {""table"": ""virts_geometry_columns_auth"", ""count"": 0}, Traceback (most recent call last): File ""/usr/local/bin/sqlite-utils"", line 8, in sys.exit(cli()) File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py"", line 829, in __call__ return self.main(*args, **kwargs) File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py"", line 782, in main rv = self.invoke(ctx) File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py"", line 610, in invoke return callback(*args, **kwargs) File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py"", line 143, in tables for line in output_rows(_iter(), headers, nl, arrays, json_cols): File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py"", line 922, in output_rows for row, next_row in itertools.zip_longest(current_iter, next_iter): File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py"", line 123, in _iter row.append(db[name].count) File ""/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/db.py"", line 458, in count return self.db.conn.execute( sqlite3.OperationalError: no such module: VirtualSpatialIndex ``` The `tables` command could take `--load-extension` too - as could `rows` and other similar commands. Follow-on from #134 ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/137/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 709920027,MDU6SXNzdWU3MDk5MjAwMjc=,181,"pk=[""id""] should have same effect as pk=""id""",9599,simonw,closed,0,,,,,1,2020-09-28T04:28:07Z,2020-10-14T21:59:47Z,2020-10-14T21:59:47Z,OWNER,,"``` In [11]: db['one'].insert({""id"": 1, ""name"": ""oentuh""}, pk=""id"") Out[11]:
In [12]: db['two'].insert({""id"": 1, ""name"": ""oentuh""}, pk=[""id""]) Out[12]:
In [13]: db['one'].schema Out[13]: 'CREATE TABLE [one] (\n [id] INTEGER PRIMARY KEY,\n [name] TEXT\n)' In [14]: db['two'].schema Out[14]: 'CREATE TABLE [two] (\n [id] INTEGER,\n [name] TEXT\n)' ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/181/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 718952107,MDU6SXNzdWU3MTg5NTIxMDc=,185,Use db[table] consistently in documentation,9599,simonw,closed,0,,,,,0,2020-10-11T23:39:12Z,2020-10-12T00:13:41Z,2020-10-12T00:13:41Z,OWNER,,"The Python docs have a bunch of examples like this: https://sqlite-utils.readthedocs.io/en/stable/python-api.html ```python dogs.enable_fts([""name"", ""twitter""], create_triggers=True) ``` This would be easier for people to understand if it looked like this instead: ```python db[""dogs""].enable_fts([""name"", ""twitter""], create_triggers=True) ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/185/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 716955793,MDExOlB1bGxSZXF1ZXN0NDk5NjAzMzU5,184,Test against Python 3.9,9599,simonw,closed,0,,,,,0,2020-10-08T01:37:05Z,2020-10-08T01:44:06Z,2020-10-08T01:44:06Z,OWNER,simonw/sqlite-utils/pulls/184,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/184/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 712316959,MDU6SXNzdWU3MTIzMTY5NTk=,183,Try out GitHub code scanning,9599,simonw,closed,0,,,,,1,2020-09-30T22:16:14Z,2020-09-30T22:23:44Z,2020-09-30T22:23:44Z,OWNER,,https://github.blog/2020-09-30-code-scanning-is-now-available/,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/183/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 684118950,MDU6SXNzdWU2ODQxMTg5NTA=,138,extracts= doesn't configure foreign keys,9599,simonw,closed,0,,,,,2,2020-08-23T05:21:15Z,2020-09-24T22:47:01Z,2020-09-24T22:46:52Z,OWNER,,"In using `extracts=` for `shapefiles-to-sqlite` in https://github.com/simonw/shapefile-to-sqlite/issues/9 I've run into a couple of pretty serious flaws: - The columns in the original table are still `TEXT` even when the foreign key they are supposed to reference is an `INTEGER` - which means Datasette foreign key features don't actually work - Those foreign key relationships aren't setup automatically - creating them is left as an exercise for the developer",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/138/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 652700770,MDU6SXNzdWU2NTI3MDA3NzA=,119,Ability to remove a foreign key,9599,simonw,closed,0,,,,,3,2020-07-07T22:31:37Z,2020-09-24T20:36:59Z,2020-09-24T20:36:59Z,OWNER,,Useful if you add one but make a mistake and need to undo it without recreating the database from scratch.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/119/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 581795570,MDU6SXNzdWU1ODE3OTU1NzA=,93,Support more string values for types in .add_column(),9599,simonw,open,0,,,,,0,2020-03-15T19:32:49Z,2020-09-24T20:36:46Z,,OWNER,,"https://sqlite-utils.readthedocs.io/en/2.4.2/python-api.html#adding-columns says: > SQLite types you can specify are ""TEXT"", ""INTEGER"", ""FLOAT"" or ""BLOB"". As discovered in #92 this isn't the right list of values. I should expand this to match https://www.sqlite.org/datatype3.html",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/93/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 652961907,MDU6SXNzdWU2NTI5NjE5MDc=,121,Improved (and better documented) support for transactions,9599,simonw,open,0,,,,,3,2020-07-08T04:56:51Z,2020-09-24T20:36:46Z,,OWNER,,"_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_ We should put some thought into how this library supports and encourages smart use of transactions.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/121/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 688352145,MDU6SXNzdWU2ODgzNTIxNDU=,141,insert-files support for compressed values,9599,simonw,open,0,,,,,0,2020-08-28T20:59:46Z,2020-09-24T20:36:08Z,,OWNER,,"The `sqlar` format supports this, it would be useful if `insert-files` could support this too. https://www.sqlite.org/sqlar.html",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/141/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 708261775,MDU6SXNzdWU3MDgyNjE3NzU=,175,Add docs for .transform(column_order=),9599,simonw,closed,0,,,,,3,2020-09-24T15:19:04Z,2020-09-24T20:35:48Z,2020-09-24T16:00:56Z,OWNER,,"> Need to update docs for `.transform()` now that `column_order=` is available. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/174#discussion_r494403327_ Maybe also add this as an option to `sqlite-utils transform` - since reordering columns is actually a pretty nice capability.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/175/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 705190723,MDU6SXNzdWU3MDUxOTA3MjM=,160,"table.enable_fts(..., replace=True)",9599,simonw,closed,0,,,5896742,2.19,1,2020-09-20T21:36:23Z,2020-09-24T20:35:47Z,2020-09-20T22:05:51Z,OWNER,,"I noticed that https://til.simonwillison.net/ search doesn't use porter stemming. I'd like to add that, but since [the build script](https://github.com/simonw/til/blob/9d3f0fca30e94df3970df52b0447907a077e4673/build_database.py) always operates on an existing database (to avoid re-rendering markdown and re-building image thumbnails) I'd like it to only add porter stemming if it's not there already. So I'd like to be able to say ""set up FTS to look like this, and fix it if it doesn't"". I think the neatest way to do that is with a `replace=True` argument to `.enable_fts()`, for consistency with `def .create_view(self, name, sql, replace=True)`. So the `replace=True` argument would check and see if the configured FTS exists already with the correct options (columns, stemming, triggers) - and if any of those are incorrect it would call `.disable_fts()` and then create a new FTS configuration with the correct options. ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/160/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 706091046,MDU6SXNzdWU3MDYwOTEwNDY=,165,Make .transform() a keyword arguments only function,9599,simonw,closed,0,,,5897911,2.20,0,2020-09-22T05:37:29Z,2020-09-24T20:35:47Z,2020-09-22T06:39:12Z,OWNER,,And rename the first argument from `columns=` to `types=`,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/165/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 695377804,MDU6SXNzdWU2OTUzNzc4MDQ=,153,table.optimize() should delete junk rows from *_fts_docsize,9599,simonw,closed,0,,,,,3,2020-09-07T20:31:09Z,2020-09-24T20:35:46Z,2020-09-07T21:16:33Z,OWNER,,"> The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL: > ```sql > DELETE FROM [licenses_fts_docsize] WHERE id NOT IN ( > SELECT rowid FROM [licenses_fts]); > ``` > I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/153/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 696045581,MDU6SXNzdWU2OTYwNDU1ODE=,155,rebuild-fts command and table.rebuild_fts() method,9599,simonw,closed,0,,,,,2,2020-09-08T17:19:26Z,2020-09-24T20:35:46Z,2020-09-08T23:16:10Z,OWNER,,"https://sqlite.org/forum/forumpost/fa777fff86 > Easiest thing would be to run a 'rebuild' to rebuild the FTS index from scratch based on the contents of the content table. i.e. > > INSERT INTO licenses_fts(licenses_fts) VALUES('rebuild'); > > https://www.sqlite.org/fts5.html#the_rebuild_command",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/155/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 708301810,MDU6SXNzdWU3MDgzMDE4MTA=,177,Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key,9599,simonw,closed,0,,,,,1,2020-09-24T16:13:50Z,2020-09-24T20:35:03Z,2020-09-24T16:19:13Z,OWNER,,"These both currently require you to provide three strings, for `column`, `other_table`, `other_column`. Just providing `column` should be enough information.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/177/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 708293114,MDU6SXNzdWU3MDgyOTMxMTQ=,176,sqlite-utils transform column order option,9599,simonw,closed,0,,,,,2,2020-09-24T16:01:21Z,2020-09-24T20:34:51Z,2020-09-24T16:11:59Z,OWNER,,Split from #175,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/176/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 697179806,MDU6SXNzdWU2OTcxNzk4MDY=,157,sqlite-utils add-foreign-keys command,9599,simonw,closed,0,,,5896742,2.19,2,2020-09-09T21:44:30Z,2020-09-24T20:34:50Z,2020-09-20T20:14:30Z,OWNER,,"Like `add-foreign-key` but can do multiple foreign keys at once. Inspired by https://github.com/simonw/calands-datasette/blob/99de39dd80a906f5c1f16724467b0cd55ba4ef36/build.sh which does this: ``` sqlite-utils add-foreign-key calands.db units_with_maps ACCESS_TYP sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_NAME sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_LEV sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_TYP sqlite-utils add-foreign-key calands.db units_with_maps LAYER sqlite-utils add-foreign-key calands.db units_with_maps MNG_AGENCY sqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_LEV sqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_TYP sqlite-utils add-foreign-key calands.db units_with_maps COUNTY sqlite-utils add-foreign-key calands.db units_with_maps DES_TP ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/157/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 706017416,MDU6SXNzdWU3MDYwMTc0MTY=,164,sqlite-utils transform sub-command,9599,simonw,closed,0,,,5897911,2.20,4,2020-09-22T01:32:20Z,2020-09-24T20:34:50Z,2020-09-22T07:48:05Z,OWNER,,The `.transform()` method in #114 warrants an equivalent CLI tool.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/164/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 706757891,MDU6SXNzdWU3MDY3NTc4OTE=,169,"Progress bar for ""sqlite-utils extract""",9599,simonw,closed,0,,,5897911,2.20,0,2020-09-22T23:40:21Z,2020-09-24T20:34:50Z,2020-09-23T00:02:40Z,OWNER,,"> Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command. > > The operations are full table scans so calculating progress shouldn't be too difficult. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/169/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 707944044,MDExOlB1bGxSZXF1ZXN0NDkyMjU3NDA1,174,"Much, much faster extract() implementation",9599,simonw,closed,0,,,,,7,2020-09-24T07:52:31Z,2020-09-24T15:44:00Z,2020-09-24T15:43:56Z,OWNER,simonw/sqlite-utils/pulls/174,Takes my test down from ten minutes to four seconds. Refs #172.,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/174/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 707427200,MDU6SXNzdWU3MDc0MjcyMDA=,172,Improve performance of extract operations,9599,simonw,closed,0,,,,,9,2020-09-23T14:40:50Z,2020-09-24T15:43:57Z,2020-09-24T15:43:57Z,OWNER,,"This command took about 12 minutes (against a 150MB file with 680,000 rows in it): ``` sqlite-utils extract salaries.db salaries \ 'Organization Group Code' 'Organization Group' \ --table 'organization_groups' \ --fk-column 'organization_group_id' \ --rename 'Organization Group Code' code \ --rename 'Organization Group' name ``` I'm pretty confident we can do better than that.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/172/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 706768798,MDU6SXNzdWU3MDY3Njg3OTg=,170,Release notes for 2.20,9599,simonw,closed,0,,,5897911,2.20,1,2020-09-23T00:13:22Z,2020-09-23T00:31:25Z,2020-09-23T00:31:25Z,OWNER,,https://github.com/simonw/sqlite-utils/compare/2.19...b8e004,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/170/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 706098005,MDU6SXNzdWU3MDYwOTgwMDU=,167,Review the foreign key pragma stuff,9599,simonw,closed,0,,,5897911,2.20,1,2020-09-22T05:55:20Z,2020-09-23T00:13:02Z,2020-09-23T00:13:02Z,OWNER,,"> It is not possible to enable or disable foreign key constraints in the middle of a multi-statement transaction (when SQLite is not in autocommit mode). Attempting to do so does not return an error; it simply has no effect. https://sqlite.org/foreignkeys.html",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/167/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 470345929,MDU6SXNzdWU0NzAzNDU5Mjk=,42,"table.extract(...) method and ""sqlite-utils extract"" command",9599,simonw,closed,0,,,5897911,2.20,21,2019-07-19T14:09:36Z,2020-09-22T23:39:31Z,2020-09-22T23:37:49Z,OWNER,,"One of my favourite features of [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite) is that it can ""extract"" columns into a separate lookup table - for example: csvs-to-sqlite big_csv_file.csv -c country output.db This will turn the `country` column in the resulting table into a integer foreign key against a new `country` table. You can see an example of what that looks like here: https://san-francisco.datasettes.com/registered-business-locations-3d50679/Business+Corridor was extracted from https://san-francisco.datasettes.com/registered-business-locations-3d50679/Registered_Business_Locations_-_San_Francisco?Business%20Corridor=1 I'd like to have the same capability in `sqlite-utils` - but with the ability to run it against an existing SQLite table rather than just against a CSV.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/42/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 621989740,MDU6SXNzdWU2MjE5ODk3NDA=,114,table.transform() method for advanced alter table,9599,simonw,closed,0,,,5897911,2.20,26,2020-05-20T18:20:46Z,2020-09-22T07:51:37Z,2020-09-22T04:20:02Z,OWNER,,"SQLite's `ALTER TABLE` can only do the following: * Rename a table * Rename a column * Add a column Notably, it cannot drop columns - so tricks like ""add a float version of this text column, populate it, then drop the old one and rename"" won't work. The docs here https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes describe a way of implementing full alters safely within a transaction, but it's fiddly. 1. Create new table 2. Copy data 3. Drop old table 4. Rename new into old It would be great if `sqlite-utils` provided an abstraction to help make these kinds of changes safely.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/114/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 557830332,MDExOlB1bGxSZXF1ZXN0MzY5MzQ4MDg0,78,"New conversions= feature, refs #77",9599,simonw,closed,0,,,,,0,2020-01-31T00:02:33Z,2020-09-22T07:48:29Z,2020-01-31T00:24:31Z,OWNER,simonw/sqlite-utils/pulls/78,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/78/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 705975133,MDExOlB1bGxSZXF1ZXN0NDkwNjA3OTQ5,161,table.transform() method,9599,simonw,closed,0,,,5897911,2.20,13,2020-09-21T23:16:59Z,2020-09-22T07:48:24Z,2020-09-22T04:20:02Z,OWNER,simonw/sqlite-utils/pulls/161,"Refs #114 - [x] Ability to change the primary key - [x] Support for changing default value for columns - [x] Support for changing `NOT NULL` status of columns - [x] Support for copying existing foreign keys and removing them - Support for `conversions=` parameter - [x] Detailed documentation - [x] `PRAGMA foreign_keys` stuff",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/161/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 706092617,MDExOlB1bGxSZXF1ZXN0NDkwNzAzMTcz,166,Keyword only arguments for transform(),9599,simonw,closed,0,,,,,0,2020-09-22T05:41:44Z,2020-09-22T06:39:11Z,2020-09-22T06:39:11Z,OWNER,simonw/sqlite-utils/pulls/166,Refs #165,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/166/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 705995722,MDU6SXNzdWU3MDU5OTU3MjI=,162,A decorator for registering custom SQL functions,9599,simonw,closed,0,,,,,2,2020-09-22T00:18:32Z,2020-09-22T00:40:44Z,2020-09-22T00:32:17Z,OWNER,,"Syntactic sugar for `db.conn.create_function` - it would work something like this: ```python db = sqlite_utils.Database(""mydb.db"") @db.register_function def scramble(text): chars = list(text) random.shuffle(chars) return """".join(chars) ``` The decorator would inspect the function to find its name and arity (number of arguments). Having run the above you could then do: ```python db.execute(""select scramble('hello')"").fetchall() ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/162/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 616271236,MDU6SXNzdWU2MTYyNzEyMzY=,112,"add_foreign_key(...., ignore=True)",9599,simonw,closed,0,,,5896742,2.19,4,2020-05-12T00:24:00Z,2020-09-20T22:17:34Z,2020-09-20T22:17:34Z,OWNER,,"When using this library I often find myself wanting to ""add this foreign key, but only if it doesn't exist yet"". The `ignore=True` parameter is increasingly being used for this else where in the library (e.g. in `create_view()`).",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/112/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 531583658,MDU6SXNzdWU1MzE1ODM2NTg=,68,Add support for porter stemming in FTS,9599,simonw,closed,0,,,,,1,2019-12-02T22:35:52Z,2020-09-20T04:25:53Z,2020-09-20T04:25:47Z,OWNER,,FTS5 can have porter stemming enabled.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/68/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 695441530,MDU6SXNzdWU2OTU0NDE1MzA=,154,OperationalError: cannot change into wal mode from within a transaction,9599,simonw,open,0,,,,,2,2020-09-07T23:42:44Z,2020-09-07T23:47:10Z,,OWNER,,"I'm getting this error when running: sqlite-utils enable-wal beta.db `OperationalError: cannot change into wal mode from within a transaction` I'm worried that maybe that's because of this new code from #152: https://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 695359607,MDU6SXNzdWU2OTUzNTk2MDc=,150,Feature for tracing SQL queries,9599,simonw,closed,0,,,,,0,2020-09-07T19:43:08Z,2020-09-07T21:57:01Z,2020-09-07T21:57:01Z,OWNER,,"Debugging `sqlite-utils` when something weird happens (e.g. #149) can be a bit tricky since it runs a bunch of different SQL statements behind the scenes. An optional ""tracing"" mechanism for seeing what SQL is being executed would be useful.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/150/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 695360889,MDExOlB1bGxSZXF1ZXN0NDgxNjE2NzA0,151,Tracer mechanism for seeing underlying SQL,9599,simonw,closed,0,,,,,0,2020-09-07T19:46:43Z,2020-09-07T21:57:00Z,2020-09-07T21:57:00Z,OWNER,simonw/sqlite-utils/pulls/151,"Refs #150. Needs tests and documentation, including for the new `db.execute()` and `db.executescript()` methods.",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/151/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 695376054,MDU6SXNzdWU2OTUzNzYwNTQ=,152,Turn on recursive_triggers by default,9599,simonw,closed,0,,,,,2,2020-09-07T20:26:36Z,2020-09-07T21:17:48Z,2020-09-07T20:45:14Z,OWNER,,"https://www.sqlite.org/pragma.html#pragma_recursive_triggers says: > Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite. So I think the fix for the complex issue in #149 is to turn on `recursive_triggers` globally by default for `sqlite-utils`. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/152/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 695319258,MDU6SXNzdWU2OTUzMTkyNTg=,149,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",9599,simonw,closed,0,,,,,10,2020-09-07T18:06:16Z,2020-09-07T21:16:34Z,2020-09-07T21:16:34Z,OWNER,,"I'm seeing a weird issue with some of the SQLite databases that I am using with the FTS5 module. I have a database with a `licenses` table that contains 7 rows: The FTS table also has 7 rows: Somehow the accompanying `licenses_fts_docsize` shadow table now has 9,141 rows in it! And `licenses_fts_data` has 41 rows - should I expect that to have 7 rows? I have a hunch that it might be a problem with the triggers. These are the triggers that are updating that FTS table: | type | name | tbl_name | rootpage | sql | | --- | --- | --- | --- | --- | | trigger | licenses_ai | licenses | 0 | `CREATE TRIGGER [licenses_ai] AFTER INSERT ON [licenses] BEGIN INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` | | trigger | licenses_ad | licenses | 0 | `CREATE TRIGGER [licenses_ad] AFTER DELETE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); END` | | trigger | licenses_au | licenses | 0 | `CREATE TRIGGER [licenses_au] AFTER UPDATE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/149/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 695276328,MDU6SXNzdWU2OTUyNzYzMjg=,148,More attractive indentation of created FTS table schema,9599,simonw,closed,0,,,,,1,2020-09-07T16:49:30Z,2020-09-07T18:12:50Z,2020-09-07T18:12:50Z,OWNER,,"On https://github-to-sqlite.dogsheep.net/github/licenses_fts the create table SQL is displayed as: ```sql CREATE VIRTUAL TABLE [licenses_fts] USING FTS5 ( [name], content=[licenses] ); ``` It would be more aesthetically pleasing if it looked like this: ```sql CREATE VIRTUAL TABLE [licenses_fts] USING FTS5 ( [name], content=[licenses] ); ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/148/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 688395275,MDU6SXNzdWU2ODgzOTUyNzU=,144,Run some tests against numpy,9599,simonw,closed,0,,,,,2,2020-08-28T22:53:00Z,2020-08-28T22:57:05Z,2020-08-28T22:57:04Z,OWNER,,"Accidentally removed in #143: https://github.com/simonw/sqlite-utils/blob/d7d3f962861ef32c5ead8f514c8756f5b6f7c4a0/.travis.yml#L18-L19",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 688389933,MDU6SXNzdWU2ODgzODk5MzM=,143,Move to GitHub Actions CI,9599,simonw,closed,0,,,,,1,2020-08-28T22:34:11Z,2020-08-28T22:41:35Z,2020-08-28T22:41:35Z,OWNER,,,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/143/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 683804172,MDU6SXNzdWU2ODM4MDQxNzI=,134,--load-extension option for sqlite-utils query,9599,simonw,closed,0,,,,,4,2020-08-21T20:12:42Z,2020-08-21T21:06:26Z,2020-08-21T20:54:19Z,OWNER,,"I got this error: ``` % sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view' Traceback (most recent call last): ... cursor = db.conn.execute(sql, dict(param)) sqlite3.OperationalError: no such function: AsGeoJSON ``` A `--load-extension=/usr/local/lib/mod_spatialite.dylib` option (imitating the same option for Datasette) would help.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/134/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 677839979,MDU6SXNzdWU2Nzc4Mzk5Nzk=,133,Release a sdist to PyPI,9599,simonw,closed,0,,,,,1,2020-08-12T16:55:09Z,2020-08-12T17:05:06Z,2020-08-12T17:05:06Z,OWNER,,https://pypi.org/project/sqlite-utils/#files currently just has a wheel. I need this to package for homebrew: https://github.com/simonw/homebrew-datasette/issues/10,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/133/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 675839512,MDU6SXNzdWU2NzU4Mzk1MTI=,132,Features for enabling and disabling WAL mode,9599,simonw,closed,0,,,,,5,2020-08-10T03:25:44Z,2020-08-10T18:59:35Z,2020-08-10T18:59:35Z,OWNER,,I finally figured out how to enable WAL - turns out it's a property of the database file itself: https://github.com/simonw/til/blob/master/sqlite/enabling-wal-mode.md,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/132/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 671130371,MDU6SXNzdWU2NzExMzAzNzE=,130,Support tokenize option for FTS,9599,simonw,closed,0,,,,,3,2020-08-01T19:27:22Z,2020-08-01T20:51:28Z,2020-08-01T20:51:14Z,OWNER,,"FTS5 supports things like porter stemming using a `tokenize=` option: https://www.sqlite.org/fts5.html#tokenizers Something like this in code: ``` CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} ( {columns}, tokenize='porter', content=[{table}] ); ``` I tried this out just now and it worked exactly as expected. So... `db[table].enable_fts(...) should accept a 'tokenize=` argument, and `sqlite-utils enable-fts ...` should support a `--tokenize` option.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/130/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 665802405,MDU6SXNzdWU2NjU4MDI0MDU=,124,sqlite-utils query should support named parameters,9599,simonw,closed,0,,,,,1,2020-07-26T15:25:10Z,2020-07-30T22:57:51Z,2020-07-27T03:53:58Z,OWNER,,"To help out with escaping - so you can run this: sqlite-utils query ""insert into foo (blah) values (:blah)"" --param blah `something here`",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 668308777,MDU6SXNzdWU2NjgzMDg3Nzc=,129,"""insert-files --sqlar"" for creating SQLite archives",9599,simonw,closed,0,,,,,2,2020-07-30T02:28:29Z,2020-07-30T22:41:01Z,2020-07-30T22:40:55Z,OWNER,,"A `--sqlar` option could cause `insert-files` to behave in the same way as SQLite's own sqlar mechanism. https://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 666040390,MDU6SXNzdWU2NjYwNDAzOTA=,127,Ability to insert files piped to insert-files stdin,9599,simonw,closed,0,,,,,3,2020-07-27T07:09:33Z,2020-07-30T03:08:52Z,2020-07-30T03:08:18Z,OWNER,,"> Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option. > > cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif > _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/127/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 666639051,MDU6SXNzdWU2NjY2MzkwNTE=,128,Support UUID and memoryview types,9599,simonw,closed,0,,,,,1,2020-07-27T23:08:34Z,2020-07-30T01:10:43Z,2020-07-30T01:10:43Z,OWNER,,`psycopg2` can return data from PostgreSQL as `uuid.UUID` or `memoryview` objects. These should to be supported by `sqlite-utils` - mainly for https://github.com/simonw/db-to-sqlite,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/128/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 665700495,MDU6SXNzdWU2NjU3MDA0OTU=,122,CLI utility for inserting binary files into SQLite,9599,simonw,closed,0,,,,,10,2020-07-26T03:27:39Z,2020-07-27T07:10:41Z,2020-07-27T07:09:03Z,OWNER,,"SQLite BLOB columns can store entire binary files. The challenge is inserting them, since they don't neatly fit into JSON objects. It would be great if the `sqlite-utils` CLI had a trick for helping with this. Inspired by https://github.com/simonw/datasette-media/issues/14",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/122/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 665819048,MDU6SXNzdWU2NjU4MTkwNDg=,126,Ability to insert binary data on the CLI using JSON,9599,simonw,closed,0,,,,,2,2020-07-26T16:54:14Z,2020-07-27T04:00:33Z,2020-07-27T03:59:45Z,OWNER,,"> I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says ""these columns are base64 encoded, store the decoded data in a BLOB"". > > That would solve inserting binary data using JSON too. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 665817570,MDU6SXNzdWU2NjU4MTc1NzA=,125,"Output binary columns in ""sqlite-utils query"" JSON",9599,simonw,closed,0,,,,,4,2020-07-26T16:47:02Z,2020-07-27T00:49:41Z,2020-07-27T00:48:45Z,OWNER,,You get an error if you try to run a query that returns data from a BLOB.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/125/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 665701216,MDU6SXNzdWU2NjU3MDEyMTY=,123,--raw option for outputting binary content,9599,simonw,closed,0,,,,,0,2020-07-26T03:35:39Z,2020-07-26T16:44:11Z,2020-07-26T16:44:11Z,OWNER,,"Related to the `insert-files` work in #122 - it should be easy to get binary data back out of the database again. One way to do that could be: sqlite-utils files.db ""select content from files where key = 'foo.jpg'"" --raw The `--raw` option would cause just the contents of the first column to be output directly to stdout.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/123/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 644161221,MDU6SXNzdWU2NDQxNjEyMjE=,117,Support for compound (composite) foreign keys,9599,simonw,open,0,,,,,3,2020-06-23T21:33:42Z,2020-06-23T21:40:31Z,,OWNER,,"It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite Their example looks like this: ```sql CREATE TABLE album( albumartist TEXT, albumname TEXT, albumcover BINARY, PRIMARY KEY(albumartist, albumname) ); CREATE TABLE song( songid INTEGER, songartist TEXT, songalbum TEXT, songname TEXT, FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) ); ``` Here's what that looks like in sqlite-utils: ``` In [1]: import sqlite_utils In [2]: import sqlite3 In [3]: conn = sqlite3.connect("":memory:"") In [4]: conn Out[4]: In [5]: conn.executescript("""""" ...: CREATE TABLE album( ...: albumartist TEXT, ...: albumname TEXT, ...: albumcover BINARY, ...: PRIMARY KEY(albumartist, albumname) ...: ); ...: ...: CREATE TABLE song( ...: songid INTEGER, ...: songartist TEXT, ...: songalbum TEXT, ...: songname TEXT, ...: FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) ...: ); ...: """""") Out[5]: In [6]: db = sqlite_utils.Database(conn) In [7]: db.tables Out[7]: [
,
] In [8]: db.tables[0].foreign_keys Out[8]: [] In [9]: db.tables[1].foreign_keys Out[9]: [ForeignKey(table='song', column='songartist', other_table='album', other_column='albumartist'), ForeignKey(table='song', column='songalbum', other_table='album', other_column='albumname')] ``` The table appears to have two separate foreign keys, when actually it has a single compound composite foreign key.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/117/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",, 644122661,MDU6SXNzdWU2NDQxMjI2NjE=,116,Documentation for table.pks introspection property,9599,simonw,closed,0,,,,,2,2020-06-23T20:27:24Z,2020-06-23T21:21:33Z,2020-06-23T21:03:14Z,OWNER,,https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L535-L540,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/116/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 637889964,MDU6SXNzdWU2Mzc4ODk5NjQ=,115,Ability to execute insert/update statements with the CLI,9599,simonw,closed,0,,,,,1,2020-06-12T17:01:17Z,2020-06-12T17:51:11Z,2020-06-12T17:41:10Z,OWNER,,"``` $ sqlite-utils github.db ""update stars set starred_at = ''"" Traceback (most recent call last): File ""/Users/simon/.local/bin/sqlite-utils"", line 8, in sys.exit(cli()) File ""/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py"", line 829, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py"", line 782, in main rv = self.invoke(ctx) File ""/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py"", line 610, in invoke return callback(*args, **kwargs) File ""/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/cli.py"", line 673, in query headers = [c[0] for c in cursor.description] TypeError: 'NoneType' object is not iterable ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/115/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 461215118,MDU6SXNzdWU0NjEyMTUxMTg=,30,Option to open database in read-only mode,9599,simonw,closed,0,,,,,1,2019-06-26T22:50:38Z,2020-05-11T19:17:17Z,2020-05-11T19:17:17Z,OWNER,,Would this make it 100% safe to run reads against a database file that is being written to by another process?,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/30/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 615477131,MDU6SXNzdWU2MTU0NzcxMzE=,111,sqlite-utils drop-table and drop-view commands,9599,simonw,closed,0,,,,,2,2020-05-10T21:10:42Z,2020-05-11T01:58:36Z,2020-05-11T00:44:26Z,OWNER,,Would be useful to be able to drop views and tables from the CLI.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 612658444,MDU6SXNzdWU2MTI2NTg0NDQ=,109,"table.create_index(..., ignore=True)",9599,simonw,closed,0,,,,,1,2020-05-05T14:44:21Z,2020-05-05T14:46:53Z,2020-05-05T14:46:53Z,OWNER,,Option to silently do nothing if the index already exists.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/109/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 611222968,MDU6SXNzdWU2MTEyMjI5Njg=,107,sqlite-utils create-view CLI command,9599,simonw,closed,0,,,,,2,2020-05-02T16:15:13Z,2020-05-03T15:36:58Z,2020-05-03T15:36:37Z,OWNER,,Can go with #27 - `sqlite-utils create-table`.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 455496504,MDU6SXNzdWU0NTU0OTY1MDQ=,27,sqlite-utils create-table command,9599,simonw,closed,0,,,,,8,2019-06-13T01:43:30Z,2020-05-03T15:26:15Z,2020-05-03T15:26:15Z,OWNER,,"Spun off from #24 - it would be useful if CLI users could create new tables (with explicit column types, not null rules and defaults) without having to insert an example record. - [x] Get it working - [x] Support `--pk` - [x] Support `--not-null` - [x] Support `--default` - [x] Support `--fk colname othertable othercol` - [x] Support `--replace` and `--ignore` - [x] Documentation",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/27/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 611326701,MDU6SXNzdWU2MTEzMjY3MDE=,108,Documentation unit tests for CLI commands,9599,simonw,closed,0,,,,,2,2020-05-03T03:58:42Z,2020-05-03T04:13:57Z,2020-05-03T04:13:57Z,OWNER,,Have a test that ensures all CLI commands are documented.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/108/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 611216862,MDU6SXNzdWU2MTEyMTY4NjI=,106,"create_view(..., ignore=True, replace=True) parameters",9599,simonw,closed,0,,,,,1,2020-05-02T15:45:21Z,2020-05-02T16:04:51Z,2020-05-02T16:02:10Z,OWNER,,"Two new parameters which specify what should happen if the view already exists. I want this for https://github.com/dogsheep/github-to-sqlite/issues/37 Here's the current `create_view()` implementation: https://github.com/simonw/sqlite-utils/blob/b4d953d3ccef28bb81cea40ca165a647b59971fa/sqlite_utils/db.py#L325-L332 `ignore=True` will not do anything if the view exists already. `replace=True` will drop and redefine the view - but only if its SQL definition differs, otherwise it will be left alone.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/106/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 602569315,MDU6SXNzdWU2MDI1NjkzMTU=,102,Can't store an array or dictionary containing a bytes value,9599,simonw,closed,0,,,,,0,2020-04-18T22:49:21Z,2020-05-01T20:45:45Z,2020-05-01T20:45:45Z,OWNER,,"``` In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database(memory=True) In [3]: db[""t""].insert({""id"": 1, ""data"": {""foo"": b""bytes""}}) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in ----> 1 db[""t""].insert({""id"": 1, ""data"": {""foo"": b""bytes""}}) ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert(self, record, pk, foreign_keys, column_order, not_null, defaults, hash_id, alter, ignore, replace, extracts, conversions, columns) 950 extracts=extracts, 951 conversions=conversions, --> 952 columns=columns, 953 ) 954 ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, conversions, columns, upsert) 1052 for key in all_columns: 1053 value = jsonify_if_needed( -> 1054 record.get(key, None if key != hash_id else _hash(record)) 1055 ) 1056 if key in extracts: ~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in jsonify_if_needed(value) 1318 def jsonify_if_needed(value): 1319 if isinstance(value, (dict, list, tuple)): -> 1320 return json.dumps(value) 1321 elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)): 1322 return value.isoformat() /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw) 229 cls is None and indent is None and separators is None and 230 default is None and not sort_keys and not kw): --> 231 return _default_encoder.encode(obj) 232 if cls is None: 233 cls = JSONEncoder /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in encode(self, o) 197 # exceptions aren't as detailed. The list call should be roughly 198 # equivalent to the PySequence_Fast that ''.join() would do. --> 199 chunks = self.iterencode(o, _one_shot=True) 200 if not isinstance(chunks, (list, tuple)): 201 chunks = list(chunks) /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in iterencode(self, o, _one_shot) 255 self.key_separator, self.item_separator, self.sort_keys, 256 self.skipkeys, _one_shot) --> 257 return _iterencode(o, 0) 258 259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr, /usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in default(self, o) 177 178 """""" --> 179 raise TypeError(f'Object of type {o.__class__.__name__} ' 180 f'is not JSON serializable') 181 TypeError: Object of type bytes is not JSON serializable ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/102/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 610853576,MDU6SXNzdWU2MTA4NTM1NzY=,105,"""sqlite-utils views"" command",9599,simonw,closed,0,,,,,1,2020-05-01T16:56:11Z,2020-05-01T20:40:07Z,2020-05-01T20:38:36Z,OWNER,,Similar to `sqlite-utils tables`. See also #104.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/105/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 610853393,MDU6SXNzdWU2MTA4NTMzOTM=,104,"--schema option to ""sqlite-utils tables""",9599,simonw,closed,0,,,,,0,2020-05-01T16:55:49Z,2020-05-01T17:12:37Z,2020-05-01T17:12:37Z,OWNER,,Adds output showing the table schema.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/104/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 601392318,MDU6SXNzdWU2MDEzOTIzMTg=,101,README should include an example of CLI data insertion,9599,simonw,closed,0,,,,,0,2020-04-16T19:45:37Z,2020-04-17T23:59:49Z,2020-04-17T23:59:49Z,OWNER,,Maybe using `curl` from the GitHub API.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/101/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 601358649,MDU6SXNzdWU2MDEzNTg2NDk=,100,"Mechanism for forcing column-type, over-riding auto-detection",9599,simonw,closed,0,,,,,3,2020-04-16T19:12:52Z,2020-04-17T23:53:32Z,2020-04-17T23:53:32Z,OWNER,,"As seen in https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406 - there's a problem where you insert a record with a `None` value for a column and that column is created as `TEXT` - but actually you intended it to be an `INT` (as later examples will demonstrate). Some kind of mechanism for over-riding the detected types of columns would be useful here.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/100/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 598640234,MDU6SXNzdWU1OTg2NDAyMzQ=,99,.upsert_all() should maybe error if dictionaries passed to it do not have the same keys,9599,simonw,closed,0,,,,,2,2020-04-13T03:02:25Z,2020-04-13T03:05:20Z,2020-04-13T03:05:04Z,OWNER,,"While investigating #98 I stumbled across this: ``` def test_upsert_compound_primary_key(fresh_db): table = fresh_db[""table""] table.upsert_all( [ {""species"": ""dog"", ""id"": 1, ""name"": ""Cleo"", ""age"": 4}, {""species"": ""cat"", ""id"": 1, ""name"": ""Catbag""}, ], pk=(""species"", ""id""), ) table.upsert_all( [ {""species"": ""dog"", ""id"": 1, ""age"": 5}, {""species"": ""dog"", ""id"": 2, ""name"": ""New Dog"", ""age"": 1}, ], pk=(""species"", ""id""), ) > assert [ {""species"": ""dog"", ""id"": 1, ""name"": ""Cleo"", ""age"": 5}, {""species"": ""cat"", ""id"": 1, ""name"": ""Catbag"", ""age"": None}, {""species"": ""dog"", ""id"": 2, ""name"": ""New Dog"", ""age"": 1}, ] == list(table.rows) E AssertionError: assert [{'age': 5, '...cies': 'dog'}] == [{'age': 5, '...cies': 'dog'}] E At index 0 diff: {'species': 'dog', 'id': 1, 'name': 'Cleo', 'age': 5} != {'species': 'dog', 'id': 1, 'name': None, 'age': 5} E Full diff: E - [{'age': 5, 'id': 1, 'name': 'Cleo', 'species': 'dog'}, E ? ^^^ -- E + [{'age': 5, 'id': 1, 'name': None, 'species': 'dog'}, E ? ^^^ E {'age': None, 'id': 1, 'name': 'Catbag', 'species': 'cat'}, E {'age': 1, 'id': 2, 'name': 'New Dog', 'species': 'dog'}] ``` If you run `.upsert_all()` with multiple dictionaries it doesn't quite have the effect you might expect.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/99/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 586486367,MDU6SXNzdWU1ODY0ODYzNjc=,95,Columns with only null values are no longer created in the database,9599,simonw,closed,0,,,,,0,2020-03-23T20:07:42Z,2020-03-23T20:31:15Z,2020-03-23T20:31:15Z,OWNER,,"Bug introduced in #94, and released in `2.4.3`.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/95/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 586477757,MDU6SXNzdWU1ODY0Nzc3NTc=,94,"If column data is a mixture of integers and nulls, detected type should be INTEGER",9599,simonw,closed,0,,,,,0,2020-03-23T19:51:46Z,2020-03-23T19:57:10Z,2020-03-23T19:57:10Z,OWNER,,It looks like detected type for that case is TEXT at the moment.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/94/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 581339961,MDU6SXNzdWU1ODEzMzk5NjE=,92,.columns_dict doesn't work for all possible column types,9599,simonw,closed,0,,,,,7,2020-03-14T19:30:35Z,2020-03-15T18:37:43Z,2020-03-14T20:04:14Z,OWNER,,"Got this error: ``` File "".../python3.7/site-packages/sqlite_utils/db.py"", line 462, in for column in self.columns KeyError: 'REAL' ``` `.columns_dict` uses `REVERSE_COLUMN_TYPE_MAPPING`: https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L457-L463 `REVERSE_COLUMN_TYPE_MAPPING` defines `FLOAT` not `REAL`A https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L74",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/92/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 573740712,MDU6SXNzdWU1NzM3NDA3MTI=,90,Cannot .enable_fts() for columns with spaces in their names,9599,simonw,closed,0,,,,,0,2020-03-02T06:06:03Z,2020-03-02T06:10:49Z,2020-03-02T06:10:49Z,OWNER,,"``` import sqlite_utils db = sqlite_utils.Database(memory=True) db[""test""].insert({""space in name"": ""hello""}) db[""test""].enable_fts([""space in name""]) --------------------------------------------------------------------------- OperationalError Traceback (most recent call last) in ----> 1 db['test'].enable_fts([""space in name""]) /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in enable_fts(self, columns, fts_version, create_triggers) 755 ) 756 self.db.conn.executescript(sql) --> 757 self.populate_fts(columns) 758 759 if create_triggers: /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in populate_fts(self, columns) 787 table=self.name, columns="", "".join(columns) 788 ) --> 789 self.db.conn.executescript(sql) 790 return self 791 OperationalError: near ""in"": syntax error ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/90/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 471780443,MDU6SXNzdWU0NzE3ODA0NDM=,46,extracts= option for insert/update/etc,9599,simonw,closed,0,,,,,3,2019-07-23T15:55:46Z,2020-03-01T16:53:40Z,2019-07-23T17:00:44Z,OWNER,,"Relates to #42 and #44. I want the ability to extract values out into lookup tables during bulk insert/upsert operations. `db.insert_all(rows, extracts=[""species""])` - creates species table for values in the species column `db.insert_all(rows, extracts={""species"": ""Species""})` - as above but the new table is called `Species`.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/46/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 571805300,MDU6SXNzdWU1NzE4MDUzMDA=,88,"table.disable_fts() method and ""sqlite-utils disable-fts ..."" command",9599,simonw,closed,0,,,,,5,2020-02-27T04:00:50Z,2020-02-27T04:40:44Z,2020-02-27T04:40:44Z,OWNER,,This would make it easier to iterate on the FTS configuration for a database without having to wipe and recreate the database each time.,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/88/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 565837965,MDU6SXNzdWU1NjU4Mzc5NjU=,87,Should detect collections.OrderedDict as a regular dictionary,9599,simonw,closed,0,,,,,2,2020-02-16T02:06:34Z,2020-02-16T02:20:59Z,2020-02-16T02:20:59Z,OWNER,,"``` File ""...python3.7/site-packages/sqlite_utils/db.py"", line 292, in create_table column_type=COLUMN_TYPE_MAPPING[column_type], KeyError: ```",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/87/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 562911863,MDU6SXNzdWU1NjI5MTE4NjM=,85,Create index doesn't work for columns containing spaces,9599,simonw,closed,0,,,,,1,2020-02-11T00:34:46Z,2020-02-11T05:13:20Z,2020-02-11T05:13:20Z,OWNER,,,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/85/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 559374410,MDU6SXNzdWU1NTkzNzQ0MTA=,83,"Make db[""table""].exists a documented API",9599,simonw,closed,0,,,,,1,2020-02-03T22:31:44Z,2020-02-08T23:58:35Z,2020-02-08T23:56:23Z,OWNER,,Right now it's a static thing which might get out-of-sync with the database. It should probably be a live check. Maybe call it `.exists()` instead?,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/83/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 561460274,MDU6SXNzdWU1NjE0NjAyNzQ=,84,.upsert() with hash_id throws error,9599,simonw,closed,0,,,,,0,2020-02-07T07:08:19Z,2020-02-07T07:17:11Z,2020-02-07T07:17:11Z,OWNER,,"```python db[table_name].upsert_all(rows, hash_id=""pk"") ``` This throws an error: `PrimaryKeyRequired('upsert() requires a pk')` The problem is, if you try this: ```python db[table_name].upsert_all(rows, hash_id=""pk"", pk=""pk"") ``` You get this error: `AssertionError('Use either pk= or hash_id=')` `hash_id=` should imply that `pk=` that column.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/84/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 558600274,MDU6SXNzdWU1NTg2MDAyNzQ=,81,"Remove .detect_column_types() from table, make it a documented API",9599,simonw,closed,0,,,,,4,2020-02-01T21:25:54Z,2020-02-01T21:55:35Z,2020-02-01T21:55:35Z,OWNER,,"I used it in `geojson-to-sqlite` here: https://github.com/simonw/geojson-to-sqlite/blob/f10e44264712dd59ae7dfa2e6fd5a904b682fb33/geojson_to_sqlite/utils.py#L45-L50 It would make more sense for this method to live on the Database rather than the Table - or even to exist as a separate utility method entirely. Then it should be documented.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/81/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 557892819,MDExOlB1bGxSZXF1ZXN0MzY5Mzk0MDQz,80,on_create mechanism for after table creation,9599,simonw,closed,0,,,,,5,2020-01-31T03:38:48Z,2020-01-31T05:08:04Z,2020-01-31T05:08:04Z,OWNER,simonw/sqlite-utils/pulls/80,"I need this for `geojson-to-sqlite`, in particular https://github.com/simonw/geojson-to-sqlite/issues/6",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/80/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 521868864,MDU6SXNzdWU1MjE4Njg4NjQ=,66,"The "".upsert()"" method is misnamed",9599,simonw,closed,0,,,,,15,2019-11-12T23:48:28Z,2019-12-31T01:30:21Z,2019-12-31T01:30:20Z,OWNER,,"This thread here is illuminating: https://stackoverflow.com/questions/3634984/insert-if-not-exists-else-update The term `UPSERT` in SQLite has a specific meaning as-of 3.24.0 (2018-06-04): https://www.sqlite.org/lang_UPSERT.html It means ""behave as an UPDATE or a no-op if the INSERT would violate a uniqueness constraint"". The syntax in 3.24.0+ looks like this (confusingly it does not use the term ""upsert""): ```sql INSERT INTO phonebook(name,phonenumber) VALUES('Alice','704-555-1212') ON CONFLICT(name) DO UPDATE SET phonenumber=excluded.phonenumber ``` Here's the problem: the `sqlite-utils` `.upsert()` and `.upsert_all()` methods don't do this. They use the following SQL: ```sql INSERT OR REPLACE INTO [{table}] ({columns}) VALUES {rows}; ``` If the record already exists, it will be entirely replaced by a new record - as opposed to updating any specified fields but leaving existing fields as they are (the behaviour of ""upsert"" in SQLite itself).",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/66/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 529376481,MDExOlB1bGxSZXF1ZXN0MzQ2MjY0OTI2,67,Run tests against 3.5 too,9599,simonw,closed,0,,,,,2,2019-11-27T14:20:35Z,2019-12-31T01:29:44Z,2019-12-31T01:29:43Z,OWNER,simonw/sqlite-utils/pulls/67,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/67/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 543738004,MDExOlB1bGxSZXF1ZXN0MzU3OTkyNTg4,72,Fixed implementation of upsert,9599,simonw,closed,0,,,,,0,2019-12-30T05:08:05Z,2019-12-30T05:29:24Z,2019-12-30T05:29:24Z,OWNER,simonw/sqlite-utils/pulls/72,Refs #66,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/72/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 542814756,MDU6SXNzdWU1NDI4MTQ3NTY=,71,Tests are failing due to missing FTS5,9599,simonw,closed,0,,,,,3,2019-12-27T09:41:16Z,2019-12-27T09:49:37Z,2019-12-27T09:49:37Z,OWNER,,"https://travis-ci.com/simonw/sqlite-utils/jobs/268436167 This is a recent change: 2 months ago they worked fine. I'm not sure what changed here. Maybe something to do with https://launchpad.net/~jonathonf/+archive/ubuntu/backports ?",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/71/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 519039316,MDExOlB1bGxSZXF1ZXN0MzM3ODUzMzk0,65,Release 1.12.1,9599,simonw,closed,0,,,,,0,2019-11-07T04:51:29Z,2019-11-07T04:58:48Z,2019-11-07T04:58:47Z,OWNER,simonw/sqlite-utils/pulls/65,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/65/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 476413293,MDU6SXNzdWU0NzY0MTMyOTM=,52,Throws error if .insert_all() / .upsert_all() called with empty list,9599,simonw,closed,0,,,,,1,2019-08-03T04:09:00Z,2019-11-07T04:32:39Z,2019-11-07T04:32:39Z,OWNER,,See also https://github.com/simonw/db-to-sqlite/issues/18,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/52/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 519032008,MDExOlB1bGxSZXF1ZXN0MzM3ODQ3NTcz,64,test_insert_upsert_all_empty_list,9599,simonw,closed,0,,,,,0,2019-11-07T04:24:45Z,2019-11-07T04:32:38Z,2019-11-07T04:32:38Z,OWNER,simonw/sqlite-utils/pulls/64,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/64/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 517241040,MDU6SXNzdWU1MTcyNDEwNDA=,63,ensure_index() method,9599,simonw,closed,0,,,,,1,2019-11-04T15:51:22Z,2019-11-04T16:20:36Z,2019-11-04T16:20:35Z,OWNER,,"```python db[""table""].ensure_index([""col1"", ""col2""]) ``` This will do the following: - if the specified table or column does not exist, do nothing - if they exist and already have an index, do nothing - otherwise, create the index I want this for tools like [twitter-to-sqlite search](https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches) where the `search_runs` table may or not have been created yet but, if it IS created, I want to put an index on the `hash` column.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/63/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 488338965,MDU6SXNzdWU0ODgzMzg5NjU=,59,Ability to introspect triggers,9599,simonw,closed,0,,,,,0,2019-09-02T23:47:16Z,2019-09-03T01:52:36Z,2019-09-03T00:09:42Z,OWNER,,"Now that we're creating triggers (thanks to @amjith in #57) it would be neat if we could introspect them too. I'm thinking: `db.triggers` - lists all triggers for the database `db[""tablename""].triggers` - lists triggers for that table The underlying query for this is `select * from sqlite_master where type = 'trigger'` I'll return the trigger information in a new namedtuple, similar to how Indexes and ForeignKeys work.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/59/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 488341021,MDExOlB1bGxSZXF1ZXN0MzEzMzgzMzE3,60,db.triggers and table.triggers introspection,9599,simonw,closed,0,,,,,0,2019-09-03T00:04:32Z,2019-09-03T00:09:42Z,2019-09-03T00:09:42Z,OWNER,simonw/sqlite-utils/pulls/60,Closes #59,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/60/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 481887482,MDExOlB1bGxSZXF1ZXN0MzA4MjkyNDQ3,55,Ability to introspect and run queries against views,9599,simonw,closed,0,,,,,1,2019-08-17T13:40:56Z,2019-08-23T12:19:42Z,2019-08-23T12:19:42Z,OWNER,simonw/sqlite-utils/pulls/55,See #54 ,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/55/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 449565204,MDU6SXNzdWU0NDk1NjUyMDQ=,23,Syntactic sugar for creating m2m records,9599,simonw,closed,0,,,,,10,2019-05-29T02:17:48Z,2019-08-04T03:54:58Z,2019-08-04T03:37:34Z,OWNER,,Python library only. What would be a syntactically pleasant way of creating a m2m record?,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/23/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 476436920,MDExOlB1bGxSZXF1ZXN0MzAzOTkwNjgz,53,Work in progress: m2m() method for creating many-to-many records,9599,simonw,closed,0,,,,,0,2019-08-03T10:03:56Z,2019-08-04T03:38:10Z,2019-08-04T03:37:33Z,OWNER,simonw/sqlite-utils/pulls/53,"- [x] `table.insert({""name"": ""Barry""}).m2m(""tags"", lookup={""tag"": ""Coworker""})` - [x] Explicit table name `.m2m(""humans"", ..., m2m_table=""relationships"")` - [x] Automatically use an existing m2m table if a single obvious candidate exists (a table with two foreign keys in the correct directions) - [x] Require the explicit `m2m_table=` argument if multiple candidates for the m2m table exist - [x] Documentation Refs #23",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/53/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 462430920,MDU6SXNzdWU0NjI0MzA5MjA=,35,table.update(...) method,9599,simonw,closed,0,,,,,2,2019-06-30T18:06:15Z,2019-07-28T15:43:52Z,2019-07-28T15:43:52Z,OWNER,,"Spun off from #23 - this method will allow a user to update a specific row. Currently the only way to do that it is to call `.upsert({full record})` with the primary key field matching an existing record - but this does not support partial updates. ```python db[""events""].update(3, {""name"": ""Renamed""}) ``` This method only works on an existing table, so there's no need for a `pk=""id""` specifier - it can detect the primary key by looking at the table. If the primary key is compound the first argument can be a tuple: ```python db[""events_venues""].update((3, 2), {""custom_label"": ""Label""}) ``` The method can be called without the second dictionary argument. Doing this selects the row specified by the primary key (throwing an error if it does not exist) and remembers it so that chained operations can be carried out - see proposal in https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507055345 ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/35/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 467862459,MDExOlB1bGxSZXF1ZXN0Mjk3NDEyNDY0,38,table.update() method,9599,simonw,closed,0,,,,,2,2019-07-14T17:03:49Z,2019-07-28T15:43:51Z,2019-07-28T15:43:51Z,OWNER,simonw/sqlite-utils/pulls/38,"Refs #35 Still to do: - [x] Unit tests - [x] Switch to using `.get()` - [x] Better exceptions, plus unit tests for what happens if pk does not exist - [x] Documentation - [x] Ensure compound primary keys work properly - [x] `alter=True` support",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/38/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 473733752,MDExOlB1bGxSZXF1ZXN0MzAxODI0MDk3,51,"Fix for too many SQL variables, closes #50",9599,simonw,closed,0,,,,,1,2019-07-28T11:30:30Z,2019-07-28T11:59:32Z,2019-07-28T11:59:32Z,OWNER,simonw/sqlite-utils/pulls/51,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/51/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 471797101,MDExOlB1bGxSZXF1ZXN0MzAwMzc3NTk5,47,extracts= table parameter,9599,simonw,closed,0,,,,,0,2019-07-23T16:30:29Z,2019-07-23T17:00:43Z,2019-07-23T17:00:43Z,OWNER,simonw/sqlite-utils/pulls/47,Still needs docs. Refs #46,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/47/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 470691999,MDU6SXNzdWU0NzA2OTE5OTk=,43,.add_column() doesn't match indentation of initial creation,9599,simonw,closed,0,,,,,3,2019-07-20T16:33:10Z,2019-07-23T13:09:11Z,2019-07-23T13:09:05Z,OWNER,,"I spotted a table which was created once and then had columns added to it and the formatted SQL looks like this: ```sql CREATE TABLE [records] ( [type] TEXT, [sourceName] TEXT, [sourceVersion] TEXT, [unit] TEXT, [creationDate] TEXT, [startDate] TEXT, [endDate] TEXT, [value] TEXT, [metadata_Health Mate App Version] TEXT, [metadata_Withings User Identifier] TEXT, [metadata_Modified Date] TEXT, [metadata_Withings Link] TEXT, [metadata_HKWasUserEntered] TEXT , [device] TEXT, [metadata_HKMetadataKeyHeartRateMotionContext] TEXT, [metadata_HKDeviceManufacturerName] TEXT, [metadata_HKMetadataKeySyncVersion] TEXT, [metadata_HKMetadataKeySyncIdentifier] TEXT, [metadata_HKSwimmingStrokeStyle] TEXT, [metadata_HKVO2MaxTestType] TEXT, [metadata_HKTimeZone] TEXT, [metadata_Average HR] TEXT, [metadata_Recharge] TEXT, [metadata_Lights] TEXT, [metadata_Asleep] TEXT, [metadata_Rating] TEXT, [metadata_Energy Threshold] TEXT, [metadata_Deep Sleep] TEXT, [metadata_Nap] TEXT, [metadata_Edit Slots] TEXT, [metadata_Tags] TEXT, [metadata_Daytime HR] TEXT) ``` It would be nice if the columns that were added later matched the indentation of the initial columns.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/43/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 471628483,MDU6SXNzdWU0NzE2Mjg0ODM=,44,Utilities for building lookup tables,9599,simonw,closed,0,,,,,2,2019-07-23T10:59:58Z,2019-07-23T13:07:01Z,2019-07-23T13:07:01Z,OWNER,,"While building https://github.com/dogsheep/healthkit-to-sqlite I found a need for a neat mechanism for easily building lookup tables - tables where each unique value in a column is replaced by a foreign key to a separate table. csvs-to-sqlite currently creates those with its ""extract"" mechanism - but that's written as custom code against Pandas. I'd like to eventually replace Pandas with sqlite-utils there. See also #42 ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/44/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 471684708,MDExOlB1bGxSZXF1ZXN0MzAwMjg2NTM1,45,"Implemented table.lookup(...), closes #44",9599,simonw,closed,0,,,,,0,2019-07-23T13:03:30Z,2019-07-23T13:07:00Z,2019-07-23T13:07:00Z,OWNER,simonw/sqlite-utils/pulls/45,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/45/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 351845423,MDU6SXNzdWUzNTE4NDU0MjM=,3,Experiment with contentless FTS tables,9599,simonw,closed,0,,,,,1,2018-08-18T19:31:01Z,2019-07-22T20:58:55Z,2019-07-22T20:58:55Z,OWNER,,Could greatly reduce size of resulting database for large datasets: http://cocoamine.net/blog/2015/09/07/contentless-fts4-for-large-immutable-documents/,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/3/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 470131537,MDU6SXNzdWU0NzAxMzE1Mzc=,41,sqlite-utils insert --tsv option,9599,simonw,closed,0,,,,,0,2019-07-19T04:27:21Z,2019-07-19T04:50:47Z,2019-07-19T04:50:47Z,OWNER,,"Right now we only support ingesting CSV, but sometimes interesting data is released as TSV. https://www.washingtonpost.com/national/2019/07/18/how-download-use-dea-pain-pills-database/ for example.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/41/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 467928674,MDExOlB1bGxSZXF1ZXN0Mjk3NDU5Nzk3,40,.get() method plus support for compound primary keys,9599,simonw,closed,0,,,,,1,2019-07-15T03:43:13Z,2019-07-15T04:28:57Z,2019-07-15T04:28:52Z,OWNER,simonw/sqlite-utils/pulls/40,"- [x] Tests for the `NotFoundError` exception - [x] Documentation for `.get()` method - [x] Support `--pk` multiple times to define CLI compound primary keys - [x] Documentation for compound primary keys",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/40/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 467864071,MDU6SXNzdWU0Njc4NjQwNzE=,39,table.get(...) method,9599,simonw,closed,0,,,,,0,2019-07-14T17:20:51Z,2019-07-15T04:28:53Z,2019-07-15T04:28:53Z,OWNER,,"Utility method for fetching a record by its primary key. Accepts a single value (for primary key / rowid tables) or a list/tuple of values (for compound primary keys, refs #36). Raises a `NotFoundError` if the record cannot be found.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/39/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 462817589,MDU6SXNzdWU0NjI4MTc1ODk=,36,Support compound primary keys,9599,simonw,closed,0,,,,,0,2019-07-01T17:00:07Z,2019-07-15T04:28:52Z,2019-07-15T04:28:52Z,OWNER,,"This should work: ```python table = db[""dog_breeds""].insert({ ""dog_id"": 1, ""breed_id"": 2 }, pk=(""dog_id"", ""breed_id"")) ``` Needed for m2m work in #23",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/36/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 455996809,MDU6SXNzdWU0NTU5OTY4MDk=,28,"Rearrange the docs by area, not CLI vs Python",9599,simonw,closed,0,,,,,1,2019-06-13T23:33:35Z,2019-07-15T02:37:20Z,2019-07-15T02:37:20Z,OWNER,,"The docs for eg inserting data should live on the same page, rather than being split across the API and CLI pages.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/28/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 462423839,MDU6SXNzdWU0NjI0MjM4Mzk=,33,index_foreign_keys / index-foreign-keys utilities,9599,simonw,closed,0,,,,,2,2019-06-30T16:42:03Z,2019-06-30T23:54:11Z,2019-06-30T23:50:55Z,OWNER,,"Sometimes it's good to have indices on all columns that are foreign keys, to allow for efficient reverse lookups. This would be a useful utility: $ sqlite-utils index-foreign-keys database.db ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/33/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 462423972,MDExOlB1bGxSZXF1ZXN0MjkzMTE3MTgz,34,sqlite-utils index-foreign-keys / db.index_foreign_keys(),9599,simonw,closed,0,,,,,0,2019-06-30T16:43:40Z,2019-06-30T23:50:55Z,2019-06-30T23:50:55Z,OWNER,simonw/sqlite-utils/pulls/34,"Refs #33 - [x] `sqlite-utils index-foreign-keys` command - [x] `db.index_foreign_keys()` method - [x] unit tests - [x] documentation",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/34/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 461237618,MDU6SXNzdWU0NjEyMzc2MTg=,31,Mechanism for adding multiple foreign key constraints at once,9599,simonw,closed,0,,,,,0,2019-06-27T00:04:30Z,2019-06-29T06:27:40Z,2019-06-29T06:27:40Z,OWNER,,"Needed by [db-to-sqlite](https://github.com/simonw/db-to-sqlite). It currently works by collecting all of the foreign key relationships it can find and then applying them at the end of the process. The problem is, the `add_foreign_key()` method looks like this: https://github.com/simonw/sqlite-utils/blob/86bd2bba689e25f09551d611ccfbee1e069e5b66/sqlite_utils/db.py#L498-L516 That means it's doing a full `VACUUM` for every single relationship it sets up - and if you have hundreds of foreign key relationships in your database this can take hours. I think the right solution is to have a `.add_foreign_keys(list_of_args)` method which does the bulk operation and then a single `VACUUM`. `.add_foreign_key(...)` can then call the bulk action with a single list item.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/31/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 462094937,MDExOlB1bGxSZXF1ZXN0MjkyODc5MjA0,32,db.add_foreign_keys() method,9599,simonw,closed,0,,,,,1,2019-06-28T15:40:33Z,2019-06-29T06:27:39Z,2019-06-29T06:27:39Z,OWNER,simonw/sqlite-utils/pulls/32,"Refs #31. Still TODO: - [x] Unit tests - [x] Documentation",140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/32/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 458941203,MDU6SXNzdWU0NTg5NDEyMDM=,29,Prevent accidental add-foreign-key with invalid column,9599,simonw,closed,0,,,,,0,2019-06-20T23:57:24Z,2019-06-20T23:58:26Z,2019-06-20T23:58:26Z,OWNER,,"You can corrupt your database by running: $ sqlite-utils add-foreign-key my.db table non_existent_column other_table other_column ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/29/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 449848803,MDU6SXNzdWU0NDk4NDg4MDM=,25,"Allow .insert(..., foreign_keys=()) to auto-detect table and primary key",9599,simonw,closed,0,,,,,4,2019-05-29T14:39:22Z,2019-06-13T05:32:32Z,2019-06-13T05:32:32Z,OWNER,,"The `foreign_keys=` argument currently takes a list of triples: ```python db[""usages""].insert_all( usages_to_insert, foreign_keys=( (""line_id"", ""lines"", ""id""), (""definition_id"", ""definitions"", ""id""), ), ) ``` As of #16 we have a mechanism for detecting the primary key column (the third item in this triple) - we should use that here too, so foreign keys can be optionally defined as a list of pairs.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/25/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 413867537,MDU6SXNzdWU0MTM4Njc1Mzc=,16,add_column() should support REFERENCES {other_table}({other_column}),9599,simonw,closed,0,,,,,4,2019-02-24T21:00:45Z,2019-05-29T05:17:59Z,2019-05-29T04:56:18Z,OWNER,,Related to #2 ,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/16/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 448391492,MDU6SXNzdWU0NDgzOTE0OTI=,21,Option to ignore inserts if primary key exists already,9599,simonw,closed,0,,,,,3,2019-05-25T00:17:12Z,2019-05-29T05:09:01Z,2019-05-29T04:18:26Z,OWNER,,"> I've just noticed that SQLite lets you IGNORE inserts that collide with a pre-existing key. This can be quite handy if you have a dataset that keeps changing in part, and you don't want to upsert and replace pre-existing PK rows but you do want to ignore collisions to existing PK rows. > > Do `sqlite_utils` support such (cavalier!) behaviour? _Originally posted by @psychemedia in https://github.com/simonw/sqlite-utils/issues/18#issuecomment-480621924_",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/21/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 448395665,MDU6SXNzdWU0NDgzOTU2NjU=,22,Release notes for 1.0,9599,simonw,closed,0,,,4348046,1.0,2,2019-05-25T00:58:03Z,2019-05-25T01:18:27Z,2019-05-25T01:06:52Z,OWNER,,https://github.com/simonw/sqlite-utils/compare/0.14...251e473,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/22/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 432217625,MDU6SXNzdWU0MzIyMTc2MjU=,19,Incorrect help text for enable-fts command,9599,simonw,closed,0,,,4348046,1.0,0,2019-04-11T19:46:44Z,2019-05-25T00:44:31Z,2019-05-25T00:44:31Z,OWNER,,"I clearly copied-and-pasted this from the `tables` command without updating it: https://github.com/simonw/sqlite-utils/blob/0b1af42ead3b3902347951180b3364ce1942da6e/sqlite_utils/cli.py#L216-L222",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/19/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 413871266,MDU6SXNzdWU0MTM4NzEyNjY=,18,.insert/.upsert/.insert_all/.upsert_all should add missing columns,9599,simonw,closed,0,,,4348046,1.0,2,2019-02-24T21:36:11Z,2019-05-25T00:42:11Z,2019-05-25T00:42:11Z,OWNER,,"This is a larger change, but it would be incredibly useful: if you attempt to insert or update a document with a field that does not currently exist in the underlying table, sqlite-utils should add the appropriate column for you.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/18/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 413868452,MDU6SXNzdWU0MTM4Njg0NTI=,17,Improve and document foreign_keys=... argument to insert/create/etc,9599,simonw,closed,0,,,,,7,2019-02-24T21:09:11Z,2019-02-24T23:45:48Z,2019-02-24T23:45:48Z,OWNER,,"The `foreign_keys=` argument to `table.insert_all()` and friends can be used to specify foreign key relationships that should be created. It is not yet documented. It also requires you to specify the SQLite type of each column, even though this can be detected by introspecting the referenced table: cols = [c for c in self.db[other_table].columns if c.name == other_column] cols[0].type Relates to #2 ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/17/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 349850687,MDU6SXNzdWUzNDk4NTA2ODc=,2,Mechanism for adding foreign keys to an existing table,9599,simonw,closed,0,,,,,1,2018-08-12T22:50:56Z,2019-02-24T21:34:41Z,2019-02-24T21:34:41Z,OWNER,,"SQLite does not have ALTER TABLE support for adding new foreign keys... but it turns out it's possible to make these changes without having to duplicate the entire table by carefully running `UPDATE sqlite_master SET sql=... WHERE type='table' AND name='X';` Here's how Django does it: https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/db/backends/sqlite3/schema.py#L103-L125 And here's the official documentation about this: https://sqlite.org/lang_altertable.html#otheralter (scroll to the very bottom of the page)",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/2/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 413857257,MDU6SXNzdWU0MTM4NTcyNTc=,15,Ability to add columns to tables,9599,simonw,closed,0,,,,,0,2019-02-24T19:20:51Z,2019-02-24T20:04:40Z,2019-02-24T20:04:40Z,OWNER,,"Makes sense to do this before foreign keys in #2 Python: db[""table""].add_column(""new_column"", int) CLI: $ sqlite-utils add-column table new_column INTEGER ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/15/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 413842611,MDU6SXNzdWU0MTM4NDI2MTE=,14,Utilities for adding indexes,9599,simonw,closed,0,,,,,3,2019-02-24T16:57:28Z,2019-02-24T19:11:28Z,2019-02-24T19:11:28Z,OWNER,,"Both in the Python API and the CLI tool. For the CLI tool this should work: $ sqlite-utils create-index mydb.db mytable col1 col2 This will create a compound index across col1 and col2. The name of the index will be automatically chosen unless you use the `--name=...` option. Support a `--unique` option too.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/14/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 413779210,MDU6SXNzdWU0MTM3NzkyMTA=,13,Ability to automatically create IDs from content hash of row,9599,simonw,closed,0,,,,,1,2019-02-24T04:07:08Z,2019-02-24T04:36:48Z,2019-02-24T04:36:48Z,OWNER,,"Sometimes when you are importing data the underlying source provides records without IDs that can be uniquely identified by their contents. A utility mechanism for calculating a sha1 hash of the contents and using that as a unique ID would be useful.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/13/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 413740684,MDU6SXNzdWU0MTM3NDA2ODQ=,11,Detect numpy types when creating tables,9599,simonw,closed,0,,,,,2,2019-02-23T21:09:35Z,2019-02-24T04:02:20Z,2019-02-24T04:02:20Z,OWNER,,Inspired by #8,140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/11/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 413778585,MDExOlB1bGxSZXF1ZXN0MjU1NjU4MTEy,12,"Support for numpy types, closes #11",9599,simonw,closed,0,,,,,0,2019-02-24T03:57:32Z,2019-02-24T04:02:20Z,2019-02-24T04:02:20Z,OWNER,simonw/sqlite-utils/pulls/12,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/12/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 403625674,MDU6SXNzdWU0MDM2MjU2NzQ=,7,.insert_all() should accept a generator and process it efficiently,9599,simonw,closed,0,,,,,3,2019-01-28T02:11:58Z,2019-01-28T06:26:53Z,2019-01-28T06:26:53Z,OWNER,,"Right now you have to load every record into memory before passing the list to `.insert_all()` and friends. If you want to process millions of rows, this is inefficient. Python has generators - we should use them! The only catch here is that part of the magic of `sqlite-utils` is that it guesses the column types and creates the table for you. This code will need to be updated to notice if the table needs creating and, if it does, create it using the first X (where x=1,000 but can be customized) records. If a record outside of those first 1,000 has a rogue column, we can crash with an error. This will free us up to make the `--nl` option added in #6 much more efficient.",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/7/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 403624090,MDU6SXNzdWU0MDM2MjQwOTA=,6,"""sqlite-utils insert"" should support newline-delimited JSON",9599,simonw,closed,0,,,,,1,2019-01-28T02:00:02Z,2019-01-28T02:17:45Z,2019-01-28T02:17:45Z,OWNER,,"We can already export newline delimited JSON. We should learn to import it as well. The neat thing about importing it is that you can import GBs of data without having to read the whole lot into memory in order to decode the wrapping JSON array. Datasette can export it now: https://github.com/simonw/datasette/issues/405 Demo: https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on It should be possible to do this: $ curl ""https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on"" \ | sqlite-utils insert data.db facetable - --nl ",140912432,sqlite-utils,issue,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/6/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed 403396009,MDExOlB1bGxSZXF1ZXN0MjQ3ODYxNDE5,5,Run Travis tests against Python 3.8-dev,9599,simonw,closed,0,,,,,0,2019-01-26T02:30:55Z,2019-01-26T02:37:54Z,2019-01-26T02:37:54Z,OWNER,simonw/sqlite-utils/pulls/5,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/5/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 403028630,MDExOlB1bGxSZXF1ZXN0MjQ3NTc2OTQy,4,Fts5,9599,simonw,closed,0,,,,,0,2019-01-25T06:54:05Z,2019-01-25T06:54:33Z,2019-01-25T06:54:33Z,OWNER,simonw/sqlite-utils/pulls/4,,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/4/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0, 347058326,MDExOlB1bGxSZXF1ZXN0MjA1NzcwOTk2,1,Make .indexes compatible with older SQLite versions,9599,simonw,closed,0,,,,,0,2018-08-02T15:17:05Z,2018-08-02T15:17:30Z,2018-08-02T15:17:30Z,OWNER,simonw/sqlite-utils/pulls/1,Older SQLite versions return a different set of columns from the PRAGMA we are using.,140912432,sqlite-utils,pull,,,"{""url"": ""https://api.github.com/repos/simonw/sqlite-utils/issues/1/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",0,