{"id": 652961907, "node_id": "MDU6SXNzdWU2NTI5NjE5MDc=", "number": 121, "title": "Improved (and better documented) support for transactions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-08T04:56:51Z", "updated_at": "2020-09-24T20:36:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_\r\n\r\nWe should put some thought into how this library supports and encourages smart use of transactions.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/121/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 665700495, "node_id": "MDU6SXNzdWU2NjU3MDA0OTU=", "number": 122, "title": "CLI utility for inserting binary files into SQLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-07-26T03:27:39Z", "updated_at": "2020-07-27T07:10:41Z", "closed_at": "2020-07-27T07:09:03Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite BLOB columns can store entire binary files. The challenge is inserting them, since they don't neatly fit into JSON objects.\r\n\r\nIt would be great if the `sqlite-utils` CLI had a trick for helping with this.\r\n\r\nInspired by https://github.com/simonw/datasette-media/issues/14", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/122/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665701216, "node_id": "MDU6SXNzdWU2NjU3MDEyMTY=", "number": 123, "title": "--raw option for outputting binary content", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-07-26T03:35:39Z", "updated_at": "2020-07-26T16:44:11Z", "closed_at": "2020-07-26T16:44:11Z", "author_association": "OWNER", "pull_request": null, "body": "Related to the `insert-files` work in #122 - it should be easy to get binary data back out of the database again.\r\n\r\nOne way to do that could be:\r\n\r\n sqlite-utils files.db \"select content from files where key = 'foo.jpg'\" --raw\r\n\r\nThe `--raw` option would cause just the contents of the first column to be output directly to stdout.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/123/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665802405, "node_id": "MDU6SXNzdWU2NjU4MDI0MDU=", "number": 124, "title": "sqlite-utils query should support named parameters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-26T15:25:10Z", "updated_at": "2020-07-30T22:57:51Z", "closed_at": "2020-07-27T03:53:58Z", "author_association": "OWNER", "pull_request": null, "body": "To help out with escaping - so you can run this:\r\n\r\n sqlite-utils query \"insert into foo (blah) values (:blah)\" --param blah `something here`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665817570, "node_id": "MDU6SXNzdWU2NjU4MTc1NzA=", "number": 125, "title": "Output binary columns in \"sqlite-utils query\" JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-07-26T16:47:02Z", "updated_at": "2020-07-27T00:49:41Z", "closed_at": "2020-07-27T00:48:45Z", "author_association": "OWNER", "pull_request": null, "body": "You get an error if you try to run a query that returns data from a BLOB.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/125/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665819048, "node_id": "MDU6SXNzdWU2NjU4MTkwNDg=", "number": 126, "title": "Ability to insert binary data on the CLI using JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-26T16:54:14Z", "updated_at": "2020-07-27T04:00:33Z", "closed_at": "2020-07-27T03:59:45Z", "author_association": "OWNER", "pull_request": null, "body": "> I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says \"these columns are base64 encoded, store the decoded data in a BLOB\".\r\n>\r\n> That would solve inserting binary data using JSON too.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 666040390, "node_id": "MDU6SXNzdWU2NjYwNDAzOTA=", "number": 127, "title": "Ability to insert files piped to insert-files stdin", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-27T07:09:33Z", "updated_at": "2020-07-30T03:08:52Z", "closed_at": "2020-07-30T03:08:18Z", "author_association": "OWNER", "pull_request": null, "body": "> Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.\r\n>\r\n> cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif\r\n>\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/127/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 666639051, "node_id": "MDU6SXNzdWU2NjY2MzkwNTE=", "number": 128, "title": "Support UUID and memoryview types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-27T23:08:34Z", "updated_at": "2020-07-30T01:10:43Z", "closed_at": "2020-07-30T01:10:43Z", "author_association": "OWNER", "pull_request": null, "body": "`psycopg2` can return data from PostgreSQL as `uuid.UUID` or `memoryview` objects. These should to be supported by `sqlite-utils` - mainly for https://github.com/simonw/db-to-sqlite", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/128/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 668308777, "node_id": "MDU6SXNzdWU2NjgzMDg3Nzc=", "number": 129, "title": "\"insert-files --sqlar\" for creating SQLite archives", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-30T02:28:29Z", "updated_at": "2020-07-30T22:41:01Z", "closed_at": "2020-07-30T22:40:55Z", "author_association": "OWNER", "pull_request": null, "body": "A `--sqlar` option could cause `insert-files` to behave in the same way as SQLite's own sqlar mechanism.\r\n\r\nhttps://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 671130371, "node_id": "MDU6SXNzdWU2NzExMzAzNzE=", "number": 130, "title": "Support tokenize option for FTS", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-01T19:27:22Z", "updated_at": "2020-08-01T20:51:28Z", "closed_at": "2020-08-01T20:51:14Z", "author_association": "OWNER", "pull_request": null, "body": "FTS5 supports things like porter stemming using a `tokenize=` option:\r\n\r\nhttps://www.sqlite.org/fts5.html#tokenizers\r\n\r\nSomething like this in code:\r\n```\r\n CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} (\r\n {columns},\r\n tokenize='porter',\r\n content=[{table}]\r\n );\r\n```\r\nI tried this out just now and it worked exactly as expected.\r\n\r\nSo... `db[table].enable_fts(...) should accept a 'tokenize=` argument, and `sqlite-utils enable-fts ...` should support a `--tokenize` option.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/130/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 675753042, "node_id": "MDU6SXNzdWU2NzU3NTMwNDI=", "number": 131, "title": "sqlite-utils insert: options for column types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-09T18:59:11Z", "updated_at": "2022-03-15T13:21:42Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs.\r\n\r\nIt would be useful if you could do the following:\r\n\r\n- automatically detects the column types based on eg the first 1000 records\r\n- explicitly state the rule for specific columns\r\n\r\n`--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types`\r\n\r\nFor specific columns maybe this:\r\n\r\n sqlite-utils insert db.db images images.tsv \\\r\n --tsv \\\r\n -c id int \\\r\n -c score float", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 675839512, "node_id": "MDU6SXNzdWU2NzU4Mzk1MTI=", "number": 132, "title": "Features for enabling and disabling WAL mode", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-10T03:25:44Z", "updated_at": "2020-08-10T18:59:35Z", "closed_at": "2020-08-10T18:59:35Z", "author_association": "OWNER", "pull_request": null, "body": "I finally figured out how to enable WAL - turns out it's a property of the database file itself: https://github.com/simonw/til/blob/master/sqlite/enabling-wal-mode.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/132/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 677839979, "node_id": "MDU6SXNzdWU2Nzc4Mzk5Nzk=", "number": 133, "title": "Release a sdist to PyPI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-12T16:55:09Z", "updated_at": "2020-08-12T17:05:06Z", "closed_at": "2020-08-12T17:05:06Z", "author_association": "OWNER", "pull_request": null, "body": "https://pypi.org/project/sqlite-utils/#files currently just has a wheel. I need this to package for homebrew: https://github.com/simonw/homebrew-datasette/issues/10", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/133/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683804172, "node_id": "MDU6SXNzdWU2ODM4MDQxNzI=", "number": 134, "title": "--load-extension option for sqlite-utils query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-08-21T20:12:42Z", "updated_at": "2020-08-21T21:06:26Z", "closed_at": "2020-08-21T20:54:19Z", "author_association": "OWNER", "pull_request": null, "body": "I got this error:\r\n```\r\n% sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view'\r\nTraceback (most recent call last):\r\n...\r\n cursor = db.conn.execute(sql, dict(param))\r\nsqlite3.OperationalError: no such function: AsGeoJSON\r\n```\r\nA `--load-extension=/usr/local/lib/mod_spatialite.dylib` option (imitating the same option for Datasette) would help.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/134/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683805434, "node_id": "MDU6SXNzdWU2ODM4MDU0MzQ=", "number": 135, "title": "Code for finding SpatiaLite in the usual locations", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-21T20:15:34Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-08-21T20:30:13Z", "author_association": "OWNER", "pull_request": null, "body": "I built this for `shapefile-to-sqlite` but it would be useful in `sqlite-utils` too:\r\n\r\nhttps://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19\r\n\r\n```python\r\nSPATIALITE_PATHS = (\r\n \"/usr/lib/x86_64-linux-gnu/mod_spatialite.so\",\r\n \"/usr/local/lib/mod_spatialite.dylib\",\r\n)\r\n```\r\n\r\nhttps://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109\r\n\r\n```python\r\ndef find_spatialite():\r\n for path in SPATIALITE_PATHS:\r\n if os.path.exists(path):\r\n return path\r\n return None\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683812642, "node_id": "MDU6SXNzdWU2ODM4MTI2NDI=", "number": 136, "title": "--load-extension=spatialite shortcut option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-21T20:31:25Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-16T19:14:32Z", "author_association": "OWNER", "pull_request": null, "body": "In conjunction with #135 - this would do the same thing as `--load-extension=path-to-spatialite` (see #134)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683830416, "node_id": "MDU6SXNzdWU2ODM4MzA0MTY=", "number": 137, "title": "--load-extension for other sqlite-utils commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-21T21:12:56Z", "updated_at": "2020-10-16T19:14:32Z", "closed_at": "2020-10-16T19:14:32Z", "author_association": "OWNER", "pull_request": null, "body": "e.g. for this:\r\n```\r\ncalands-datasette % sqlite-utils tables calands.db --counts\r\n[{\"table\": \"spatial_ref_sys\", \"count\": 4924},\r\n {\"table\": \"spatialite_history\", \"count\": 14},\r\n {\"table\": \"sqlite_sequence\", \"count\": 1},\r\n {\"table\": \"geometry_columns\", \"count\": 2},\r\n {\"table\": \"spatial_ref_sys_aux\", \"count\": 4873},\r\n {\"table\": \"views_geometry_columns\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns\", \"count\": 0},\r\n {\"table\": \"geometry_columns_statistics\", \"count\": 2},\r\n {\"table\": \"views_geometry_columns_statistics\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_statistics\", \"count\": 0},\r\n {\"table\": \"geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"views_geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"geometry_columns_time\", \"count\": 2},\r\n {\"table\": \"geometry_columns_auth\", \"count\": 2},\r\n {\"table\": \"views_geometry_columns_auth\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_auth\", \"count\": 0},\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 143, in tables\r\n for line in output_rows(_iter(), headers, nl, arrays, json_cols):\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 922, in output_rows\r\n for row, next_row in itertools.zip_longest(current_iter, next_iter):\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 123, in _iter\r\n row.append(db[name].count)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/db.py\", line 458, in count\r\n return self.db.conn.execute(\r\nsqlite3.OperationalError: no such module: VirtualSpatialIndex\r\n```\r\nThe `tables` command could take `--load-extension` too - as could `rows` and other similar commands.\r\n\r\nFollow-on from #134 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/137/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 684118950, "node_id": "MDU6SXNzdWU2ODQxMTg5NTA=", "number": 138, "title": "extracts= doesn't configure foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-23T05:21:15Z", "updated_at": "2020-09-24T22:47:01Z", "closed_at": "2020-09-24T22:46:52Z", "author_association": "OWNER", "pull_request": null, "body": "In using `extracts=` for `shapefiles-to-sqlite` in https://github.com/simonw/shapefile-to-sqlite/issues/9 I've run into a couple of pretty serious flaws:\r\n\r\n- The columns in the original table are still `TEXT` even when the foreign key they are supposed to reference is an `INTEGER` - which means Datasette foreign key features don't actually work\r\n- Those foreign key relationships aren't setup automatically - creating them is left as an exercise for the developer", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/138/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688351054, "node_id": "MDU6SXNzdWU2ODgzNTEwNTQ=", "number": 140, "title": "Idea: insert-files mechanism for adding extra columns with fixed values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T20:57:36Z", "updated_at": "2022-03-20T19:45:45Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Say for example you want to populate a `file_type` column with the value `gif`. That could work like this:\r\n\r\n```\r\nsqlite-utils insert-files gifs.db images *.gif \\\r\n -c path -c md5 -c last_modified:mtime \\\r\n -c file_type:text:gif --pk=path\r\n```\r\nSo a column defined as a `text` column with a value that follows a second colon.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 688352145, "node_id": "MDU6SXNzdWU2ODgzNTIxNDU=", "number": 141, "title": "insert-files support for compressed values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-08-28T20:59:46Z", "updated_at": "2020-09-24T20:36:08Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `sqlar` format supports this, it would be useful if `insert-files` could support this too.\r\n\r\nhttps://www.sqlite.org/sqlar.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/141/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 688389933, "node_id": "MDU6SXNzdWU2ODgzODk5MzM=", "number": 143, "title": "Move to GitHub Actions CI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T22:34:11Z", "updated_at": "2020-08-28T22:41:35Z", "closed_at": "2020-08-28T22:41:35Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/143/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688395275, "node_id": "MDU6SXNzdWU2ODgzOTUyNzU=", "number": 144, "title": "Run some tests against numpy", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-28T22:53:00Z", "updated_at": "2020-08-28T22:57:05Z", "closed_at": "2020-08-28T22:57:04Z", "author_association": "OWNER", "pull_request": null, "body": "Accidentally removed in #143:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/d7d3f962861ef32c5ead8f514c8756f5b6f7c4a0/.travis.yml#L18-L19", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695276328, "node_id": "MDU6SXNzdWU2OTUyNzYzMjg=", "number": 148, "title": "More attractive indentation of created FTS table schema", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-07T16:49:30Z", "updated_at": "2020-09-07T18:12:50Z", "closed_at": "2020-09-07T18:12:50Z", "author_association": "OWNER", "pull_request": null, "body": "On https://github-to-sqlite.dogsheep.net/github/licenses_fts the create table SQL is displayed as:\r\n```sql\r\nCREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (\r\n [name],\r\n content=[licenses]\r\n );\r\n```\r\nIt would be more aesthetically pleasing if it looked like this:\r\n```sql\r\nCREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (\r\n [name],\r\n content=[licenses]\r\n);\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/148/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695319258, "node_id": "MDU6SXNzdWU2OTUzMTkyNTg=", "number": 149, "title": "FTS table with 7 rows has _fts_docsize table with 9,141 rows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-09-07T18:06:16Z", "updated_at": "2020-09-07T21:16:34Z", "closed_at": "2020-09-07T21:16:34Z", "author_association": "OWNER", "pull_request": null, "body": "I'm seeing a weird issue with some of the SQLite databases that I am using with the FTS5 module.\r\n\r\nI have a database with a `licenses` table that contains 7 rows: \r\n\r\nThe FTS table also has 7 rows: \r\n\r\nSomehow the accompanying `licenses_fts_docsize` shadow table now has 9,141 rows in it! \r\n\r\nAnd `licenses_fts_data` has 41 rows - should I expect that to have 7 rows? \r\n\r\nI have a hunch that it might be a problem with the triggers. These are the triggers that are updating that FTS table: \r\n\r\n| type | name | tbl_name | rootpage | sql |\r\n| --- | --- | --- | --- | --- |\r\n| trigger | licenses_ai | licenses | 0 | `CREATE TRIGGER [licenses_ai] AFTER INSERT ON [licenses] BEGIN INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |\r\n| trigger | licenses_ad | licenses | 0 | `CREATE TRIGGER [licenses_ad] AFTER DELETE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); END` |\r\n| trigger | licenses_au | licenses | 0 | `CREATE TRIGGER [licenses_au] AFTER UPDATE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/149/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695359607, "node_id": "MDU6SXNzdWU2OTUzNTk2MDc=", "number": 150, "title": "Feature for tracing SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-07T19:43:08Z", "updated_at": "2020-09-07T21:57:01Z", "closed_at": "2020-09-07T21:57:01Z", "author_association": "OWNER", "pull_request": null, "body": "Debugging `sqlite-utils` when something weird happens (e.g. #149) can be a bit tricky since it runs a bunch of different SQL statements behind the scenes.\r\n\r\nAn optional \"tracing\" mechanism for seeing what SQL is being executed would be useful.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/150/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695360889, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgxNjE2NzA0", "number": 151, "title": "Tracer mechanism for seeing underlying SQL", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-07T19:46:43Z", "updated_at": "2020-09-07T21:57:00Z", "closed_at": "2020-09-07T21:57:00Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/151", "body": "Refs #150. Needs tests and documentation, including for the new `db.execute()` and `db.executescript()` methods.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/151/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 695376054, "node_id": "MDU6SXNzdWU2OTUzNzYwNTQ=", "number": 152, "title": "Turn on recursive_triggers by default", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T20:26:36Z", "updated_at": "2020-09-07T21:17:48Z", "closed_at": "2020-09-07T20:45:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:\r\n\r\n> Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.\r\n\r\nSo I think the fix for the complex issue in #149 is to turn on `recursive_triggers` globally by default for `sqlite-utils`.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/152/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695377804, "node_id": "MDU6SXNzdWU2OTUzNzc4MDQ=", "number": 153, "title": "table.optimize() should delete junk rows from *_fts_docsize", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-09-07T20:31:09Z", "updated_at": "2020-09-24T20:35:46Z", "closed_at": "2020-09-07T21:16:33Z", "author_association": "OWNER", "pull_request": null, "body": "> The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL:\r\n> ```sql\r\n> DELETE FROM [licenses_fts_docsize] WHERE id NOT IN (\r\n> SELECT rowid FROM [licenses_fts]);\r\n> ```\r\n> I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/153/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695441530, "node_id": "MDU6SXNzdWU2OTU0NDE1MzA=", "number": 154, "title": "OperationalError: cannot change into wal mode from within a transaction", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T23:42:44Z", "updated_at": "2020-09-07T23:47:10Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm getting this error when running:\r\n\r\n sqlite-utils enable-wal beta.db\r\n\r\n`OperationalError: cannot change into wal mode from within a transaction`\r\n\r\nI'm worried that maybe that's because of this new code from #152:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 696045581, "node_id": "MDU6SXNzdWU2OTYwNDU1ODE=", "number": 155, "title": "rebuild-fts command and table.rebuild_fts() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-08T17:19:26Z", "updated_at": "2020-09-24T20:35:46Z", "closed_at": "2020-09-08T23:16:10Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite.org/forum/forumpost/fa777fff86\r\n\r\n> Easiest thing would be to run a 'rebuild' to rebuild the FTS index from scratch based on the contents of the content table. i.e.\r\n>\r\n> INSERT INTO licenses_fts(licenses_fts) VALUES('rebuild');\r\n>\r\n> https://www.sqlite.org/fts5.html#the_rebuild_command", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/155/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 697179806, "node_id": "MDU6SXNzdWU2OTcxNzk4MDY=", "number": 157, "title": "sqlite-utils add-foreign-keys command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 2, "created_at": "2020-09-09T21:44:30Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-20T20:14:30Z", "author_association": "OWNER", "pull_request": null, "body": "Like `add-foreign-key` but can do multiple foreign keys at once. Inspired by https://github.com/simonw/calands-datasette/blob/99de39dd80a906f5c1f16724467b0cd55ba4ef36/build.sh which does this:\r\n\r\n```\r\nsqlite-utils add-foreign-key calands.db units_with_maps ACCESS_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_NAME\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_LEV\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps LAYER\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AGENCY\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_LEV\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps COUNTY\r\nsqlite-utils add-foreign-key calands.db units_with_maps DES_TP\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/157/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 705190723, "node_id": "MDU6SXNzdWU3MDUxOTA3MjM=", "number": 160, "title": "table.enable_fts(..., replace=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 1, "created_at": "2020-09-20T21:36:23Z", "updated_at": "2020-09-24T20:35:47Z", "closed_at": "2020-09-20T22:05:51Z", "author_association": "OWNER", "pull_request": null, "body": "I noticed that https://til.simonwillison.net/ search doesn't use porter stemming. I'd like to add that, but since [the build script](https://github.com/simonw/til/blob/9d3f0fca30e94df3970df52b0447907a077e4673/build_database.py) always operates on an existing database (to avoid re-rendering markdown and re-building image thumbnails) I'd like it to only add porter stemming if it's not there already.\r\n\r\nSo I'd like to be able to say \"set up FTS to look like this, and fix it if it doesn't\".\r\n\r\nI think the neatest way to do that is with a `replace=True` argument to `.enable_fts()`, for consistency with `def .create_view(self, name, sql, replace=True)`.\r\n\r\nSo the `replace=True` argument would check and see if the configured FTS exists already with the correct options (columns, stemming, triggers) - and if any of those are incorrect it would call `.disable_fts()` and then create a new FTS configuration with the correct options.\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/160/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 705975133, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkwNjA3OTQ5", "number": 161, "title": "table.transform() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 13, "created_at": "2020-09-21T23:16:59Z", "updated_at": "2020-09-22T07:48:24Z", "closed_at": "2020-09-22T04:20:02Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/161", "body": "Refs #114\r\n\r\n- [x] Ability to change the primary key\r\n- [x] Support for changing default value for columns\r\n- [x] Support for changing `NOT NULL` status of columns\r\n- [x] Support for copying existing foreign keys and removing them\r\n- Support for `conversions=` parameter\r\n- [x] Detailed documentation\r\n- [x] `PRAGMA foreign_keys` stuff", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/161/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 705995722, "node_id": "MDU6SXNzdWU3MDU5OTU3MjI=", "number": 162, "title": "A decorator for registering custom SQL functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-22T00:18:32Z", "updated_at": "2020-09-22T00:40:44Z", "closed_at": "2020-09-22T00:32:17Z", "author_association": "OWNER", "pull_request": null, "body": "Syntactic sugar for `db.conn.create_function` - it would work something like this:\r\n\r\n```python\r\ndb = sqlite_utils.Database(\"mydb.db\")\r\n\r\n@db.register_function\r\ndef scramble(text):\r\n chars = list(text)\r\n random.shuffle(chars)\r\n return \"\".join(chars)\r\n```\r\nThe decorator would inspect the function to find its name and arity (number of arguments). Having run the above you could then do:\r\n```python\r\ndb.execute(\"select scramble('hello')\").fetchall()\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/162/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706001517, "node_id": "MDU6SXNzdWU3MDYwMDE1MTc=", "number": 163, "title": "Idea: conversions= could take Python functions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-22T00:37:12Z", "updated_at": "2021-12-20T00:56:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now you use `conversions=` like this:\r\n\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": \"upper(?)\"})\r\n```\r\nHow about if you could optionally provide a Python function (or a lambda) like this?\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": lambda s: s.upper()})\r\n```\r\nThis would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 706017416, "node_id": "MDU6SXNzdWU3MDYwMTc0MTY=", "number": 164, "title": "sqlite-utils transform sub-command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 4, "created_at": "2020-09-22T01:32:20Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-22T07:48:05Z", "author_association": "OWNER", "pull_request": null, "body": "The `.transform()` method in #114 warrants an equivalent CLI tool.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/164/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706091046, "node_id": "MDU6SXNzdWU3MDYwOTEwNDY=", "number": 165, "title": "Make .transform() a keyword arguments only function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 0, "created_at": "2020-09-22T05:37:29Z", "updated_at": "2020-09-24T20:35:47Z", "closed_at": "2020-09-22T06:39:12Z", "author_association": "OWNER", "pull_request": null, "body": "And rename the first argument from `columns=` to `types=`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/165/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706092617, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkwNzAzMTcz", "number": 166, "title": "Keyword only arguments for transform()", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-22T05:41:44Z", "updated_at": "2020-09-22T06:39:11Z", "closed_at": "2020-09-22T06:39:11Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/166", "body": "Refs #165", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/166/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 706098005, "node_id": "MDU6SXNzdWU3MDYwOTgwMDU=", "number": 167, "title": "Review the foreign key pragma stuff", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 1, "created_at": "2020-09-22T05:55:20Z", "updated_at": "2020-09-23T00:13:02Z", "closed_at": "2020-09-23T00:13:02Z", "author_association": "OWNER", "pull_request": null, "body": "> It is not possible to enable or disable foreign key constraints in the middle of a multi-statement transaction (when SQLite is not in autocommit mode). Attempting to do so does not return an error; it simply has no effect.\r\n\r\nhttps://sqlite.org/foreignkeys.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/167/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706167456, "node_id": "MDU6SXNzdWU3MDYxNjc0NTY=", "number": 168, "title": "Automate (as much as possible) updates published to Homebrew", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-22T08:08:37Z", "updated_at": "2020-11-09T07:43:30Z", "closed_at": "2020-11-09T07:43:30Z", "author_association": "OWNER", "pull_request": null, "body": "I'd like to get new `sqlite-utils` (and Datasette) releases submitted to Homebrew as painlessly as possible.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/168/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706757891, "node_id": "MDU6SXNzdWU3MDY3NTc4OTE=", "number": 169, "title": "Progress bar for \"sqlite-utils extract\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 0, "created_at": "2020-09-22T23:40:21Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-23T00:02:40Z", "author_association": "OWNER", "pull_request": null, "body": "> Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.\r\n>\r\n> The operations are full table scans so calculating progress shouldn't be too difficult.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/169/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706768798, "node_id": "MDU6SXNzdWU3MDY3Njg3OTg=", "number": 170, "title": "Release notes for 2.20", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 1, "created_at": "2020-09-23T00:13:22Z", "updated_at": "2020-09-23T00:31:25Z", "closed_at": "2020-09-23T00:31:25Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/compare/2.19...b8e004", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/170/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707427200, "node_id": "MDU6SXNzdWU3MDc0MjcyMDA=", "number": 172, "title": "Improve performance of extract operations", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-09-23T14:40:50Z", "updated_at": "2020-09-24T15:43:57Z", "closed_at": "2020-09-24T15:43:57Z", "author_association": "OWNER", "pull_request": null, "body": "This command took about 12 minutes (against a 150MB file with 680,000 rows in it):\r\n```\r\nsqlite-utils extract salaries.db salaries \\\r\n 'Organization Group Code' 'Organization Group' \\\r\n --table 'organization_groups' \\\r\n --fk-column 'organization_group_id' \\\r\n --rename 'Organization Group Code' code \\\r\n --rename 'Organization Group' name\r\n```\r\nI'm pretty confident we can do better than that.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/172/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707478649, "node_id": "MDU6SXNzdWU3MDc0Nzg2NDk=", "number": 173, "title": "Progress bar for sqlite-utils insert", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-09-23T15:43:56Z", "updated_at": "2021-11-01T08:42:24Z", "closed_at": "2020-10-27T18:16:04Z", "author_association": "OWNER", "pull_request": null, "body": "It would be nice if `sqlite-utils insert` had a progress bar, for when it's churning through huge CSV files.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/173/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707944044, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkyMjU3NDA1", "number": 174, "title": "Much, much faster extract() implementation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-09-24T07:52:31Z", "updated_at": "2020-09-24T15:44:00Z", "closed_at": "2020-09-24T15:43:56Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/174", "body": "Takes my test down from ten minutes to four seconds. Refs #172.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/174/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 708261775, "node_id": "MDU6SXNzdWU3MDgyNjE3NzU=", "number": 175, "title": "Add docs for .transform(column_order=)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-09-24T15:19:04Z", "updated_at": "2020-09-24T20:35:48Z", "closed_at": "2020-09-24T16:00:56Z", "author_association": "OWNER", "pull_request": null, "body": "> Need to update docs for `.transform()` now that `column_order=` is available.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/174#discussion_r494403327_\r\n\r\nMaybe also add this as an option to `sqlite-utils transform` - since reordering columns is actually a pretty nice capability.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/175/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 708293114, "node_id": "MDU6SXNzdWU3MDgyOTMxMTQ=", "number": 176, "title": "sqlite-utils transform column order option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-24T16:01:21Z", "updated_at": "2020-09-24T20:34:51Z", "closed_at": "2020-09-24T16:11:59Z", "author_association": "OWNER", "pull_request": null, "body": "Split from #175", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/176/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 708301810, "node_id": "MDU6SXNzdWU3MDgzMDE4MTA=", "number": 177, "title": "Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-24T16:13:50Z", "updated_at": "2020-09-24T20:35:03Z", "closed_at": "2020-09-24T16:19:13Z", "author_association": "OWNER", "pull_request": null, "body": "These both currently require you to provide three strings, for `column`, `other_table`, `other_column`.\r\n\r\nJust providing `column` should be enough information.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/177/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709577625, "node_id": "MDU6SXNzdWU3MDk1Nzc2MjU=", "number": 179, "title": "sqlite-utils transform/insert --detect-types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-26T17:28:55Z", "updated_at": "2021-06-19T03:36:16Z", "closed_at": "2021-06-19T03:36:05Z", "author_association": "OWNER", "pull_request": null, "body": "Idea from https://github.com/simonw/datasette-edit-tables/issues/13 - provide Python utility methods and accompanying CLI options for detecting the likely types of TEXT columns.\r\n\r\nSo if you have a text column that actually contained exclusively integer string values, it can let you know and let you run transform against it.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/179/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709861194, "node_id": "MDU6SXNzdWU3MDk4NjExOTQ=", "number": 180, "title": "Try running some tests using Hypothesis", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-28T01:11:30Z", "updated_at": "2020-10-19T04:51:55Z", "closed_at": "2020-10-19T04:51:55Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by this Twitter conversation: https://twitter.com/simonw/status/1310386009465479168", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/180/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709920027, "node_id": "MDU6SXNzdWU3MDk5MjAwMjc=", "number": 181, "title": "pk=[\"id\"] should have same effect as pk=\"id\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-28T04:28:07Z", "updated_at": "2020-10-14T21:59:47Z", "closed_at": "2020-10-14T21:59:47Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\nIn [11]: db['one'].insert({\"id\": 1, \"name\": \"oentuh\"}, pk=\"id\")\r\nOut[11]: \r\n\r\nIn [12]: db['two'].insert({\"id\": 1, \"name\": \"oentuh\"}, pk=[\"id\"])\r\nOut[12]:
\r\n\r\nIn [13]: db['one'].schema\r\nOut[13]: 'CREATE TABLE [one] (\\n [id] INTEGER PRIMARY KEY,\\n [name] TEXT\\n)'\r\n\r\nIn [14]: db['two'].schema\r\nOut[14]: 'CREATE TABLE [two] (\\n [id] INTEGER,\\n [name] TEXT\\n)'\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/181/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 712316959, "node_id": "MDU6SXNzdWU3MTIzMTY5NTk=", "number": 183, "title": "Try out GitHub code scanning", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-30T22:16:14Z", "updated_at": "2020-09-30T22:23:44Z", "closed_at": "2020-09-30T22:23:44Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.blog/2020-09-30-code-scanning-is-now-available/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/183/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 716955793, "node_id": "MDExOlB1bGxSZXF1ZXN0NDk5NjAzMzU5", "number": 184, "title": "Test against Python 3.9", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-08T01:37:05Z", "updated_at": "2020-10-08T01:44:06Z", "closed_at": "2020-10-08T01:44:06Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/184", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/184/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 718952107, "node_id": "MDU6SXNzdWU3MTg5NTIxMDc=", "number": 185, "title": "Use db[table] consistently in documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-11T23:39:12Z", "updated_at": "2020-10-12T00:13:41Z", "closed_at": "2020-10-12T00:13:41Z", "author_association": "OWNER", "pull_request": null, "body": "The Python docs have a bunch of examples like this: https://sqlite-utils.readthedocs.io/en/stable/python-api.html\r\n```python\r\ndogs.enable_fts([\"name\", \"twitter\"], create_triggers=True)\r\n```\r\nThis would be easier for people to understand if it looked like this instead:\r\n```python\r\ndb[\"dogs\"].enable_fts([\"name\", \"twitter\"], create_triggers=True)\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/185/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 722816436, "node_id": "MDU6SXNzdWU3MjI4MTY0MzY=", "number": 186, "title": ".extract() shouldn't extract null values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-10-16T02:41:08Z", "updated_at": "2021-08-12T12:32:14Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This almost works, but it creates a rogue `type` record with a value of None.\r\n```\r\nIn [1]: import sqlite_utils\r\nIn [2]: db = sqlite_utils.Database(memory=True)\r\nIn [5]: db[\"creatures\"].insert_all([\r\n {\"id\": 1, \"name\": \"Simon\", \"type\": None},\r\n {\"id\": 2, \"name\": \"Natalie\", \"type\": None},\r\n {\"id\": 3, \"name\": \"Cleo\", \"type\": \"dog\"}], pk=\"id\")\r\nOut[5]:
\r\nIn [7]: db[\"creatures\"].extract(\"type\")\r\nOut[7]:
\r\nIn [8]: list(db[\"creatures\"].rows)\r\nOut[8]: \r\n[{'id': 1, 'name': 'Simon', 'type_id': None},\r\n {'id': 2, 'name': 'Natalie', 'type_id': None},\r\n {'id': 3, 'name': 'Cleo', 'type_id': 2}]\r\nIn [9]: db[\"type\"]\r\nOut[9]:
\r\nIn [10]: list(db[\"type\"].rows)\r\nOut[10]: [{'id': 1, 'type': None}, {'id': 2, 'type': 'dog'}]\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/186/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 723460107, "node_id": "MDU6SXNzdWU3MjM0NjAxMDc=", "number": 187, "title": "Maybe: Utility method / CLI tool for initializing SpatiaLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-16T19:04:03Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-16T19:15:13Z", "author_association": "OWNER", "pull_request": null, "body": "> I think this should initialize SpatiaLite against the current database if it has not been initialized already.\r\n>\r\n> Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/187/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 730693696, "node_id": "MDExOlB1bGxSZXF1ZXN0NTEwOTU2MTM0", "number": 190, "title": "Progress bar for sqlite-utils insert command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-27T18:08:53Z", "updated_at": "2020-10-27T18:16:03Z", "closed_at": "2020-10-27T18:16:03Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/190", "body": "Refs #173", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/190/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 731740458, "node_id": "MDU6SXNzdWU3MzE3NDA0NTg=", "number": 191, "title": "Idea: @db.register_function(deterministic=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-28T19:45:18Z", "updated_at": "2020-10-28T21:31:06Z", "closed_at": "2020-10-28T21:31:06Z", "author_association": "OWNER", "pull_request": null, "body": "Python 3.8 added a `deterministic` parameter to `db.create_function()`: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function\r\n\r\n`sqlite-utils` could expose this in the decorator, only actually applying it if the Python version supports it (using feature detection) - since nothing will break if it's not applied.\r\n\r\nhttps://sqlite-utils.readthedocs.io/en/stable/python-api.html#registering-custom-sql-functions\r\n\r\n```python\r\ndb = Database(memory=True)\r\n\r\n@db.register_function(deterministic=True)\r\ndef reverse_string(s):\r\n return \"\".join(reversed(list(s)))\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/191/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735532751, "node_id": "MDU6SXNzdWU3MzU1MzI3NTE=", "number": 192, "title": "sqlite-utils search command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 9, "created_at": "2020-11-03T18:07:59Z", "updated_at": "2020-11-08T17:07:01Z", "closed_at": "2020-11-08T17:07:01Z", "author_association": "OWNER", "pull_request": null, "body": "A command that knows how to run a search against a FTS enabled table and return results ranked by relevance.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/192/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735648209, "node_id": "MDU6SXNzdWU3MzU2NDgyMDk=", "number": 193, "title": "--tsv output format option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 0, "created_at": "2020-11-03T21:31:18Z", "updated_at": "2020-11-07T00:09:52Z", "closed_at": "2020-11-07T00:09:52Z", "author_association": "OWNER", "pull_request": null, "body": "We already support `--csv` for output, and the `insert` command accepts `--tsv`. The output format options should accept `--tsv` too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/193/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735650864, "node_id": "MDU6SXNzdWU3MzU2NTA4NjQ=", "number": 194, "title": "3.0 release with some minor breaking changes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 3, "created_at": "2020-11-03T21:36:31Z", "updated_at": "2020-11-08T17:19:35Z", "closed_at": "2020-11-08T17:19:34Z", "author_association": "OWNER", "pull_request": null, "body": "While working on search (#192) I've spotted a few small changes I would like to make that would break backwards compatibility in minor ways, hence requiring a 3.x release.\r\n\r\n`db[table].search()` - I would like this to default to sorting by rank\r\n\r\nAlso I'd like to free up the `-c` and `-f` options for other purposes from the standard output formats here:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/43eae8b193d362f2b292df73e087ed6f10838144/sqlite_utils/cli.py#L48-L58\r\n\r\nI'd like `-f` to be used to indicate a full-text search column during an insert and `-c` to indicate a column (so you can specify which columns you want to output).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/194/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735663855, "node_id": "MDExOlB1bGxSZXF1ZXN0NTE1MDE0ODgz", "number": 195, "title": "table.search() improvements plus sqlite-utils search command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-11-03T22:02:08Z", "updated_at": "2020-11-06T18:30:49Z", "closed_at": "2020-11-06T18:30:42Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/195", "body": "Refs #192. Still needs tests.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/195/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 736520310, "node_id": "MDU6SXNzdWU3MzY1MjAzMTA=", "number": 196, "title": "Introspect if table is FTS4 or FTS5", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 19, "created_at": "2020-11-05T00:45:50Z", "updated_at": "2020-11-05T03:54:07Z", "closed_at": "2020-11-05T03:54:07Z", "author_association": "OWNER", "pull_request": null, "body": "> I want `.search()` to work against both FTS5 and FTS4 tables - but sort by rank should only work for FTS5.\r\n>\r\n> This means I need to be able to introspect and tell if a table is FTS4 or FTS5.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/192#issuecomment-722054264_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/196/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 737153927, "node_id": "MDU6SXNzdWU3MzcxNTM5Mjc=", "number": 197, "title": "Rethink how table.search() method works", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 5, "created_at": "2020-11-05T18:04:34Z", "updated_at": "2020-11-08T17:07:37Z", "closed_at": "2020-11-08T17:07:37Z", "author_association": "OWNER", "pull_request": null, "body": "I need to improve this method to help build `sqlite-utils search` in #192 (PR is #195).\r\n\r\nThe challenge is deciding how it should handle sorting by relevance - especially since that is easy in FTS5 but not at all easy in FTS4.\r\n\r\n> Latest test failure:\r\n> ```\r\n> 114 ->\t assert [(\"racoons are biting trash pandas\", \"USA\", \"bar\")] == table.search(\r\n> 115 \t \"bite\", order=\"rowid\"\r\n> 116 \t )\r\n> 117 \t\r\n> 118 \t\r\n> 119 \tdef test_optimize_fts(fresh_db):\r\n> (Pdb) table.search(\"bite\")\r\n> [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)]\r\n> ```\r\n> The problem here is that the `table.search()` method now behaves differently for FTS4 v.s. FTS5 tables.\r\n> \r\n> With FTS4 you get back just the table columns.\r\n>\r\n> With FTS5 you also get back the `rowid` as the first column and the `rank` score as the last column.\r\n> \r\n> This is weird. It also makes me question whether having `.search()` return a list of tuples is the right API design.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/197/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 737476423, "node_id": "MDU6SXNzdWU3Mzc0NzY0MjM=", "number": 198, "title": "Support order by relevance against FTS4", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-11-06T05:36:31Z", "updated_at": "2020-11-06T18:30:44Z", "closed_at": "2020-11-06T18:30:44Z", "author_association": "OWNER", "pull_request": null, "body": "For #192 and #197 I've decided I want to be able to order by relevance in FTS4 as well as FTS5.\r\n\r\nThis means I need to port over my work on bm25() from https://github.com/simonw/sqlite-fts4 (since I don't want to add a full dependency).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/198/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 737855731, "node_id": "MDU6SXNzdWU3Mzc4NTU3MzE=", "number": 199, "title": "@db.register_function(..., replace=False) to avoid double-registering custom functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-06T15:39:21Z", "updated_at": "2020-11-06T18:30:44Z", "closed_at": "2020-11-06T18:30:44Z", "author_association": "OWNER", "pull_request": null, "body": "I'd like a mechanism to optionally avoid registering a custom function if it has already been registered.\r\n\r\nSQLite doesn't seem to offer a way to introspect registered custom functions so I'll need to track what has already been registered in `sqlite-utils` instead.\r\n\r\n> Should I register the custom `rank_bm25` SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/199/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 738115165, "node_id": "MDU6SXNzdWU3MzgxMTUxNjU=", "number": 200, "title": "sqlite-utils rows -c option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 1, "created_at": "2020-11-07T00:22:12Z", "updated_at": "2020-11-07T00:28:48Z", "closed_at": "2020-11-07T00:28:47Z", "author_association": "OWNER", "pull_request": null, "body": "To let you specify the exact columns you want. Based on the `-c` option to `sqlite-utils search` in #192.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/200/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 738128913, "node_id": "MDU6SXNzdWU3MzgxMjg5MTM=", "number": 201, "title": ".search(columns=) and sqlite-utils search -c ... bug", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 1, "created_at": "2020-11-07T01:27:26Z", "updated_at": "2020-11-08T16:54:15Z", "closed_at": "2020-11-08T16:54:15Z", "author_association": "OWNER", "pull_request": null, "body": "Both `table.search(columns=)` and the `sqlite-utils search -c` option do not work as expected - they always return both the `rowid` and the `rank` columns even if those have not been requested.\r\n\r\nThis should be fixed before the 3.0 non-alpha release.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/201/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 738514367, "node_id": "MDU6SXNzdWU3Mzg1MTQzNjc=", "number": 202, "title": "sqlite-utils insert -f colname - for configuring full-text search", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-11-08T17:30:09Z", "updated_at": "2021-01-03T05:00:36Z", "closed_at": "2021-01-03T05:00:27Z", "author_association": "OWNER", "pull_request": null, "body": "A mechanism for specifying columns that should be configured for full-text search as part of the initial data import:\r\n\r\n sqlite-utils insert mydb.db articles articles.csv --csv -f title -f body", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/202/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 761915790, "node_id": "MDU6SXNzdWU3NjE5MTU3OTA=", "number": 206, "title": "sqlite-utils should suggest --csv if JSON parsing fails", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-12-11T05:17:56Z", "updated_at": "2021-10-30T15:52:17Z", "closed_at": "2021-01-03T18:42:22Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n~ % gsutil cat gs://ossf-criticality-score/python_top_200.csv | sqlite-utils insert /tmp/crit.db crit -\r\n...\r\n File \"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py\", line 337, in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n File \"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py\", line 355, in raw_decode\r\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```\r\nA nicer error message here would be one that says the JSON is invalid but suggests that maybe you could try `--csv`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/206/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 763283616, "node_id": "MDU6SXNzdWU3NjMyODM2MTY=", "number": 207, "title": "sqlite-utils analyze-tables command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-12-12T04:33:12Z", "updated_at": "2020-12-13T07:25:23Z", "closed_at": "2020-12-13T07:20:13Z", "author_association": "OWNER", "pull_request": null, "body": "A command which analyzes a table (potentially taking quite a while if the table is large) and outputs information for each column - things like:\r\n\r\n- How many unique values does this column have?\r\n- How many null rows?\r\n- How many blank rows? (defined as empty string)\r\n- What are the 10 most common values?\r\n- What are the 10 least common values?\r\n\r\nThe command can output this information to the terminal, but it should also provide an option for writing the information to a database table so it can be explored later.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/207/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 763320133, "node_id": "MDExOlB1bGxSZXF1ZXN0NTM3NzkxNjc1", "number": 208, "title": "sqlite-utils analyze-tables command and table.analyze_column() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-12-12T05:27:49Z", "updated_at": "2020-12-13T07:20:16Z", "closed_at": "2020-12-13T07:20:12Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/208", "body": "Refs #207\r\n\r\n- [x] Improve design of CLI output\r\n- [x] Truncate long values in least/most common\r\n- [x] Add a `-c` column selection option\r\n- [x] Tests\r\n- [x] Documentation", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/208/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 777386465, "node_id": "MDU6SXNzdWU3NzczODY0NjU=", "number": 211, "title": "table.triggers_dict introspection property", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-02T02:04:00Z", "updated_at": "2021-01-02T02:10:10Z", "closed_at": "2021-01-02T02:10:10Z", "author_association": "OWNER", "pull_request": null, "body": "`table.triggers` currently returns a list of `Trigger` values. A `table.triggers_dict` property could behave like `columns_dict`, returning a dictionary mapping trigger names to their SQL definitions for that table.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/211/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777392020, "node_id": "MDU6SXNzdWU3NzczOTIwMjA=", "number": 212, "title": "Mechanism for maintaining cache of table counts using triggers", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-01-02T02:58:53Z", "updated_at": "2021-01-02T21:40:27Z", "closed_at": "2021-01-02T21:40:27Z", "author_association": "OWNER", "pull_request": null, "body": "Counting all of the rows in a large table is expensive - this is one of the main causes of performance problems in Datasette when running against large databases.\r\n\r\nCarefully constructed SQL triggers could be used to maintain accurate cached counts for a table, by incrementing and decrementing a counter every time a row is inserted or deleted.\r\n\r\n`sqlite-utils` already has a mechanism for creating triggers for FTS - the `table.enable_fts(..., create_triggers=True)` method. How about a similar mechanism for setting up triggers to maintain a count of table rows?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/212/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777529979, "node_id": "MDU6SXNzdWU3Nzc1Mjk5Nzk=", "number": 213, "title": "db.enable_counts() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-02T21:44:55Z", "updated_at": "2021-01-02T22:04:02Z", "closed_at": "2021-01-02T22:04:02Z", "author_association": "OWNER", "pull_request": null, "body": "Following #212 it would be useful if there was a utility method for enabling counts for ALL tables in a database:\r\n\r\n```python\r\ndb.enable_counts()\r\n```\r\n\r\nOpen question: should this setup triggers for virtual tables such as FTS tables? Could that break things?\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/213/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777530107, "node_id": "MDU6SXNzdWU3Nzc1MzAxMDc=", "number": 214, "title": "sqlite-utils enable-counts command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-02T21:45:48Z", "updated_at": "2021-01-03T04:26:44Z", "closed_at": "2021-01-03T04:26:44Z", "author_association": "OWNER", "pull_request": null, "body": "The CLI version of #212 and #213.\r\n\r\n # Enable counts for all tables:\r\n sqlite-utils enable-counts data.db\r\n\r\n # Enable counts for specific tables:\r\n sqlite-utils enable-counts data.db table1 table2", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/214/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777535402, "node_id": "MDU6SXNzdWU3Nzc1MzU0MDI=", "number": 215, "title": "Use _counts to speed up counts", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-01-02T22:30:17Z", "updated_at": "2021-01-03T20:19:40Z", "closed_at": "2021-01-03T20:19:40Z", "author_association": "OWNER", "pull_request": null, "body": "Utility mechanism for taking advantage of the new `_counts` table from #212 would be nice.\r\n\r\nThese can trigger automatically if the `_counts` table exists, but since `sqlite-utils` needs to work against any existing database there should be a way of opting out of this optimization.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/215/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777540352, "node_id": "MDU6SXNzdWU3Nzc1NDAzNTI=", "number": 216, "title": "database.triggers_dict introspection property", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-01-02T23:13:00Z", "updated_at": "2021-01-03T04:27:14Z", "closed_at": "2021-01-03T04:25:36Z", "author_association": "OWNER", "pull_request": null, "body": "Following #211", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/216/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777543336, "node_id": "MDU6SXNzdWU3Nzc1NDMzMzY=", "number": 217, "title": "Rename .escape() to .quote()", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-02T23:40:52Z", "updated_at": "2021-01-03T04:27:38Z", "closed_at": "2021-01-03T04:15:23Z", "author_association": "OWNER", "pull_request": null, "body": "`.quote()` is a better name because it reflects that the method adds quotes around the value.\r\n\r\nThis method has never been documented so I'm going to rename it without a major version bump. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/217/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777560474, "node_id": "MDU6SXNzdWU3Nzc1NjA0NzQ=", "number": 218, "title": "\"sqlite-utils triggers\" command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-01-03T02:34:50Z", "updated_at": "2021-01-03T03:49:51Z", "closed_at": "2021-01-03T03:03:35Z", "author_association": "OWNER", "pull_request": null, "body": "A command to list the triggers in the database.\r\n\r\n sqlite-utils triggers my.db\r\n\r\nCan optionally take one or more tables:\r\n\r\n sqlite-utils triggers my.db table1 table2", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/218/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 777707544, "node_id": "MDU6SXNzdWU3Nzc3MDc1NDQ=", "number": 219, "title": "reset_counts() method and command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-01-03T20:08:28Z", "updated_at": "2021-01-03T20:59:37Z", "closed_at": "2021-01-03T20:59:37Z", "author_association": "OWNER", "pull_request": null, "body": "> Thought: maybe there should be a `.reset_counts()` method too, for if the table gets out of date with the triggers.\r\n>\r\n> One way that could happen is if a table is dropped and recreated - the counts in the `_counts` table would likely no longer match the number of rows in that table.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545757_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/219/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 783910901, "node_id": "MDU6SXNzdWU3ODM5MTA5MDE=", "number": 221, "title": ".add_missing_columns() does not take case insensitivity into account", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-12T05:01:00Z", "updated_at": "2021-01-12T23:17:33Z", "closed_at": "2021-01-12T23:17:33Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite columns are case insensitive - but the `.add_missing_columns()` method doesn't know that. This means that it can crash if it identifies a column that is a case-insensitive duplicate of an existing column. https://github.com/simonw/sqlite-utils/blob/4cc82fd0bccc9d2eeb3510beb4e691d7da099f84/sqlite_utils/db.py#L1974-L1980", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/221/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 787900412, "node_id": "MDU6SXNzdWU3ODc5MDA0MTI=", "number": 222, "title": ".m2m() should accept alter=True parameter", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-01-18T04:15:43Z", "updated_at": "2021-01-18T04:26:10Z", "closed_at": "2021-01-18T04:26:10Z", "author_association": "OWNER", "pull_request": null, "body": "Needed by https://github.com/dogsheep/swarm-to-sqlite/issues/11", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/222/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 788527932, "node_id": "MDU6SXNzdWU3ODg1Mjc5MzI=", "number": 223, "title": "--delimiter option for CSV import", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-01-18T20:25:03Z", "updated_at": "2021-02-06T01:39:47Z", "closed_at": "2021-02-06T01:34:54Z", "author_association": "OWNER", "pull_request": null, "body": "https://bruxellesdata.opendatasoft.com/explore/dataset/dog-toilets/export/?location=12,50.85802,4.38054 says:\r\n\r\n> CSV uses semicolon (;) as a separator.\r\n\r\nWould be useful to be able to do this:\r\n\r\n sqlite-utils insert places.db places places.csv --delimiter ';'\r\n\r\n`--delimiter` could imply `--csv`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/223/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 802583450, "node_id": "MDU6SXNzdWU4MDI1ODM0NTA=", "number": 226, "title": "3.4 release is broken - includes a rogue line", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-06T02:08:01Z", "updated_at": "2021-02-06T02:10:26Z", "closed_at": "2021-02-06T02:10:26Z", "author_association": "OWNER", "pull_request": null, "body": "I started seeing weird errors, caused by this line: https://github.com/simonw/sqlite-utils/blob/f8010ca78fed8c5fca6cde19658ec09fdd468420/sqlite_utils/cli.py#L1-L3\r\n\r\nThat was added by accident in 1b666f9315d4ea6bb332b2e75e48480c26100199\r\n\r\nI'm surprised the tests didn't catch this!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/226/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 807437089, "node_id": "MDU6SXNzdWU4MDc0MzcwODk=", "number": 228, "title": "--no-headers option for CSV and TSV", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2021-02-12T17:56:51Z", "updated_at": "2021-12-26T07:01:31Z", "closed_at": "2021-02-14T22:25:17Z", "author_association": "OWNER", "pull_request": null, "body": "https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c has a fascinating CSV file that doesn't have a header row - it starts like this:\r\n\r\n```csv\r\nComputation and measurement of turbulent flow through idealized turbine blade passages,,\"Loizou, Panos A.\",https://isni.org/isni/0000000136122593,,University of Manchester,https://isni.org/isni/0000000121662407,1989,Thesis (Ph.D.),,Physical Sciences,,,https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.232781,\r\n\"Prolactin and growth hormone secretion in normal, hyperprolactinaemic and acromegalic man\",,\"Prescott, R. W. G.\",https://isni.org/isni/0000000134992122,,University of Newcastle upon Tyne,https://isni.org/isni/0000000104627212,1983,Thesis (Ph.D.),,Biological Sciences,,,https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.232784,\r\n```\r\n\r\nIt would be useful if `sqlite-utils insert ... --csv` had a mechanism for importing files like this one.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/228/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 808008305, "node_id": "MDU6SXNzdWU4MDgwMDgzMDU=", "number": 230, "title": "--sniff option for sniffing delimiters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-02-14T17:43:54Z", "updated_at": "2021-02-14T21:15:33Z", "closed_at": "2021-02-14T19:24:32Z", "author_association": "OWNER", "pull_request": null, "body": "> I just spotted that `csv.Sniffer` in the Python standard library has a `.has_header(sample)` method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/230/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 808028757, "node_id": "MDU6SXNzdWU4MDgwMjg3NTc=", "number": 231, "title": "limit=X, offset=Y parameters for more Python methods", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-02-14T19:31:23Z", "updated_at": "2021-02-14T20:03:08Z", "closed_at": "2021-02-14T20:03:08Z", "author_association": "OWNER", "pull_request": null, "body": "> I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion!\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/224#issuecomment-778828495_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/231/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 808036774, "node_id": "MDU6SXNzdWU4MDgwMzY3NzQ=", "number": 232, "title": "Run tests against Windows in GitHub Actions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-14T20:09:45Z", "updated_at": "2021-02-14T20:39:55Z", "closed_at": "2021-02-14T20:39:55Z", "author_association": "OWNER", "pull_request": null, "body": "> I'm going to try and get the test suite to run in Windows on GitHub Actions.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/225#issuecomment-778834504_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/232/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 808037010, "node_id": "MDExOlB1bGxSZXF1ZXN0NTczMTQ3MTY4", "number": 233, "title": "Run tests against Ubuntu, macOS and Windows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-14T20:11:02Z", "updated_at": "2021-02-14T20:39:54Z", "closed_at": "2021-02-14T20:39:54Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/233", "body": "Refs #232", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/233/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 808046597, "node_id": "MDU6SXNzdWU4MDgwNDY1OTc=", "number": 234, "title": ".insert_all() fails if subsequent chunks contain additional columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-14T21:01:51Z", "updated_at": "2021-02-14T21:03:40Z", "closed_at": "2021-02-14T21:03:40Z", "author_association": "OWNER", "pull_request": null, "body": "Reported by @nieuwenhoven in #225 along with a proposed fix.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/234/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 811680502, "node_id": "MDU6SXNzdWU4MTE2ODA1MDI=", "number": 236, "title": "--attach command line option for attaching extra databases", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-19T04:38:30Z", "updated_at": "2021-02-19T05:10:41Z", "closed_at": "2021-02-19T05:08:43Z", "author_association": "OWNER", "pull_request": null, "body": "This will enable cross-database joins, as seen in https://github.com/simonw/datasette/issues/283\r\n\r\nAlso refs #113", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/236/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 816523763, "node_id": "MDU6SXNzdWU4MTY1MjM3NjM=", "number": 238, "title": ".add_foreign_key() corrupts database if column contains a space", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-02-25T15:07:20Z", "updated_at": "2021-02-25T16:54:02Z", "closed_at": "2021-02-25T16:54:02Z", "author_association": "OWNER", "pull_request": null, "body": "I ran this:\r\n\r\n db[\"Reports\"].add_foreign_key(\"Reported by ID\", \"Reporters\", \"id\")\r\n\r\nAnd got this:\r\n\r\n```\r\n~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in add_foreign_keys(self, foreign_keys)\r\n 616 # Have to VACUUM outside the transaction to ensure .foreign_keys property\r\n 617 # can see the newly created foreign key.\r\n--> 618 self.vacuum()\r\n 619 \r\n 620 def index_foreign_keys(self):\r\n\r\n~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in vacuum(self)\r\n 629 \r\n 630 def vacuum(self):\r\n--> 631 self.execute(\"VACUUM;\")\r\n 632 \r\n 633 \r\n\r\n~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in execute(self, sql, parameters)\r\n 234 return self.conn.execute(sql, parameters)\r\n 235 else:\r\n--> 236 return self.conn.execute(sql)\r\n 237 \r\n 238 def executescript(self, sql):\r\n\r\nDatabaseError: database disk image is malformed\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/238/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 816526538, "node_id": "MDU6SXNzdWU4MTY1MjY1Mzg=", "number": 239, "title": "sqlite-utils extract could handle nested objects", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 16, "created_at": "2021-02-25T15:10:28Z", "updated_at": "2022-09-03T23:46:02Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Imagine a table (imported from a nested JSON file) where one of the columns contains values that look like this:\r\n\r\n {\"email\": \"anonymous@noreply.airtable.com\", \"id\": \"usrROSHARE0000000\", \"name\": \"Anonymous\"}\r\n\r\nThe `sqlite-utils extract` command already uses single text values in a column to populate a new table. It would not be much of a stretch for it to be able to use JSON instead, including specifying which of those values should be used as the primary key in the new table.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/239/reactions\", \"total_count\": 6, \"+1\": 5, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 816560819, "node_id": "MDU6SXNzdWU4MTY1NjA4MTk=", "number": 240, "title": "table.pks_and_rows_where() method returning primary keys along with the rows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2021-02-25T15:49:28Z", "updated_at": "2021-02-25T16:39:23Z", "closed_at": "2021-02-25T16:28:23Z", "author_association": "OWNER", "pull_request": null, "body": "*Original title: Easier way to update a row returned from .rows*\r\n\r\nHere's a surprisingly hard problem I ran into while trying to implement #239 - given a row returned by `db[table].rows` how can you update that row?\r\n\r\nThe problem is that the `db[table].update(...)` method requires a primary key. But if you have a row from the `db[table].rows` iterator it might not even contain the primary key - provided the table is a `rowid` table.\r\n\r\nInstead, currently, you need to introspect the table and, if `rowid` is a primary key, explicitly include that in the `select=` argument to `table.rows_where(...)` - otherwise it will not be returned.\r\n\r\nA utility mechanism to make this easier would be very welcome.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/240/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 816601354, "node_id": "MDExOlB1bGxSZXF1ZXN0NTgwMjM1NDI3", "number": 241, "title": "Extract expand - work in progress", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-02-25T16:36:38Z", "updated_at": "2021-02-25T16:36:38Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/241", "body": "Refs #239. Still needs documentation and CLI implementation.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/241/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null} {"id": 838148087, "node_id": "MDU6SXNzdWU4MzgxNDgwODc=", "number": 250, "title": "Handle byte order marks (BOMs) in CSV files", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-03-22T22:13:18Z", "updated_at": "2021-05-29T05:34:21Z", "closed_at": "2021-05-29T05:34:21Z", "author_association": "OWNER", "pull_request": null, "body": "I often find `sqlite-utils insert ... --csv` creates a first column with a weird character at the start of it - which it turns out is the UTF-8 BOM. Fix that.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/250/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 841377702, "node_id": "MDU6SXNzdWU4NDEzNzc3MDI=", "number": 251, "title": "\"sqlite-utils convert\" command to replace the separate \"sqlite-transform\" tool", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 15, "created_at": "2021-03-25T22:36:36Z", "updated_at": "2021-08-02T22:39:46Z", "closed_at": "2021-08-02T04:47:40Z", "author_association": "OWNER", "pull_request": null, "body": "See https://github.com/simonw/sqlite-transform/issues/11 - I built a separate `sqlite-transform` tool a while ago that uses the word \"transform\" to means something entirely different from `sqlite-utils transform` - I'd like to resolve this by merging the two tools.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/251/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 858501079, "node_id": "MDU6SXNzdWU4NTg1MDEwNzk=", "number": 255, "title": "transform --help should tell you the available types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-04-15T05:24:48Z", "updated_at": "2021-05-29T03:55:52Z", "closed_at": "2021-05-29T03:55:52Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\nUsage: sqlite-utils transform [OPTIONS] PATH TABLE\r\n\r\n Transform a table beyond the capabilities of ALTER TABLE\r\n\r\nOptions:\r\n --type ... Change column type to X\r\n```\r\nThis should specify that the possible types are 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/255/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 894948100, "node_id": "MDU6SXNzdWU4OTQ5NDgxMDA=", "number": 259, "title": "Suggest the --alter option if a new column cannot be added", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-05-19T03:17:38Z", "updated_at": "2021-05-19T03:27:33Z", "closed_at": "2021-05-19T03:26:26Z", "author_association": "OWNER", "pull_request": null, "body": "Refs #256.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/259/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}