{"html_url": "https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010462035", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/381", "id": 1010462035, "node_id": "IC_kwDOCGYnMM48Om1T", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T23:33:37Z", "updated_at": "2022-01-11T23:33:37Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1099584685, "label": "`sqlite-utils rows` options `--limit` and `--offset`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/382#issuecomment-1010461844", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/382", "id": 1010461844, "node_id": "IC_kwDOCGYnMM48OmyU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T23:33:14Z", "updated_at": "2022-01-11T23:33:14Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#returning-all-rows-in-a-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1099585611, "label": "`--where` option for `sqlite-rows`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/381#issuecomment-1010441118", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/381", "id": 1010441118, "node_id": "IC_kwDOCGYnMM48Ohue", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T22:56:53Z", "updated_at": "2022-01-11T22:57:09Z", "author_association": "OWNER", "body": "`sqlite-utils search` has `--limit` already: https://sqlite-utils.datasette.io/en/latest/cli-reference.html#search\r\n\r\n```\r\n --limit INTEGER Number of rows to return - defaults to everything\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1099584685, "label": "`sqlite-utils rows` options `--limit` and `--offset`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010440166", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/383", "id": 1010440166, "node_id": "IC_kwDOCGYnMM48Ohfm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T22:55:05Z", "updated_at": "2022-01-11T22:55:05Z", "author_association": "OWNER", "body": "Twitter thread about this: https://twitter.com/simonw/status/1481020195074293761", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1099586786, "label": "Add documentation page with the output of `--help`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010387223", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/383", "id": 1010387223, "node_id": "IC_kwDOCGYnMM48OUkX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T21:45:32Z", "updated_at": "2022-01-11T21:45:32Z", "author_association": "OWNER", "body": "The new page of documentation: https://sqlite-utils.datasette.io/en/latest/cli-reference.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1099586786, "label": "Add documentation page with the output of `--help`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010386802", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/383", "id": 1010386802, "node_id": "IC_kwDOCGYnMM48OUdy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T21:44:53Z", "updated_at": "2022-01-11T21:44:53Z", "author_association": "OWNER", "body": "Here's the `cog` code I used: https://github.com/simonw/sqlite-utils/blob/1d44b0cc2784c94aed1bcf350225cd86ee1aa7e5/docs/cli-reference.rst#L11-L76", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1099586786, "label": "Add documentation page with the output of `--help`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/383#issuecomment-1010333511", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/383", "id": 1010333511, "node_id": "IC_kwDOCGYnMM48OHdH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T20:27:08Z", "updated_at": "2022-01-11T20:27:08Z", "author_association": "OWNER", "body": "I'll call the new page \"CLI reference\", for consistency with the API reference page here: https://sqlite-utils.datasette.io/en/stable/reference.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1099586786, "label": "Add documentation page with the output of `--help`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009548580", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1009548580, "node_id": "IC_kwDOCGYnMM48LH0k", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-11T02:43:34Z", "updated_at": "2022-01-11T02:43:34Z", "author_association": "CONTRIBUTOR", "body": "thanks so much! always a pleasure to see how you work through these things", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/380#issuecomment-1009544785", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/380", "id": 1009544785, "node_id": "IC_kwDOCGYnMM48LG5R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T02:32:56Z", "updated_at": "2022-01-11T02:32:56Z", "author_association": "OWNER", "body": "CLI and Python library improvements to help run [ANALYZE](https://www.sqlite.org/lang_analyze.html) after creating indexes or inserting rows, to gain better performance from the SQLite query planner when it runs against indexes.\r\n\r\nThree new CLI commands: `create-database`, `analyze` and `bulk`.\r\n\r\n- New `sqlite-utils create-database` command for creating new empty database files. ([#348](https://github.com/simonw/sqlite-utils/issues/348))\r\n- New Python methods for running `ANALYZE` against a database, table or index: `db.analyze()` and `table.analyze()`, see [Optimizing index usage with ANALYZE](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-analyze). ([#366](https://github.com/simonw/sqlite-utils/issues/366))\r\n- New [sqlite-utils analyze command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-analyze) for running `ANALYZE` using the CLI. ([#379](https://github.com/simonw/sqlite-utils/issues/379))\r\n- The `create-index`, `insert` and `update` commands now have a new `--analyze` option for running `ANALYZE` after the command has completed. ([#379](https://github.com/simonw/sqlite-utils/issues/379))\r\n- New [sqlite-utils bulk command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-bulk) which can import records in the same way as `sqlite-utils insert` (from JSON, CSV or TSV) and use them to bulk execute a parametrized SQL query. ([#375](https://github.com/simonw/sqlite-utils/issues/375))\r\n- The CLI tool can now also be run using `python -m sqlite_utils`. ([#368](https://github.com/simonw/sqlite-utils/issues/368))\r\n- Using `--fmt` now implies `--table`, so you don't need to pass both options. ([#374](https://github.com/simonw/sqlite-utils/issues/374))\r\n- The `--convert` function applied to rows can now modify the row in place. ([#371](https://github.com/simonw/sqlite-utils/issues/371))\r\n- The [insert-files command](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-files) supports two new columns: `stem` and `suffix`. ([#372](https://github.com/simonw/sqlite-utils/issues/372))\r\n- The `--nl` import option now ignores blank lines in the input. ([#376](https://github.com/simonw/sqlite-utils/issues/376))\r\n- Fixed bug where streaming input to the `insert` command with `--batch-size 1` would appear to only commit after several rows had been ingested, due to unnecessary input buffering. ([#364](https://github.com/simonw/sqlite-utils/issues/364))", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1098574572, "label": "Release notes for 3.21"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1009536276", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/375", "id": 1009536276, "node_id": "IC_kwDOCGYnMM48LE0U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T02:12:58Z", "updated_at": "2022-01-11T02:12:58Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-sql-in-bulk", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097251014, "label": "`sqlite-utils bulk` command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009534817", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/377", "id": 1009534817, "node_id": "IC_kwDOCGYnMM48LEdh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T02:09:38Z", "updated_at": "2022-01-11T02:09:38Z", "author_association": "OWNER", "body": "I tested this like so:\r\n```\r\n% wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'\r\n% sqlite-utils create-database test.db\r\n% sqlite-utils create-table test.db power_plants url text owner text\r\n% sqlite-utils schema test.db \r\nCREATE TABLE [power_plants] (\r\n [url] TEXT,\r\n [owner] TEXT\r\n);\r\n% sqlite-utils bulk test.db 'insert into power_plants (url, owner) values (:url, :owner)' global_power_plant_database.csv --csv\r\n [------------------------------------] 0%\r\n [###################################-] 99%\r\n% sqlite-utils tables --counts test.db -t\r\ntable count\r\n------------ -------\r\npower_plants 33643\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097477582, "label": "`sqlite-utils bulk` command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009532125", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/377", "id": 1009532125, "node_id": "IC_kwDOCGYnMM48LDzd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T02:03:35Z", "updated_at": "2022-01-11T02:03:35Z", "author_association": "OWNER", "body": "Documentation: https://github.com/simonw/sqlite-utils/blob/f4ea0d32c0543373eefaa9b9f3911eb07549eecb/docs/cli.rst#executing-sql-in-bulk", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097477582, "label": "`sqlite-utils bulk` command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/377#issuecomment-1009531863", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/377", "id": 1009531863, "node_id": "IC_kwDOCGYnMM48LDvX", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-01-11T02:03:00Z", "updated_at": "2022-01-11T02:03:00Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#377](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f4ea0d3) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/b6dad08a8389736b7e960cfe9bc719cfc21a98f5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b6dad08) will **decrease** coverage by `0.01%`.\n> The diff coverage is `96.00%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/377/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #377 +/- ##\n==========================================\n- Coverage 96.52% 96.50% -0.02% \n==========================================\n Files 6 6 \n Lines 2330 2378 +48 \n==========================================\n+ Hits 2249 2295 +46 \n- Misses 81 83 +2 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <94.28%> (-0.06%)` | :arrow_down: |\n| [sqlite\\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/377/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.68% <100.00%> (+0.03%)` | :arrow_up: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b6dad08...f4ea0d3](https://codecov.io/gh/simonw/sqlite-utils/pull/377?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097477582, "label": "`sqlite-utils bulk` command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009521921", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1009521921, "node_id": "IC_kwDOCGYnMM48LBUB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T01:37:53Z", "updated_at": "2022-01-11T01:37:53Z", "author_association": "OWNER", "body": "I decided to go with making this opt-in, mainly for consistency with the other places where I added this feature - see:\r\n- #379 \r\n- #366\r\n\r\nYou can now run the following:\r\n\r\n sqlite-utils create-index mydb.db mytable mycolumn --analyze\r\n\r\nAnd ``ANALYZE`` will be run on the index once it has been created.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1009508865, "node_id": "IC_kwDOCGYnMM48K-IB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-11T01:08:51Z", "updated_at": "2022-01-11T01:08:51Z", "author_association": "OWNER", "body": "The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1009288898, "node_id": "IC_kwDOCGYnMM48KIbC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T19:54:04Z", "updated_at": "2022-01-10T19:54:04Z", "author_association": "OWNER", "body": "Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are:\r\n\r\n- `db.create_index`\r\n- `table.insert_all`\r\n- `table.upsert_all`\r\n- `table.delete_where`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009285627", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1009285627, "node_id": "IC_kwDOCGYnMM48KHn7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T19:49:19Z", "updated_at": "2022-01-10T19:51:25Z", "author_association": "OWNER", "body": "Documentation for those two new methods: https://sqlite-utils.datasette.io/en/latest/python-api.html#optimizing-index-usage-with-analyze", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009286373", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1009286373, "node_id": "IC_kwDOCGYnMM48KHzl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T19:50:22Z", "updated_at": "2022-01-10T19:50:22Z", "author_association": "OWNER", "body": "With respect to #365, I'm now thinking that having the ability to say \"... and then run ANALYZE\" could be useful for a bunch of Python methods. For example:\r\n\r\n```python\r\ndb[\"dogs\"].insert_all(list_of_dogs, analyze=True)\r\ndb[\"dogs\"].create_index([\"name\"], analyze=True)\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009273525", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1009273525, "node_id": "IC_kwDOCGYnMM48KEq1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T19:32:39Z", "updated_at": "2022-01-10T19:32:39Z", "author_association": "OWNER", "body": "I'm going to implement the Python library methods based on the prototype:\r\n```diff\r\ncommit 650f97a08f29a688c530e5f6c9eedc9269ed7bdc\r\nAuthor: Simon Willison \r\nDate: Sat Jan 8 13:34:01 2022 -0800\r\n\r\n Initial prototype of .analyze(), refs #366\r\n\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex dfc4723..1348b4a 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -923,6 +923,13 @@ class Database:\r\n \"Run a SQLite ``VACUUM`` against the database.\"\r\n self.execute(\"VACUUM;\")\r\n \r\n+ def analyze(self, name=None):\r\n+ \"Run ``ANALYZE`` against the entire database or a named table or index.\"\r\n+ sql = \"ANALYZE\"\r\n+ if name is not None:\r\n+ sql += \" [{}]\".format(name)\r\n+ self.execute(sql)\r\n+\r\n \r\n class Queryable:\r\n def exists(self) -> bool:\r\n@@ -2902,6 +2909,10 @@ class Table(Queryable):\r\n )\r\n return self\r\n \r\n+ def analyze(self):\r\n+ \"Run ANALYZE against this table\"\r\n+ self.db.analyze(self.name)\r\n+\r\n def analyze_column(\r\n self, column: str, common_limit: int = 10, value_truncate=None, total_rows=None\r\n ) -> \"ColumnDetails\":\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1009272446", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/367", "id": 1009272446, "node_id": "IC_kwDOCGYnMM48KEZ-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T19:31:08Z", "updated_at": "2022-01-10T19:31:08Z", "author_association": "OWNER", "body": "I'm going to implement this in a separate commit from this PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097041471, "label": "Initial prototype of .analyze() methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008557414", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008557414, "node_id": "IC_kwDOCGYnMM48HV1m", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T05:36:19Z", "updated_at": "2022-01-10T05:36:19Z", "author_association": "OWNER", "body": "That did the trick.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008556706", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/375", "id": 1008556706, "node_id": "IC_kwDOCGYnMM48HVqi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T05:33:41Z", "updated_at": "2022-01-10T05:33:41Z", "author_association": "OWNER", "body": "I tested the prototype like this:\r\n\r\n sqlite-utils blah.db 'create table blah (id integer primary key, name text)' \r\n echo 'id,name\r\n 1,Cleo\r\n 2,Chicken' > blah.csv\r\n sqlite-utils bulk blah.db 'insert into blah (id, name) values (:id, :name)' blah.csv --csv\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097251014, "label": "`sqlite-utils bulk` command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008546573, "node_id": "IC_kwDOCGYnMM48HTMN", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T05:05:15Z", "updated_at": "2022-01-10T05:05:15Z", "author_association": "OWNER", "body": "Bit nasty but it might work:\r\n```python\r\n def try_until(expected):\r\n tries = 0\r\n while True:\r\n rows = list(Database(db_path)[\"rows\"].rows)\r\n if rows == expected:\r\n return\r\n tries += 1\r\n if tries > 10:\r\n assert False, \"Expected {}, got {}\".format(expected, rows)\r\n time.sleep(tries * 0.1)\r\n\r\n try_until([{\"name\": \"Azi\"}])\r\n proc.stdin.write(b'{\"name\": \"Suna\"}\\n')\r\n proc.stdin.flush()\r\n try_until([{\"name\": \"Azi\"}, {\"name\": \"Suna\"}])\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008545140, "node_id": "IC_kwDOCGYnMM48HS10", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T05:01:34Z", "updated_at": "2022-01-10T05:01:34Z", "author_association": "OWNER", "body": "Urgh, tests are still failing intermittently - for example:\r\n```\r\n time.sleep(0.4)\r\n> assert list(Database(db_path)[\"rows\"].rows) == [{\"name\": \"Azi\"}]\r\nE AssertionError: assert [] == [{'name': 'Azi'}]\r\nE Right contains one more item: {'name': 'Azi'}\r\nE Full diff:\r\nE - [{'name': 'Azi'}]\r\nE + []\r\n```\r\nI'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008537194, "node_id": "IC_kwDOCGYnMM48HQ5q", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T04:29:53Z", "updated_at": "2022-01-10T04:31:29Z", "author_association": "OWNER", "body": "After a bunch of debugging with `print()` statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time.\r\n\r\nI think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770\r\n\r\nThe buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008526736, "node_id": "IC_kwDOCGYnMM48HOWQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-10T04:07:29Z", "updated_at": "2022-01-10T04:07:29Z", "author_association": "OWNER", "body": "I think this test is right:\r\n```python\r\ndef test_insert_streaming_batch_size_1(db_path):\r\n # https://github.com/simonw/sqlite-utils/issues/364\r\n # Streaming with --batch-size 1 should commit on each record\r\n # Can't use CliRunner().invoke() here bacuse we need to\r\n # run assertions in between writing to process stdin\r\n proc = subprocess.Popen(\r\n [\r\n sys.executable,\r\n \"-m\",\r\n \"sqlite_utils\",\r\n \"insert\",\r\n db_path,\r\n \"rows\",\r\n \"-\",\r\n \"--nl\",\r\n \"--batch-size\",\r\n \"1\",\r\n ],\r\n stdin=subprocess.PIPE,\r\n )\r\n proc.stdin.write(b'{\"name\": \"Azi\"}')\r\n proc.stdin.flush()\r\n assert list(Database(db_path)[\"rows\"].rows) == [{\"name\": \"Azi\"}]\r\n proc.stdin.write(b'{\"name\": \"Suna\"}')\r\n proc.stdin.flush()\r\n assert list(Database(db_path)[\"rows\"].rows) == [{\"name\": \"Azi\"}, {\"name\": \"Suna\"}]\r\n proc.stdin.close()\r\n proc.wait()\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008383293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/348", "id": 1008383293, "node_id": "IC_kwDOCGYnMM48GrU9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T20:38:17Z", "updated_at": "2022-01-09T20:38:17Z", "author_association": "OWNER", "body": "Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#creating-an-empty-database", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1067771698, "label": "Command for creating an empty database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/348#issuecomment-1008367607", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/348", "id": 1008367607, "node_id": "IC_kwDOCGYnMM48Gnf3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T20:22:43Z", "updated_at": "2022-01-09T20:22:43Z", "author_association": "OWNER", "body": "I'm not going to implement `--page-size` unless someone specifically requests it - I don't like having features that I've never needed to use myself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1067771698, "label": "Command for creating an empty database"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008364701", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/371", "id": 1008364701, "node_id": "IC_kwDOCGYnMM48Gmyd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T20:04:35Z", "updated_at": "2022-01-09T20:04:35Z", "author_association": "OWNER", "body": "The previous code for highlighting errors in syntax (which was already a bit confused thanks to the added `return`, see https://github.com/simonw/sqlite-utils/issues/355#issuecomment-991393684 - isn't compatible with this approach at all. I'm going to ditch it and just show a generic `Error: Could not compile code` message.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097128334, "label": "Support mutating row in `--convert` without returning it"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008354207", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/371", "id": 1008354207, "node_id": "IC_kwDOCGYnMM48GkOf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T18:54:54Z", "updated_at": "2022-01-09T18:54:54Z", "author_association": "OWNER", "body": "This seems to work:\r\n```python\r\ndef _compile_code(code, imports, variable=\"value\"):\r\n locals = {}\r\n globals = {\"r\": recipes, \"recipes\": recipes}\r\n # If user defined a convert() function, return that\r\n try:\r\n exec(code, globals, locals)\r\n return locals[\"convert\"]\r\n except (AttributeError, SyntaxError, NameError, KeyError, TypeError):\r\n pass\r\n\r\n # Try compiling their code as a function instead\r\n body_variants = [code]\r\n # If single line and no 'return', try adding the return\r\n if \"\\n\" not in code and not code.strip().startswith(\"return \"):\r\n body_variants.insert(0, \"return {}\".format(code))\r\n\r\n for variant in body_variants:\r\n new_code = [\"def fn({}):\".format(variable)]\r\n for line in variant.split(\"\\n\"):\r\n new_code.append(\" {}\".format(line))\r\n try:\r\n code_o = compile(\"\\n\".join(new_code), \"\", \"exec\")\r\n break\r\n except SyntaxError:\r\n # Try another variant, e.g. for 'return row[\"column\"] = 1'\r\n continue\r\n\r\n for import_ in imports:\r\n globals[import_.split(\".\")[0]] = __import__(import_)\r\n exec(code_o, globals, locals)\r\n return locals[\"fn\"]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097128334, "label": "Support mutating row in `--convert` without returning it"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008348032", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/371", "id": 1008348032, "node_id": "IC_kwDOCGYnMM48GiuA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T18:14:02Z", "updated_at": "2022-01-09T18:14:02Z", "author_association": "OWNER", "body": "Here's the code in question: https://github.com/simonw/sqlite-utils/blob/b8c134059e89f0fa040b84fb7d0bda25b9a52759/sqlite_utils/utils.py#L288-L299", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097128334, "label": "Support mutating row in `--convert` without returning it"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008347768", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/371", "id": 1008347768, "node_id": "IC_kwDOCGYnMM48Gip4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T18:12:30Z", "updated_at": "2022-01-09T18:12:30Z", "author_association": "OWNER", "body": "Tried this test:\r\n\r\n```python\r\n result = CliRunner().invoke(\r\n cli.cli,\r\n [\r\n \"insert\",\r\n db_path,\r\n \"rows\",\r\n \"-\",\r\n \"--convert\",\r\n 'row[\"is_chicken\"] = True',\r\n ],\r\n input='{\"name\": \"Azi\"}',\r\n )\r\n```\r\nAnd got this error:\r\n\r\n> `E + where 1 = ', 2, 30, ' return row[\"is_chicken\"] = True\\n'))>.exit_code`\r\n\r\nThe code snippet compilation isn't currently compatible with this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097128334, "label": "Support mutating row in `--convert` without returning it"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346841", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/374", "id": 1008346841, "node_id": "IC_kwDOCGYnMM48GibZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T18:06:50Z", "updated_at": "2022-01-09T18:06:50Z", "author_association": "OWNER", "body": "In addition to a unit test I manually tested all of the above, e.g.\r\n```\r\n% sqlite-utils indexes global-power-plants.db sqlite_master --fmt rst\r\n======= ============ ======= ===== ====== ====== ====== =====\r\ntable index_name seqno cid name desc coll key\r\n======= ============ ======= ===== ====== ====== ====== =====\r\n======= ============ ======= ===== ====== ====== ====== =====\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097135860, "label": "`--fmt` should imply `-t`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008346338", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/374", "id": 1008346338, "node_id": "IC_kwDOCGYnMM48GiTi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T18:03:22Z", "updated_at": "2022-01-09T18:03:22Z", "author_association": "OWNER", "body": "Commands that support `--fmt` (via the `@output_options` decorator) are:\r\n\r\n- `tables`\r\n- `views`\r\n- `query`\r\n- `memory`\r\n- `search`\r\n- `rows`\r\n- `triggers`\r\n- `indexes`\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097135860, "label": "`--fmt` should imply `-t`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008345267", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/374", "id": 1008345267, "node_id": "IC_kwDOCGYnMM48GiCz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T17:56:37Z", "updated_at": "2022-01-09T17:56:37Z", "author_association": "OWNER", "body": "Better:\r\n```python\r\nif fmt:\r\n table = True\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097135860, "label": "`--fmt` should imply `-t`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344980", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/373", "id": 1008344980, "node_id": "IC_kwDOCGYnMM48Gh-U", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T17:54:53Z", "updated_at": "2022-01-09T17:54:53Z", "author_association": "OWNER", "body": "Updated TIL: https://til.simonwillison.net/python/cog-to-update-help-in-readme#user-content-cog-for-restructuredtext", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097135732, "label": "List `--fmt` options in the docs "}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008344525", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/373", "id": 1008344525, "node_id": "IC_kwDOCGYnMM48Gh3N", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T17:52:22Z", "updated_at": "2022-01-09T17:52:22Z", "author_association": "OWNER", "body": "Updated docs: https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097135732, "label": "List `--fmt` options in the docs "}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/373#issuecomment-1008341078", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/373", "id": 1008341078, "node_id": "IC_kwDOCGYnMM48GhBW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T17:31:12Z", "updated_at": "2022-01-09T17:31:12Z", "author_association": "OWNER", "body": "Found an example of using `cog` in a rST file here: https://github.com/nedbat/coveragepy/blob/f3238eea7e403d13a217b30579b1a1c2cbff62e3/doc/dbschema.rst#L21\r\n```\r\n.. [[[cog\r\n from coverage.sqldata import SCHEMA_VERSION\r\n print(\".. code::\")\r\n print()\r\n print(f\" SCHEMA_VERSION = {SCHEMA_VERSION}\")\r\n print()\r\n.. ]]]\r\n.. code::\r\n\r\n SCHEMA_VERSION = 7\r\n\r\n.. [[[end]]]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097135732, "label": "List `--fmt` options in the docs "}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/375#issuecomment-1008338186", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/375", "id": 1008338186, "node_id": "IC_kwDOCGYnMM48GgUK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T17:13:33Z", "updated_at": "2022-01-09T17:13:54Z", "author_association": "OWNER", "body": " cat blah.csv | sqlite-utils bulk blah.db - \\\r\n \"insert into blah (:foo, :bar)\" --csv\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097251014, "label": "`sqlite-utils bulk` command"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008275546, "node_id": "IC_kwDOCGYnMM48GRBa", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-09T11:01:15Z", "updated_at": "2022-01-09T13:37:51Z", "author_association": "CONTRIBUTOR", "body": "i don\u2019t want to be such a partisan for analyze, but the query planner deciding *not* to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice.\r\n\r\nthe original poster in that stack overflow doesn\u2019t say there\u2019s a performance regression ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1574#issuecomment-1008279307", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1574", "id": 1008279307, "node_id": "IC_kwDOBm6k_c48GR8L", "user": {"value": 33631, "label": "fs111"}, "created_at": "2022-01-09T11:26:06Z", "updated_at": "2022-01-09T11:26:06Z", "author_association": "NONE", "body": "@fgregg my thinking was backwards compatibility. I don't know what people do to their builds, I just wanted a smaller image for my use case.\r\n\r\n@simonw any chance to take a look at this? If there is no interest, feel free to close the PR", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1084193403, "label": "introduce new option for datasette package to use a slim base image"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/374#issuecomment-1008252732", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/374", "id": 1008252732, "node_id": "IC_kwDOCGYnMM48GLc8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T08:25:30Z", "updated_at": "2022-01-09T08:25:30Z", "author_association": "OWNER", "body": "Need to change `if table:` to `if table or fmt:` in a few places.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097135860, "label": "`--fmt` should imply `-t`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/372#issuecomment-1008247370", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/372", "id": 1008247370, "node_id": "IC_kwDOCGYnMM48GKJK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T07:51:18Z", "updated_at": "2022-01-09T07:51:18Z", "author_association": "OWNER", "body": "Pathlib says the stem of that would be `dogs.and.cats.jpg` - best stick with that for consistency. https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.suffix\r\n\r\nIt calls the last bit `suffix` - maybe I should use that instead of `ext`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097129710, "label": "Idea: `suffix` and `stem` file columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246366", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/371", "id": 1008246366, "node_id": "IC_kwDOCGYnMM48GJ5e", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T07:42:14Z", "updated_at": "2022-01-09T07:42:14Z", "author_association": "OWNER", "body": "Also need to update relevant docs for that example.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097128334, "label": "Support mutating row in `--convert` without returning it"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/371#issuecomment-1008246239", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/371", "id": 1008246239, "node_id": "IC_kwDOCGYnMM48GJ3f", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T07:41:24Z", "updated_at": "2022-01-09T07:41:24Z", "author_association": "OWNER", "body": "Might be a case of modifying this line: https://github.com/simonw/sqlite-utils/blob/e0c476bc380744680c8b7675c24fb0e9f5ec6dcd/sqlite_utils/cli.py#L828\r\n\r\nTo:\r\n```python\r\ndocs = (fn(doc) or doc for doc in docs)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097128334, "label": "Support mutating row in `--convert` without returning it"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008234293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008234293, "node_id": "IC_kwDOCGYnMM48GG81", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T05:37:02Z", "updated_at": "2022-01-09T05:37:02Z", "author_association": "OWNER", "body": "Calling `p.stdin.close()` and then `p.wait()` terminates the subprocess.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008233910", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008233910, "node_id": "IC_kwDOCGYnMM48GG22", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T05:32:53Z", "updated_at": "2022-01-09T05:35:45Z", "author_association": "OWNER", "body": "This is strange. The following:\r\n```pycon\r\n>>> import subprocess\r\n>>> p = subprocess.Popen([\"sqlite-utils\", \"insert\", \"/tmp/stream.db\", \"stream\", \"-\", \"--nl\"], stdin=subprocess.PIPE)\r\n>>> p.stdin.write(b'\\n'.join(b'{\"id\": %s}' % str(i).encode(\"utf-8\") for i in range(1000)))\r\n11889\r\n>>> # At this point /tmp/stream.db is still 0 bytes - but if I then run this:\r\n>>> p.stdin.close()\r\n>>> # /tmp/stream.db is now 20K and contains the written data\r\n```\r\nNo wait, mystery solved - I can add `p.stdin.flush()` instead of `p.stdin.close()` and the file suddenly jumps up in size.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008232075", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008232075, "node_id": "IC_kwDOCGYnMM48GGaL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T05:13:15Z", "updated_at": "2022-01-09T05:13:56Z", "author_association": "OWNER", "body": "I think the query that will help solve this is:\r\n\r\n`explain query plan select * from ny_times_us_counties where state = 1 and county = 2`\r\n\r\nIn this case, the query planner needs to decide if it should use the index for the `state` column or the index for the `county` column. That's where the statistics come into play. In particular:\r\n\r\n| tbl | idx | stat |\r\n|----------------------|---------------------------------|---------------|\r\n| ny_times_us_counties | idx_ny_times_us_counties_date | 2092871 2915 |\r\n| ny_times_us_counties | idx_ny_times_us_counties_fips | 2092871 651 |\r\n| ny_times_us_counties | idx_ny_times_us_counties_county | 2092871 1085 |\r\n| ny_times_us_counties | idx_ny_times_us_counties_state | 2092871 37373 |\r\n\r\nThose numbers are explained by this comment in the SQLite C code: https://github.com/sqlite/sqlite/blob/5622c7f97106314719740098cf0854e7eaa81802/src/analyze.c#L41-L55\r\n\r\n```\r\n** There is normally one row per index, with the index identified by the\r\n** name in the idx column. The tbl column is the name of the table to\r\n** which the index belongs. In each such row, the stat column will be\r\n** a string consisting of a list of integers. The first integer in this\r\n** list is the number of rows in the index. (This is the same as the\r\n** number of rows in the table, except for partial indices.) The second\r\n** integer is the average number of rows in the index that have the same\r\n** value in the first column of the index.\r\n```\r\nSo that table is telling us that using a value in the `county` column will filter down to an average of 1,085 rows, whereas filtering on the `state` column will filter down to an average of 37,373 - so clearly the `county` index is the better index to use here!\r\n\r\nJust one catch: against both my` covid.db` and my `covid-analyzed.db` databases the `county` index is picked for both of them - so SQLite is somehow guessing that `county` is a better index even though it doesn't have statistics for that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008229839", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008229839, "node_id": "IC_kwDOCGYnMM48GF3P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:51:44Z", "updated_at": "2022-01-09T04:51:44Z", "author_association": "OWNER", "body": "Found one report on Stack Overflow from 9 years ago of someone seeing broken performance after running `ANALYZE`, hard to say that's a trend and not a single weird edge-case though! https://stackoverflow.com/q/12947214/6083", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008229341", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008229341, "node_id": "IC_kwDOCGYnMM48GFvd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:45:38Z", "updated_at": "2022-01-09T04:47:11Z", "author_association": "OWNER", "body": "This is probably too fancy. I think maybe the way to do this is with `select * from [global-power-plants] where \"country_long\" = 'United Kingdom'` - then mess around with stats to see if I can get it to use the index or not based on them.\r\n\r\nHere's the explain for that: https://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+*+from+[global-power-plants]+where+%22country_long%22+%3D+%27United+Kingdom%27", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008227625", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008227625, "node_id": "IC_kwDOCGYnMM48GFUp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:25:38Z", "updated_at": "2022-01-09T04:25:38Z", "author_association": "OWNER", "body": "```sql\r\nEXPLAIN QUERY PLAN select country_long, count(*) from [global-power-plants] group by country_long\r\n```\r\nhttps://global-power-plants.datasettes.com/global-power-plants?sql=EXPLAIN+QUERY+PLAN+select+country_long%2C+count%28*%29+from+%5Bglobal-power-plants%5D+group+by+country_long\r\n\r\n> SCAN TABLE global-power-plants USING COVERING INDEX \"global-power-plants_country_long\"", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1588#issuecomment-1008227436", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1588", "id": 1008227436, "node_id": "IC_kwDOBm6k_c48GFRs", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:23:37Z", "updated_at": "2022-01-09T04:25:04Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L163-L170\r\n\r\nhttps://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L195-L204", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097101917, "label": "`explain query plan select` is too strict about whitespace"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1588#issuecomment-1008227491", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1588", "id": 1008227491, "node_id": "IC_kwDOBm6k_c48GFSj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:24:09Z", "updated_at": "2022-01-09T04:24:09Z", "author_association": "OWNER", "body": "I think this is the fix:\r\n```python\r\nre.compile(r\"^explain\\s+query\\s+plan\\s+select\\b\"),\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097101917, "label": "`explain query plan select` is too strict about whitespace"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226862", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008226862, "node_id": "IC_kwDOCGYnMM48GFIu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:17:55Z", "updated_at": "2022-01-09T04:17:55Z", "author_association": "OWNER", "body": "There are some clues as to what effect ANALYZE has in https://www.sqlite.org/optoverview.html\r\n\r\nSome quotes:\r\n\r\n> SQLite might use a skip-scan on an index if it knows that the first one or more columns contain many duplication values. If there are too few duplicates in the left-most columns of the index, then it would be faster to simply step ahead to the next value, and thus do a full table scan, than to do a binary search on an index to locate the next left-column value.\r\n>\r\n> The only way that SQLite can know that there are many duplicates in the left-most columns of an index is if the ANALYZE command has been run on the database. Without the results of ANALYZE, SQLite has to guess at the \"shape\" of the data in the table, and the default guess is that there are an average of 10 duplicates for every value in the left-most column of the index. Skip-scan only becomes profitable (it only gets to be faster than a full table scan) when the number of duplicates is about 18 or more. Hence, a skip-scan is never used on a database that has not been analyzed. \r\n\r\nAnd\r\n\r\n> Join reordering is automatic and usually works well enough that programmers do not have to think about it, especially if ANALYZE has been used to gather statistics about the available indexes, though occasionally some hints from the programmer are needed.\r\n\r\nAnd\r\n\r\n> The various sqlite_statN tables contain information on how selective the various indexes are. For example, the sqlite_stat1 table might indicate that an equality constraint on column x reduces the search space to 10 rows on average, whereas an equality constraint on column y reduces the search space to 3 rows on average. In that case, SQLite would prefer to use index ex2i2 since that index is more selective. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008226487", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008226487, "node_id": "IC_kwDOCGYnMM48GFC3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T04:14:05Z", "updated_at": "2022-01-09T04:14:05Z", "author_association": "OWNER", "body": "Didn't manage to spot a meaningful difference with that database either:\r\n```\r\nanalyze % python3 -m timeit '__import__(\"sqlite3\").connect(\"covid.db\").execute(\"select fips, count(*) from [ny_times_us_counties] group by fips\").fetchall()' \r\n2 loops, best of 5: 101 msec per loop\r\nanalyze % python3 -m timeit '__import__(\"sqlite3\").connect(\"covid-analyzed.db\").execute(\"select fips, count(*) from [ny_times_us_counties] group by fips\").fetchall()'\r\n2 loops, best of 5: 103 msec per loop\r\n```\r\nMaybe `select fips, count(*) from [ny_times_us_counties] group by fips` isn't a good query for testing this?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008220270", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008220270, "node_id": "IC_kwDOCGYnMM48GDhu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:12:38Z", "updated_at": "2022-01-09T03:13:15Z", "author_association": "OWNER", "body": "Basically no difference using this very basic benchmark:\r\n```\r\nanalyze % python3 -m timeit '__import__(\"sqlite3\").connect(\"global-power-plants.db\").execute(\"select country_long, count(*) from [global-power-plants] group by country_long\").fetchall()'\r\n100 loops, best of 5: 2.39 msec per loop\r\nanalyze % python3 -m timeit '__import__(\"sqlite3\").connect(\"global-power-plants-analyzed.db\").execute(\"select country_long, count(*) from [global-power-plants] group by country_long\").fetchall()'\r\n100 loops, best of 5: 2.38 msec per loop\r\n```\r\nI should try this against a much larger database.\r\n\r\nhttps://covid-19.datasettes.com/covid.db is 879MB.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219844", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008219844, "node_id": "IC_kwDOCGYnMM48GDbE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:08:09Z", "updated_at": "2022-01-09T03:08:09Z", "author_association": "OWNER", "body": "```\r\nanalyze % sqlite-utils global-power-plants-analyzed.db 'analyze'\r\n[{\"rows_affected\": -1}]\r\nanalyze % sqlite-utils tables global-power-plants-analyzed.db \r\n[{\"table\": \"global-power-plants\"},\r\n {\"table\": \"global-power-plants_fts\"},\r\n {\"table\": \"global-power-plants_fts_data\"},\r\n {\"table\": \"global-power-plants_fts_idx\"},\r\n {\"table\": \"global-power-plants_fts_docsize\"},\r\n {\"table\": \"global-power-plants_fts_config\"},\r\n {\"table\": \"sqlite_stat1\"}]\r\nanalyze % sqlite-utils rows global-power-plants-analyzed.db sqlite_stat1 -t\r\ntbl idx stat\r\n------------------------------- ---------------------------------- ---------\r\nglobal-power-plants_fts_config global-power-plants_fts_config 1 1\r\nglobal-power-plants_fts_docsize 33643\r\nglobal-power-plants_fts_idx global-power-plants_fts_idx 199 40 1\r\nglobal-power-plants_fts_data 136\r\nglobal-power-plants \"global-power-plants_owner\" 33643 4\r\nglobal-power-plants \"global-power-plants_country_long\" 33643 202\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219588", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008219588, "node_id": "IC_kwDOCGYnMM48GDXE", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:06:42Z", "updated_at": "2022-01-09T03:06:42Z", "author_association": "OWNER", "body": "```\r\nanalyze % sqlite-utils indexes global-power-plants.db -t \r\ntable index_name seqno cid name desc coll key\r\n------------------------------ ------------------------------------------------- ------- ----- ------------ ------ ------ -----\r\nglobal-power-plants \"global-power-plants_owner\" 0 12 owner 0 BINARY 1\r\nglobal-power-plants \"global-power-plants_country_long\" 0 1 country_long 0 BINARY 1\r\nglobal-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 0 0 segid 0 BINARY 1\r\nglobal-power-plants_fts_idx sqlite_autoindex_global-power-plants_fts_idx_1 1 1 term 0 BINARY 1\r\nglobal-power-plants_fts_config sqlite_autoindex_global-power-plants_fts_config_1 0 0 k 0 BINARY 1\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219484", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008219484, "node_id": "IC_kwDOCGYnMM48GDVc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:05:44Z", "updated_at": "2022-01-09T03:05:44Z", "author_association": "OWNER", "body": "I'll start by running some experiments against the 11MB database file from https://global-power-plants.datasettes.com/global-power-plants.db", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/369#issuecomment-1008219191", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/369", "id": 1008219191, "node_id": "IC_kwDOCGYnMM48GDQ3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T03:03:53Z", "updated_at": "2022-01-09T03:03:53Z", "author_association": "OWNER", "body": "Refs:\r\n- #366\r\n- #365", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097091527, "label": "Research how much of a difference analyze / sqlite_stat1 makes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163585", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008163585, "node_id": "IC_kwDOCGYnMM48F1sB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T22:14:39Z", "updated_at": "2022-01-09T03:03:07Z", "author_association": "OWNER", "body": "The reason I'm hesitating on this is that I've not actually used ANALYZE at all in nearly five years of messing around with SQLite! So I'm nervous that there are surprise downsides I haven't thought of.\r\n\r\nMy hunch is that ANALYZE is only worth worrying about on much larger databases, in which case I'm OK supporting it as a thoroughly documented power-user feature rather than a default.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216371", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/368", "id": 1008216371, "node_id": "IC_kwDOCGYnMM48GCkz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T02:36:22Z", "updated_at": "2022-01-09T02:36:22Z", "author_association": "OWNER", "body": "In Python 3.6: https://docs.python.org/3.6/library/subprocess.html\r\n\r\n> This does not capture stdout or stderr by default. To do so, pass [`PIPE`](https://docs.python.org/3.6/library/subprocess.html#subprocess.PIPE \"subprocess.PIPE\") for the *stdout* and/or *stderr* arguments.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097087280, "label": "Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008216271", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/368", "id": 1008216271, "node_id": "IC_kwDOCGYnMM48GCjP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T02:35:09Z", "updated_at": "2022-01-09T02:35:09Z", "author_association": "OWNER", "body": "Test failure on Python 3.6:\r\n\r\n> `E TypeError: __init__() got an unexpected keyword argument 'capture_output'`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097087280, "label": "Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/367#issuecomment-1008158799", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/367", "id": 1008158799, "node_id": "IC_kwDOCGYnMM48F0hP", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-01-08T21:36:55Z", "updated_at": "2022-01-09T02:34:44Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#367](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9848eaa) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a8f9cc6f64f299830834428509940d448b82b4ed?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a8f9cc6) will **decrease** coverage by `0.20%`.\n> The diff coverage is `50.00%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/367/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #367 +/- ##\n==========================================\n- Coverage 96.44% 96.24% -0.21% \n==========================================\n Files 5 6 +1 \n Lines 2307 2317 +10 \n==========================================\n+ Hits 2225 2230 +5 \n- Misses 82 87 +5 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.15% <28.57%> (-0.42%)` | :arrow_down: |\n| [sqlite\\_utils/\\_\\_main\\_\\_.py](https://codecov.io/gh/simonw/sqlite-utils/pull/367/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL19fbWFpbl9fLnB5) | `100.00% <100.00%> (\u00f8)` | |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a8f9cc6...9848eaa](https://codecov.io/gh/simonw/sqlite-utils/pull/367?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097041471, "label": "Initial prototype of .analyze() methods"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008216201", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008216201, "node_id": "IC_kwDOCGYnMM48GCiJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T02:34:12Z", "updated_at": "2022-01-09T02:34:12Z", "author_association": "OWNER", "body": "I can now write tests that look like this: https://github.com/simonw/sqlite-utils/blob/539f5ccd90371fa87f946018f8b77d55929e06db/tests/test_cli.py#L2024-L2030\r\n\r\nWhich means I can write a test that exercises this bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/368#issuecomment-1008215912", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/368", "id": 1008215912, "node_id": "IC_kwDOCGYnMM48GCdo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T02:30:59Z", "updated_at": "2022-01-09T02:30:59Z", "author_association": "OWNER", "body": "Even better, inspired by `rich`, support `python -m sqlite_utils`. https://github.com/Textualize/rich/blob/master/rich/__main__.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097087280, "label": "Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008214998, "node_id": "IC_kwDOCGYnMM48GCPW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T02:23:20Z", "updated_at": "2022-01-09T02:23:20Z", "author_association": "OWNER", "body": "Possible way of running the test: add this to `sqlite_utils/cli.py`:\r\n\r\n```python\r\nif __name__ == \"__main__\":\r\n cli()\r\n```\r\nNow the tool can be run using `python -m sqlite_utils.cli --help`\r\n\r\nThen in the test use `subprocess` to call `sys.executable` (the path to the current Python interpreter) and pass it `-m sqlite_utils.cli` to run the script!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214406", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008214406, "node_id": "IC_kwDOCGYnMM48GCGG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-09T02:18:21Z", "updated_at": "2022-01-09T02:18:21Z", "author_association": "OWNER", "body": "I'm having trouble figuring out the best way to write a unit test for this. Filed a relevant feature request for Click here:\r\n- https://github.com/pallets/click/issues/2171", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008166084", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008166084, "node_id": "IC_kwDOCGYnMM48F2TE", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-08T22:32:47Z", "updated_at": "2022-01-08T22:32:47Z", "author_association": "CONTRIBUTOR", "body": "or using \u201c pragma optimize\u201d", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164786", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008164786, "node_id": "IC_kwDOCGYnMM48F1-y", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-08T22:24:19Z", "updated_at": "2022-01-08T22:24:19Z", "author_association": "CONTRIBUTOR", "body": "the out-of-date scenario you describe could be addressed by automatically adding an analyze to the insert or convert commands if they implicate an index", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164116", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008164116, "node_id": "IC_kwDOCGYnMM48F10U", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-08T22:18:57Z", "updated_at": "2022-01-08T22:18:57Z", "author_association": "CONTRIBUTOR", "body": "the table with the query ran so bad was about 50k. \r\n\r\ni think the scenario should not be worse than no stats. \r\n\r\ni also did not know that sqlite was so different from postgres and needed an explicit analyze call.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008163050, "node_id": "IC_kwDOCGYnMM48F1jq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T22:10:51Z", "updated_at": "2022-01-08T22:10:51Z", "author_association": "OWNER", "body": "Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it?\r\n\r\nImagine the following sequence of events:\r\n\r\n- User imports a few records, creating the table, using `sqlite-utils insert`\r\n- User runs `sqlite-utils create-index ...` which also creates and populates the `sqlite_stat1` table\r\n- User runs `insert` again to populate several million new records\r\n\r\nThe user now has a database file with several million records and a statistics table that is wildly out of date, having been populated when they only had a few.\r\n\r\nWill this result in surprisingly bad query performance compared to it that statistics table did not exist at all?\r\n\r\nIf so, I lean much harder towards `ANALYZE` as a strictly opt-in optimization, maybe with the `--analyze` option added to `sqlite-utils insert` top to help users opt in to updating their statistics after running big inserts.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008161965", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008161965, "node_id": "IC_kwDOCGYnMM48F1St", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-08T22:02:56Z", "updated_at": "2022-01-08T22:02:56Z", "author_association": "CONTRIBUTOR", "body": "for options 2 and 3, i would worry about discoverablity. \r\n\r\nin other db\u2019s it is not necessary to explicitly call analyze for most indices. ie for postgres\r\n\r\n> The system regularly collects statistics on all of a table's columns. Newly-created non-expression indexes can immediately use these statistics to determine an index's usefulness.\r\n\r\ni suppose i would propose raising a warning if the stats table is created that explains what is going on and informs users about a \u2014no-analyze argument.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008158616", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1008158616, "node_id": "IC_kwDOCGYnMM48F0eY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T21:35:32Z", "updated_at": "2022-01-08T21:35:32Z", "author_association": "OWNER", "body": "Built a prototype in a branch, see #367.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008158357", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1008158357, "node_id": "IC_kwDOCGYnMM48F0aV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T21:33:07Z", "updated_at": "2022-01-08T21:33:07Z", "author_association": "OWNER", "body": "The one thing that worries me a little bit about doing this by default is that it adds a surprising new table to the database - it may be confusing to users if they run `create-index` and their database suddenly has a new `sqlite_stat1` table, see https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132\r\n\r\nOptions here are:\r\n\r\n- Do it anyway. People can tolerate a surprise table appearing when they create an index.\r\n- Only run `ANALYZE` if the user says `sqlite-utils create-index ... --analyze`\r\n- Use the `--analyze` option, but also automatically run `ANALYZE` if they create an index and the database they are working with already has a `sqlite_stat1` table\r\n\r\nI'm currently leading towards that third option - @fgregg any thoughts?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1587#issuecomment-1008157998", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1587", "id": 1008157998, "node_id": "IC_kwDOBm6k_c48F0Uu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T21:29:54Z", "updated_at": "2022-01-08T21:29:54Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/datasette/blob/00a2895cd2dc42c63846216b36b2dc9f41170129/datasette/database.py#L339-L354", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097040427, "label": "Add `sqlite_stat1`(-4) tables to hidden table list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1587#issuecomment-1008157908", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1587", "id": 1008157908, "node_id": "IC_kwDOBm6k_c48F0TU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T21:29:06Z", "updated_at": "2022-01-08T21:29:06Z", "author_association": "OWNER", "body": "Depending on the SQLite version (and compile options) that ran `ANALYZE` these can be called:\r\n\r\n- `sqlite_stat1`\r\n- `sqlite_stat2`\r\n- `sqlite_stat3`\r\n- `sqlite_stat4`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1097040427, "label": "Add `sqlite_stat1`(-4) tables to hidden table list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1008157132", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1008157132, "node_id": "IC_kwDOCGYnMM48F0HM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T21:23:08Z", "updated_at": "2022-01-08T21:25:05Z", "author_association": "OWNER", "body": "Running `ANALYZE` creates a new visible table called `sqlite_stat1`: https://www.sqlite.org/fileformat.html#the_sqlite_stat1_table\r\n\r\nThis should be added to the default list of hidden tables in Datasette.\r\n\r\nIt looks something like this:\r\n\r\n| tbl | idx | stat |\r\n|---------------------------------|------------------------------------|-----------|\r\n| _counts | sqlite_autoindex__counts_1 | 5 1 |\r\n| global-power-plants_fts_config | global-power-plants_fts_config | 1 1 |\r\n| global-power-plants_fts_docsize | | 33643 |\r\n| global-power-plants_fts_idx | global-power-plants_fts_idx | 199 40 1 |\r\n| global-power-plants_fts_data | | 136 |\r\n| global-power-plants | \"global-power-plants_owner\" | 33643 4 |\r\n| global-power-plants | \"global-power-plants_country_long\" | 33643 202 |\r\n\r\n> In each such row, the sqlite_stat.stat column will be a string consisting of a list of integers followed by zero or more arguments. The first integer in this list is the approximate number of rows in the index. (The number of rows in the index is the same as the number of rows in the table, except for partial indexes.) The second integer is the approximate number of rows in the index that have the same value in the first column of the index. The third integer is the number number of rows in the index that have the same value for the first two columns. The N-th integer (for N>1) is the estimated average number of rows in the index which have the same value for the first N-1 columns. For a K-column index, there will be K+1 integers in the stat column. If the index is unique, then the last integer will be 1. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008155916", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008155916, "node_id": "IC_kwDOCGYnMM48Fz0M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T21:16:46Z", "updated_at": "2022-01-08T21:16:46Z", "author_association": "OWNER", "body": "No, `chunks()` seems to work OK in the test I just added.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008154873", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008154873, "node_id": "IC_kwDOCGYnMM48Fzj5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T21:11:55Z", "updated_at": "2022-01-08T21:11:55Z", "author_association": "OWNER", "body": "I'm suspicious that the `chunks()` utility function may not be working correctly:\r\n```pycon\r\nIn [10]: [list(d) for d in list(chunks('abc', 5))]\r\nOut[10]: [['a'], ['b'], ['c']]\r\n\r\nIn [11]: [list(d) for d in list(chunks('abcdefghi', 5))]\r\nOut[11]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']]\r\n\r\nIn [12]: [list(d) for d in list(chunks('abcdefghi', 3))]\r\nOut[12]: [['a'], ['b'], ['c'], ['d'], ['e'], ['f'], ['g'], ['h'], ['i']]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008153586", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008153586, "node_id": "IC_kwDOCGYnMM48FzPy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T21:06:15Z", "updated_at": "2022-01-08T21:06:15Z", "author_association": "OWNER", "body": "I added a print statement after `for query, params in queries_and_params` and confirmed that something in the code is waiting until 16 records are available to be inserted and then executing the inserts, even with `--batch-size 1`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008151884", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008151884, "node_id": "IC_kwDOCGYnMM48Fy1M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T20:59:21Z", "updated_at": "2022-01-08T20:59:21Z", "author_association": "OWNER", "body": "(That Heroku example doesn't record the timestamp, which limits its usefulness)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008143248", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008143248, "node_id": "IC_kwDOCGYnMM48FwuQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T20:34:12Z", "updated_at": "2022-01-08T20:34:12Z", "author_association": "OWNER", "body": "Built that tool: https://github.com/simonw/stream-delay and https://pypi.org/project/stream-delay/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008129841", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/364", "id": 1008129841, "node_id": "IC_kwDOCGYnMM48Ftcx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-08T20:04:42Z", "updated_at": "2022-01-08T20:04:42Z", "author_association": "OWNER", "body": "It would be easier to test this if I had a utility for streaming out a file one line at a time.\r\n\r\nA few recipes for this in https://superuser.com/questions/526242/cat-file-to-terminal-at-particular-speed-of-lines-per-second - I'm going to build a quick `stream-delay` tool though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1095570074, "label": "`--batch-size 1` doesn't seem to commit for every item"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1574#issuecomment-1007844190", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1574", "id": 1007844190, "node_id": "IC_kwDOBm6k_c48Ente", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-08T00:42:12Z", "updated_at": "2022-01-08T00:42:12Z", "author_association": "CONTRIBUTOR", "body": "is there a reason to not always use the slim option?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1084193403, "label": "introduce new option for datasette package to use a slim base image"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007643254", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1007643254, "node_id": "IC_kwDOCGYnMM48D2p2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-07T18:37:56Z", "updated_at": "2022-01-07T18:37:56Z", "author_association": "OWNER", "body": "Or I could leave off `--no-analyze` and tell people that if they want to add an index without running analyze they can execute the `CREATE INDEX` themselves.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007642831", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1007642831, "node_id": "IC_kwDOCGYnMM48D2jP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-07T18:37:18Z", "updated_at": "2022-01-07T18:37:18Z", "author_association": "OWNER", "body": "After implementing #366 I can make it so `sqlite-utils create-index` automatically runs `db.analyze(index_name)` afterwards, maybe with a `--no-analyze` option in case anyone wants to opt out of that for specific performance reasons.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007641634", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1007641634, "node_id": "IC_kwDOCGYnMM48D2Qi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-07T18:35:35Z", "updated_at": "2022-01-07T18:35:35Z", "author_association": "OWNER", "body": "Since the existing CLI feature is this:\r\n\r\n $ sqlite-utils analyze-tables github.db tags\r\n\r\nI can add `sqlite-utils analyze` to reflect the Python library method.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007639860", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1007639860, "node_id": "IC_kwDOCGYnMM48D100", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-07T18:32:59Z", "updated_at": "2022-01-07T18:33:07Z", "author_association": "OWNER", "body": "From the SQLite docs:\r\n\r\n> If no arguments are given, all attached databases are analyzed. If a schema name is given as the argument, then all tables and indices in that one database are analyzed. If the argument is a table name, then only that table and the indices associated with that table are analyzed. If the argument is an index name, then only that one index is analyzed.\r\n\r\nSo I think this becomes two methods:\r\n\r\n- `db.analyze()` calls analyze on the whole database\r\n- `db.analyze(name_of_table_or_index)` for a specific named table or index\r\n- `table.analyze()` is a shortcut for `db.analyze(table.name)`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1007637963", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/366", "id": 1007637963, "node_id": "IC_kwDOCGYnMM48D1XL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-07T18:30:13Z", "updated_at": "2022-01-07T18:30:13Z", "author_association": "OWNER", "body": "Annoyingly I use the word \"analyze\" to mean something else in the CLI - for these features:\r\n\r\n- #207 \r\n- #320\r\n\r\nthere's only one method with a similar name in the Python library though and that's this one:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/6e46b9913411682f3a3ec66f4d58886c1db8654b/sqlite_utils/db.py#L2904-L2906", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096563265, "label": "Python library methods for calling ANALYZE"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007636709", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1007636709, "node_id": "IC_kwDOCGYnMM48D1Dl", "user": {"value": 536941, "label": "fgregg"}, "created_at": "2022-01-07T18:28:33Z", "updated_at": "2022-01-07T18:29:43Z", "author_association": "CONTRIBUTOR", "body": "i added an index to one table with sqlite-utils, and then a query that used to take about 1 second started taking hundreds of seconds. \r\n\r\nrunning analyze got me back to sub second speed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007634999", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1007634999, "node_id": "IC_kwDOCGYnMM48D0o3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-07T18:26:22Z", "updated_at": "2022-01-07T18:26:22Z", "author_association": "OWNER", "body": "I've not used the `ANALYZE` feature in SQLite at all before. Should probably add Python library methods for it.\r\n\r\nAnnoyingly I use the word \"analyze\" to mean something else in the CLI - for these features:\r\n- #207 \r\n- #320", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/365", "id": 1007633376, "node_id": "IC_kwDOCGYnMM48D0Pg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-07T18:24:07Z", "updated_at": "2022-01-07T18:24:07Z", "author_association": "OWNER", "body": "Relevant documentation: https://www.sqlite.org/lang_analyze.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1096558279, "label": "create-index should run analyze after creating index"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/pull/36#issuecomment-1006708046", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/36", "id": 1006708046, "node_id": "IC_kwDOD079W848ASVO", "user": {"value": 71983, "label": "scoates"}, "created_at": "2022-01-06T16:04:46Z", "updated_at": "2022-01-06T16:04:46Z", "author_association": "NONE", "body": "This one got me, today, too. \ud83d\udc4d", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 988493790, "label": "Correct naming of tool in readme"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006344080", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/363", "id": 1006344080, "node_id": "IC_kwDOCGYnMM47-5eQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T07:32:05Z", "updated_at": "2022-01-06T07:32:05Z", "author_association": "OWNER", "body": "As part of this work I should add test coverage of this error message too: https://github.com/simonw/sqlite-utils/blob/413f8ed754e38d7b190de888c85fe8438336cb11/sqlite_utils/cli.py#L826", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094981339, "label": "Better error message if `--convert` code fails to return a dict"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/363#issuecomment-1006343303", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/363", "id": 1006343303, "node_id": "IC_kwDOCGYnMM47-5SH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T07:30:20Z", "updated_at": "2022-01-06T07:30:20Z", "author_association": "OWNER", "body": "This check should run inside the `.insert_all()` method. It should raise a custom exception which the CLI code can then catch and turn into a click error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094981339, "label": "Better error message if `--convert` code fails to return a dict"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318443", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 1006318443, "node_id": "IC_kwDOCGYnMM47-zNr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T06:30:13Z", "updated_at": "2022-01-06T06:30:13Z", "author_association": "OWNER", "body": "Documentation:\r\n\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-unstructured-data-with-lines-and-text\r\n- https://sqlite-utils.datasette.io/en/latest/cli.html#applying-conversions-while-inserting-data", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/356#issuecomment-1006318007", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/356", "id": 1006318007, "node_id": "IC_kwDOCGYnMM47-zG3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T06:28:53Z", "updated_at": "2022-01-06T06:28:53Z", "author_association": "OWNER", "body": "Implemented in #361.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1077431957, "label": "`sqlite-utils insert --convert` option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006219956", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006219956, "node_id": "IC_kwDOCGYnMM47-bK0", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-01-06T01:51:54Z", "updated_at": "2022-01-06T06:22:25Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\n> Merging [#361](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b7f0b88) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/f3fd8613113d21d44238a6ec54b375f5aa72c4e0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (f3fd861) will **decrease** coverage by `0.05%`.\n> The diff coverage is `92.85%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/361/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n```diff\n@@ Coverage Diff @@\n## main #361 +/- ##\n==========================================\n- Coverage 96.49% 96.44% -0.06% \n==========================================\n Files 5 5 \n Lines 2283 2306 +23 \n==========================================\n+ Hits 2203 2224 +21 \n- Misses 80 82 +2 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.49% <92.00%> (-0.11%)` | :arrow_down: |\n| [sqlite\\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/361/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.23% <100.00%> (\u00f8)` | |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [f3fd861...b7f0b88](https://codecov.io/gh/simonw/sqlite-utils/pull/361?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/361#issuecomment-1006315145", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/361", "id": 1006315145, "node_id": "IC_kwDOCGYnMM47-yaJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-01-06T06:20:51Z", "updated_at": "2022-01-06T06:20:51Z", "author_association": "OWNER", "body": "This is all documented. I'm going to rebase-merge it to keep the individual commits.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1094890366, "label": "--lines and --text and --convert and --import"}, "performed_via_github_app": null}