{"id": 1041778507, "node_id": "I_kwDOCGYnMM4-GEdL", "number": 334, "title": "Filter by datetime objects using rows_where()", "user": {"value": 11642379, "label": "viseshrp"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-02T00:44:08Z", "updated_at": "2021-11-13T19:23:21Z", "closed_at": "2021-11-13T19:23:21Z", "author_association": "NONE", "pull_request": null, "body": "Firstly, thanks for this nice utility. \r\nIt would be nice to have an example in the docs on how to filter by date range using `rows_where()`. \r\nThis doesn't seem to work:\r\n```\r\ntable.rows_where('datetime(created) between datetime(\"2021-10-31T17:29:59.277428-04:00\") AND datetime(\"2021-11-01T03:44:04.544651+00:00\")')\r\n```\r\n\r\n\r\nI could probably just use `db.query()`, which works for the above, but it would be nice if I could pass in `datetime` objects in `rows_where()`.\r\nThanks.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/334/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1042569687, "node_id": "I_kwDOCGYnMM4-JFnX", "number": 335, "title": "sqlite-utils index-foreign-keys fails due to pre-existing index", "user": {"value": 596279, "label": "zaneselvans"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2021-11-02T16:22:11Z", "updated_at": "2021-11-14T22:55:56Z", "closed_at": "2021-11-14T22:55:56Z", "author_association": "NONE", "pull_request": null, "body": "While running the command:\r\n```sh\r\nsqlite-utils index-foreign-keys $SQLITE_DIR/pudl.sqlite\r\n```\r\n\r\nI got the following error:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/home/zane/miniconda3/envs/pudl-dev/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/cli.py\", line 454, in index_foreign_keys\r\n db.index_foreign_keys()\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py\", line 902, in index_foreign_keys\r\n table.create_index([fk.column])\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1563, in create_index\r\n self.db.execute(sql)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py\", line 421, in execute\r\n return self.conn.execute(sql)\r\nsqlite3.OperationalError: index idx_generators_eia860_report_date already exists\r\n```\r\n\r\nThis DB was created with the foreign key constraint `PRAGMA` enabled and a bunch of column-level `CHECK` constraints. Is this an expected behavior? Should one not try to index foreign keys if FK constraints are already being enforced within the DB?\r\n\r\nI'm also noticing that the size of the DB after FK indexes have been added went from 483MB to 835MB, which seems like a much bigger jump than when I've done this previously.\r\n\r\nSoftware versions...\r\n* sqlite-utils 3.17.1\r\n* sqlite 3.36.0\r\n* SQLAlchemy 1.4.26 (used to create the DB)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/335/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1044267332, "node_id": "I_kwDOCGYnMM4-PkFE", "number": 336, "title": "sqlite-util tranform --column-order mangles columns of type \"timestamp\"", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-11-04T01:15:38Z", "updated_at": "2023-05-08T21:13:38Z", "closed_at": "2023-05-08T21:13:38Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Reproducible code below:\r\n\r\n```bash\r\n> echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db\r\n> sqlite3 foo.db\r\nSQLite version 3.36.0 2021-06-18 18:36:39\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP);\r\nsqlite> .exit\r\n> sqlite-utils transform foo.db bar --column-order baz\r\nsqlite3 foo.db\r\nSQLite version 3.36.0 2021-06-18 18:36:39\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE IF NOT EXISTS \"bar\" (\r\n [baz] TEXT,\r\n [created_at] FLOAT DEFAULT 'CURRENT_TIMESTAMP'\r\n);\r\nsqlite> .exit\r\n> sqlite-utils transform foo.db bar --column-order baz\r\n> sqlite3 foo.db\r\nSQLite version 3.36.0 2021-06-18 18:36:39\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE IF NOT EXISTS \"bar\" (\r\n [baz] TEXT,\r\n [created_at] FLOAT DEFAULT '''CURRENT_TIMESTAMP'''\r\n);\r\n```\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/336/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1053087862, "node_id": "I_kwDOCGYnMM4-xNh2", "number": 338, "title": "dict, list, tuple should all map to TEXT", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-15T00:28:01Z", "updated_at": "2021-11-15T00:36:03Z", "closed_at": "2021-11-15T00:36:03Z", "author_association": "OWNER", "pull_request": null, "body": "> This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947\r\n>\r\n> So the `COLUMN_TYPE_MAPPING` should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/322#issuecomment-968401459_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/338/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1053122092, "node_id": "I_kwDOCGYnMM4-xV4s", "number": 339, "title": "`table.lookup()` option to populate additional columns when creating a record", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-15T01:41:17Z", "updated_at": "2021-11-15T02:02:34Z", "closed_at": "2021-11-15T02:02:00Z", "author_association": "OWNER", "pull_request": null, "body": "> For the commits table I feel like I want a version of `table.lookup()` that can be passed additional columns to populate only if the record does not exist yet.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/git-history/issues/12#issuecomment-967455017_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/339/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1053136495, "node_id": "I_kwDOCGYnMM4-xZZv", "number": 341, "title": "`hash_id: Optional[Any]` should be `hash_id: Optional[str]`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-15T02:12:39Z", "updated_at": "2021-11-15T02:19:31Z", "closed_at": "2021-11-15T02:19:31Z", "author_association": "OWNER", "pull_request": null, "body": "In a few places:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L642\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L751\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L1049\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L1230\r\n\r\nBut it's correct here:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L2470", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/341/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1004613267, "node_id": "I_kwDOCGYnMM474S6T", "number": 328, "title": "Invalid JSON output when no rows", "user": {"value": 12752, "label": "gravis"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-09-22T18:37:26Z", "updated_at": "2021-09-22T20:21:34Z", "closed_at": "2021-09-22T20:20:18Z", "author_association": "NONE", "pull_request": null, "body": "`sqlite-utils query` generates a JSON output with the result from the query:\r\n\r\n```json\r\n[{...},{...}]\r\n```\r\nIf no rows are returned by the query, I'm expecting an empty JSON array:\r\n\r\n```json\r\n[]\r\n```\r\n\r\nBut actually I'm getting an empty string. To be consistent, the output should be `[]` when the request succeeds (return code == `0`).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/328/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1005891028, "node_id": "I_kwDOCGYnMM479K3U", "number": 329, "title": "Rethink approach to [ and ] in column names (currently throws error)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2021-09-23T22:14:24Z", "updated_at": "2021-11-15T02:57:51Z", "closed_at": "2021-11-15T02:57:51Z", "author_association": "OWNER", "pull_request": null, "body": "> I think it's best to still keep `[` and `]` out of column names though. Transforming them into `(` and `)` seems reasonable - but should that happen here or in `sqlite-utils`? I think in `sqlite-utils`.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-app/issues/121#issuecomment-926200398_\r\n\r\nThis is a rethinking of the solution to:\r\n\r\n- https://github.com/simonw/sqlite-utils/issues/86", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/329/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1026794056, "node_id": "I_kwDOCGYnMM49M6JI", "number": 331, "title": "Mypy error: found module but no type hints or library stubs", "user": {"value": 53032010, "label": "andreaslongo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-10-14T20:29:50Z", "updated_at": "2021-11-14T23:21:08Z", "closed_at": "2021-11-14T23:21:08Z", "author_association": "NONE", "pull_request": null, "body": "```\r\nPython 3.9.5\r\nmypy 0.910\r\nsqlite-utils 3.17.1\r\n```\r\n\r\nWhile using sqlite-utils as a library, when I use mypy for static type checking, it throws an error:\r\n\r\n```\r\nmypy .\r\nsrc/etl.py:5: error: Skipping analyzing \"sqlite_utils\": found module but no type hints or library stubs\r\n import sqlite_utils\r\n ^\r\nsrc/etl.py:5: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports\r\ntest/test_etl.py:4: error: Skipping analyzing \"sqlite_utils\": found module but no type hints or library stubs\r\n import sqlite_utils\r\n ^\r\nFound 2 errors in 2 files (checked 7 source files)\r\n```\r\n\r\n\r\nWhen I add a `py.typed` file to the sqlite-utils package to mark it as PEP 561 compatible, the error goes away.\r\n\r\n```\r\nal@nbal ..b/python3.9/site-packages/sqlite_utils (git)-[main] % la\r\ntotal 200\r\ndrwx------ 3 al al 4096 Oct 14 22:00 .\r\ndrwx------ 117 al al 4096 Oct 12 21:12 ..\r\n-rw------- 1 al al 64409 Oct 12 21:11 cli.py\r\n-rw------- 1 al al 109092 Oct 12 21:11 db.py\r\n-rw------- 1 al al 0 Oct 14 22:00 py.typed\r\n-rw------- 1 al al 684 Oct 12 21:11 recipes.py\r\n-rw------- 1 al al 7988 Oct 12 21:11 utils.py\r\n-rw------- 1 al al 113 Oct 12 21:11 __init__.py\r\n```\r\n\r\nI would like to suggest adding a `py.typed` file to the repository.\r\n\r\nSee also the mypy docs on creating PEP 561 compatible packages:\r\nhttps://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/331/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1028056713, "node_id": "I_kwDOCGYnMM49RuaJ", "number": 332, "title": "`sqlite-utils memory --flatten` option to flatten nested JSON", "user": {"value": 22523840, "label": "rdtq"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-10-16T14:04:42Z", "updated_at": "2021-11-14T23:05:05Z", "closed_at": "2021-11-14T23:05:05Z", "author_association": "NONE", "pull_request": null, "body": "currently --flatten option works only for `insert` command, it would be cool if it worked for `memory` as well to query nested json", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/332/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1072780607, "node_id": "I_kwDOCGYnMM4_8VU_", "number": 351, "title": "Support `--import xml.etree.ElementTree` in `sqlite-utils convert`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-07T00:40:29Z", "updated_at": "2021-12-11T00:11:25Z", "closed_at": "2021-12-11T00:11:25Z", "author_association": "OWNER", "pull_request": null, "body": "It's not possible to use a module that requires a nested import, such as `xml.etree.ElementTree`, at the moment. I found and fixed this bug in `git-history`, I should replicate that fix (and accompanying documentation) here: https://github.com/simonw/git-history/issues/39", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/351/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1058196641, "node_id": "I_kwDOCGYnMM4_Esyh", "number": 342, "title": "Extra options to `lookup()` which get passed to `insert()`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2021-11-19T06:53:03Z", "updated_at": "2021-11-19T07:26:54Z", "closed_at": "2021-11-19T07:26:54Z", "author_association": "OWNER", "pull_request": null, "body": "For https://github.com/simonw/git-history/issues/12 I found myself wanting to pass extra options to `lookup()` to set the column order, primary key etc.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/342/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1063388037, "node_id": "I_kwDOCGYnMM4_YgOF", "number": 343, "title": "Provide function to generate hash_id from specified columns", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-25T10:12:12Z", "updated_at": "2022-03-02T04:25:25Z", "closed_at": "2022-03-02T04:25:25Z", "author_association": "NONE", "pull_request": null, "body": "Hi\r\n\r\nI note that you define `_hash()` to create a `hash_id` from non-id column values in a table [here](https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2996).\r\n\r\nIt would be useful to be able to call a complementary function to generate a corresponding `_id` from a subset of specified columns when adding items to another table, eg to support the creation of foreign keys.\r\n\r\nOr is there a better pattern for doing that?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/343/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1066474200, "node_id": "I_kwDOCGYnMM4_kRrY", "number": 344, "title": "Support STRICT tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 14, "created_at": "2021-11-29T20:32:23Z", "updated_at": "2023-12-08T05:22:39Z", "closed_at": "2023-12-08T05:22:39Z", "author_association": "OWNER", "pull_request": null, "body": "New in SQLite 3.37.0, released a few days ago: https://www.sqlite.org/stricttables.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/344/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1066501534, "node_id": "I_kwDOCGYnMM4_kYWe", "number": 345, "title": "`table.strict` introspection boolean for identifying STRICT mode tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-11-29T21:05:10Z", "updated_at": "2021-11-29T22:45:26Z", "closed_at": "2021-11-29T22:44:36Z", "author_association": "OWNER", "pull_request": null, "body": "> From the STRICT docs:\r\n\r\n>> The SQLite parser accepts a comma-separated list of table options after the final close parenthesis in a CREATE TABLE statement. As of this writing (2021-08-23) only two options are recognized:\r\n>> \r\n>> - STRICT\r\n>> - [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html)\r\n> \r\n> So I think I need to read the `CREATE TABLE` statement from the `sqlite_master` table, split on the last `)`, split those tokens on `,` and see if `create` is in there (case insensitive).\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982020757_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/345/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1067771698, "node_id": "I_kwDOCGYnMM4_pOcy", "number": 348, "title": "Command for creating an empty database", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 6, "created_at": "2021-11-30T23:24:27Z", "updated_at": "2022-01-13T07:06:59Z", "closed_at": "2022-01-09T20:33:20Z", "author_association": "OWNER", "pull_request": null, "body": "I sometimes find the need to create an empty SQLite database file - for example if I want to enable WAL on it before using it with another script. I currently do that like this:\r\n\r\n sqlite3 my.db vacuum\r\n sqlite-utils enable-wal my.db\r\n\r\nIt would be nice if `sqlite-utils` had a convenience command for doing this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/348/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1976986318, "node_id": "I_kwDOCGYnMM511mrO", "number": 599, "title": "Cannot find spatialite on arm64 linux", "user": {"value": 37802088, "label": "MikeCoats"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-11-03T22:05:51Z", "updated_at": "2023-11-04T01:06:31Z", "closed_at": "2023-11-04T00:33:28Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Initially, I found an issue in `datasette` where it wouldn\u2019t find `spatialite` when running on my Radxa Rock 5B - an RK3588 powered SBC, running the arm64 build of Debian Bullseye. I confirmed the same behaviour on my Raspberry Pi 4 - a BCM2711 powered SBC, running the arm64 build of Debian Bookworm.\r\n\r\n```\r\n$ datasette --load-extension=spatialite example.db\r\nError: Could not find SpatiaLite extension\r\n```\r\n\r\nI did some digging and realised the issue originates in this project. Even with the `libsqlite3-mod-spatialite` package installed, `pytest` skips all of the GIS tests in the project.\r\n\r\n```\r\n$ apt list --installed | grep spatial\r\n[\u2026]\r\nlibsqlite3-mod-spatialite/stable,now 5.0.1-3 arm64 [installed]\r\n\r\n$ ls -l /usr/lib/*/*spatial*\r\nlrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0\r\nlrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0\r\n-rw-r--r-- 1 root root 7348584 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0\r\n```\r\n\r\n```\r\n$ pytest\r\ntests/test_get.py ...... [ 73%]\r\ntests/test_gis.py ssssssssssss [ 75%]\r\ntests/test_hypothesis.py .... [ 75%]\r\n```\r\n\r\nI tracked the issue down to the [`find_sqlite()` function in the `utils.py`](https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/utils.py#L60) file. The [`SPATIALITE_PATHS`](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L34-L39) array doesn\u2019t have an entry for the location of this module on arm64 linux.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/599/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 2007893839, "node_id": "I_kwDOCGYnMM53rgdP", "number": 605, "title": "Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here?", "user": {"value": 12229877, "label": "Zac-HD"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-11-23T10:19:46Z", "updated_at": "2023-12-08T05:07:54Z", "closed_at": "2023-12-08T05:07:54Z", "author_association": "NONE", "pull_request": null, "body": "I'm currently working on a new feature for Hypothesis, where we can dump a tidy jsonlines table of all the test cases we tried - including arguments, outcomes, timings, coverage, etc. Exploring this seems like a perfect cases for `sqlite-utils` and `datasette`, but I pretty quickly ran into an integer overflow problem and don't want to recommend that experience to my users.\r\n\r\nI originally went to report this as a bug... and then found https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038 almost exactly matched my repro \ud83d\ude05 \r\n\r\nhttps://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063 suggests that using `NUMERIC` would avoid this overflow error, although \"If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL.\" suggests that this would come at the cost of rounding to the nearest float value. Maybe I should just convert large integers to float before writing out my json?\r\n\r\nAfter a bit more hacking, \"manually cast large integers to float\" seems like a decent solution for my particular case, but having written it up I thought I might as well post this issue anyway - I hope it's useful feedback, and won't mind at all if you close as wontfix if it's not.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/605/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 2029161033, "node_id": "I_kwDOCGYnMM548opJ", "number": 606, "title": "str and int as aliases for text and integer", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-12-06T18:35:49Z", "updated_at": "2023-12-06T19:44:04Z", "closed_at": "2023-12-06T18:49:32Z", "author_association": "OWNER", "pull_request": null, "body": "I keep making this mistake:\r\n```bash\r\nsqlite-utils add-column content.db assets _since int\r\n```\r\n```\r\nUsage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME [[integer|float|b\r\n lob|text|INTEGER|FLOAT|BLOB|TEXT]]\r\nTry 'sqlite-utils add-column -h' for help.\r\n\r\nError: Invalid value for '[[integer|float|blob|text|INTEGER|FLOAT|BLOB|TEXT]]':\r\n 'int' is not one of 'integer', 'float', 'blob', 'text', 'INTEGER', 'FLOAT', 'BLOB', 'TEXT'.\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/606/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1077102934, "node_id": "I_kwDOCGYnMM5AM0lW", "number": 353, "title": "Allow passing a file of code to \"sqlite-utils convert\"", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-12-10T18:06:14Z", "updated_at": "2021-12-11T01:38:29Z", "closed_at": "2021-12-11T01:09:39Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "sqlite-utils is so nice, but the ergonomics of the multiline code in kind of tough. It's really hard (maybe impossible) to make the newlines play well with Makefiles.\r\n\r\nit would be great to write your code fragment in a separate file and direct it into the sqlite-utils\r\n\r\neither like\r\n\r\n```sqlite-utils convert my.db my_table my_column < custom_code.py```\r\n\r\nor\r\n\r\n```sqlite-utils convert my.db my_table my_column --custom-code=custom_code.py```\r\n\r\nThanks, as ever, for these great tools!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/353/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1077243232, "node_id": "I_kwDOCGYnMM5ANW1g", "number": 354, "title": "Test failure in test_rebuild_fts", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2021-12-10T21:27:55Z", "updated_at": "2021-12-11T01:08:46Z", "closed_at": "2021-12-11T01:08:46Z", "author_association": "OWNER", "pull_request": null, "body": "Not sure why this has only just started failing, but I'm getting this: https://github.com/simonw/sqlite-utils/runs/4488687639\r\n\r\n```\r\nE sqlite3.DatabaseError: database disk image is malformed\r\n\r\nsqlite_utils/db.py:425: DatabaseError\r\n_______________________ test_rebuild_fts[searchable_fts] _______________________\r\n\r\nfresh_db = >\r\ntable_to_fix = 'searchable_fts'\r\n\r\n @pytest.mark.parametrize(\"table_to_fix\", [\"searchable\", \"searchable_fts\"])\r\n def test_rebuild_fts(fresh_db, table_to_fix):\r\n table = fresh_db[\"searchable\"]\r\n table.insert(search_records[0])\r\n table.enable_fts([\"text\", \"country\"])\r\n # Run a search\r\n rows = list(table.search(\"tanuki\"))\r\n assert len(rows) == 1\r\n assert {\r\n \"rowid\": 1,\r\n \"text\": \"tanuki are running tricksters\",\r\n \"country\": \"Japan\",\r\n \"not_searchable\": \"foo\",\r\n }.items() <= rows[0].items()\r\n # Delete from searchable_fts_data\r\n fresh_db[\"searchable_fts_data\"].delete_where()\r\n # This should have broken the index\r\n with pytest.raises(sqlite3.DatabaseError):\r\n list(table.search(\"tanuki\"))\r\n # Running rebuild_fts() should fix it\r\n> fresh_db[table_to_fix].rebuild_fts()\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/354/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1077322009, "node_id": "I_kwDOCGYnMM5ANqEZ", "number": 355, "title": "Allow users to pass a full convert() function definition", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-10T23:59:58Z", "updated_at": "2021-12-11T00:51:15Z", "closed_at": "2021-12-11T00:49:31Z", "author_association": "OWNER", "pull_request": null, "body": "> I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too:\r\n> ```sh\r\n> sqlite-utils convert my.db mytable col1 '\r\n> def convert(value):\r\n> return value.upper()\r\n> '\r\n> ```\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/355/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1077431957, "node_id": "I_kwDOCGYnMM5AOE6V", "number": 356, "title": "`sqlite-utils insert --convert` option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2021-12-11T07:24:48Z", "updated_at": "2022-01-06T06:30:13Z", "closed_at": "2022-01-06T06:28:53Z", "author_association": "OWNER", "pull_request": null, "body": "Idea come to me while re-reading this: https://simonwillison.net/2021/Aug/6/sqlite-utils-convert/\r\n\r\nThis is a bit of a hack:\r\n```\r\ncat /tmp/log.txt | \\\r\n jq --raw-input '{line: .}' --compact-output | \\\r\n sqlite-utils insert /tmp/logs.db log - --nl\r\n```\r\nWould be great if you could pipe lines to `insert` and transform them on the way in.\r\n\r\nA `--convert python-code` option, modeled after `sqlite-utils convert`, could do this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/356/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1079422215, "node_id": "I_kwDOCGYnMM5AVq0H", "number": 357, "title": "pytest-runner is not required", "user": {"value": 4067843, "label": "pgajdos"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-14T07:51:24Z", "updated_at": "2021-12-16T20:43:19Z", "closed_at": "2021-12-16T20:43:13Z", "author_association": "NONE", "pull_request": null, "body": "Deprecated pytest-runner is not necessary for running the testsuite.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/357/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1091819089, "node_id": "I_kwDOCGYnMM5BE9ZR", "number": 360, "title": "MemoryError", "user": {"value": 559453, "label": "nzaar9"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-01T13:39:17Z", "updated_at": "2022-03-21T04:22:46Z", "closed_at": "2022-03-21T04:22:46Z", "author_association": "NONE", "pull_request": null, "body": "HI, when dealing with large json file (~170GB) i got the following error \r\n```\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 1126, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 1051, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 1393, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 752, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/cli.py\", line 1300, in memory\r\n rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding)\r\n File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py\", line 185, in rows_from_file\r\n return rows_from_file(buffered, format=Format.JSON)\r\n File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py\", line 156, in rows_from_file\r\n decoded = json.load(fp)\r\n File \"/usr/lib/python3.9/json/__init__.py\", line 293, in load\r\n return loads(fp.read(),\r\nMemoryError\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/360/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1094974713, "node_id": "I_kwDOCGYnMM5BQ_z5", "number": 362, "title": "upsert --detect-types is broken", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-06T05:12:10Z", "updated_at": "2022-01-06T06:54:45Z", "closed_at": "2022-01-06T06:28:34Z", "author_association": "OWNER", "pull_request": null, "body": "Noticed this thanks to syntax highlighting in VS Code showing an unused variable - need to fix it and add a test.\r\n\r\n\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/362/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1094981339, "node_id": "I_kwDOCGYnMM5BRBbb", "number": 363, "title": "Better error message if `--convert` code fails to return a dict", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-06T05:26:28Z", "updated_at": "2022-02-03T22:52:30Z", "closed_at": "2022-02-03T22:51:30Z", "author_association": "OWNER", "pull_request": null, "body": "Here's the traceback if your `--convert` function doesn't return a dict right now:\r\n```\r\n% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all \r\n\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 949, in insert\r\n insert_upsert_implementation(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 834, in insert_upsert_implementation\r\n db[table].insert_all(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2602, in insert_all\r\n first_record = next(records)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 3044, in fix_square_braces\r\n for record in records:\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 831, in \r\n docs = (decode_base64_values(doc) for doc in docs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 86, in decode_base64_values\r\n to_fix = [\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 89, in \r\n if isinstance(doc[k], dict)\r\nTypeError: string indices must be integers\r\n```\r\nIt would be nicer if that returned a more useful error message.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/361#issuecomment-1006295276_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/363/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1095570074, "node_id": "I_kwDOCGYnMM5BTRKa", "number": 364, "title": "`--batch-size 1` doesn't seem to commit for every item", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 16, "created_at": "2022-01-06T18:18:50Z", "updated_at": "2022-01-10T19:27:17Z", "closed_at": "2022-01-10T05:36:19Z", "author_association": "OWNER", "pull_request": null, "body": "I'm trying this, but it doesn't seem to write anything to the database file until I hit `CTRL+C`:\r\n\r\n```\r\nheroku logs --app=simonwillisonblog --tail | grep 'measure#nginx.service' | \\\r\n sqlite-utils insert /tmp/herokutail.db log - --import re --convert \"$(cat < Relevant documentation: https://www.sqlite.org/lang_analyze.html\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/366/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097087280, "node_id": "I_kwDOCGYnMM5BZDkw", "number": 368, "title": "Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T02:29:30Z", "updated_at": "2022-01-10T19:27:20Z", "closed_at": "2022-01-09T02:40:50Z", "author_association": "OWNER", "pull_request": null, "body": "> Add this to `sqlite_utils/cli.py`:\r\n>\r\n> ```python\r\n> if __name__ == \"__main__\":\r\n> cli()\r\n> ```\r\n> Now the tool can be run using `python -m sqlite_utils.cli --help`\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/368/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097091527, "node_id": "I_kwDOCGYnMM5BZEnH", "number": 369, "title": "Research how much of a difference analyze / sqlite_stat1 makes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2022-01-09T03:03:36Z", "updated_at": "2022-02-03T21:07:41Z", "closed_at": "2022-02-03T21:07:35Z", "author_association": "OWNER", "pull_request": null, "body": "> Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it?\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050_\r\n\r\nMore generally: how much of a difference does the `sqlite_stat1` table created by `ANALYZE` make to queries?\r\n\r\nI'm particularly interested in `group by` / `count *` queries since Datasette uses those for faceting.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/369/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097129710, "node_id": "I_kwDOCGYnMM5BZN7u", "number": 372, "title": "Idea: `suffix` and `stem` file columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 1, "created_at": "2022-01-09T07:48:53Z", "updated_at": "2022-01-10T19:27:34Z", "closed_at": "2022-01-09T20:17:00Z", "author_association": "OWNER", "pull_request": null, "body": "For https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files\r\n\r\nGiven a file called `dogs.jpg` stem would be `dogs` and ext would be `jpg`.\r\n\r\nNeed to decide what happens for `dogs.and.cats.jpg.gz`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/372/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097128334, "node_id": "I_kwDOCGYnMM5BZNmO", "number": 371, "title": "Support mutating row in `--convert` without returning it", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 6, "created_at": "2022-01-09T07:38:44Z", "updated_at": "2022-01-10T19:27:30Z", "closed_at": "2022-01-09T20:06:15Z", "author_association": "OWNER", "pull_request": null, "body": "Currently you have to do this:\r\n```\r\n$ sqlite-utils insert dogs.db dogs dogs.json --convert '\r\nrow[\"is_good\"] = 1\r\nreturn row'\r\n```\r\nWould be neat if this worked too:\r\n```\r\n$ sqlite-utils insert dogs.db dogs dogs.json \\\r\n --convert 'row[\"is_good\"] = 1'\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/371/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097135732, "node_id": "I_kwDOCGYnMM5BZPZ0", "number": 373, "title": "List `--fmt` options in the docs ", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T08:22:11Z", "updated_at": "2022-01-10T19:27:24Z", "closed_at": "2022-01-09T17:49:00Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/stable/cli.html#table-formatted-output currently cheats and tells the user to run `--help` - can fix this using `cog`. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/373/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097135860, "node_id": "I_kwDOCGYnMM5BZPb0", "number": 374, "title": "`--fmt` should imply `-t`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 4, "created_at": "2022-01-09T08:23:07Z", "updated_at": "2022-01-10T19:27:26Z", "closed_at": "2022-01-09T18:07:59Z", "author_association": "OWNER", "pull_request": null, "body": "Not sure why I didn't implement this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/374/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097251014, "node_id": "I_kwDOCGYnMM5BZrjG", "number": 375, "title": "`sqlite-utils bulk` command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T17:12:38Z", "updated_at": "2022-01-11T02:12:58Z", "closed_at": "2022-01-11T02:10:55Z", "author_association": "OWNER", "pull_request": null, "body": "The `.executemany()` method is a very efficient way to execute the same SQL query against a huge list of parameters.\r\n\r\n`sqlite-utils insert` supports a bunch of ways of loading a list of dictionaries - from CSV, TSV, JSON, newline JSON and more thanks to:\r\n- #361\r\n\r\nWhat if you could load a list of dictionaries and provide a SQL query with `:named` parameters that correspond to keys in those dictionaries instead?\r\n\r\nThis would need to be a new command - I thought about adding a `--sql` option to `insert` but that doesn't make sense as that command already requires a table name.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/375/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097436959, "node_id": "I_kwDOCGYnMM5BaY8f", "number": 376, "title": "`--nl` mode should ignore blank lines", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-10T04:10:54Z", "updated_at": "2022-01-10T19:27:41Z", "closed_at": "2022-01-10T04:12:46Z", "author_association": "OWNER", "pull_request": null, "body": "Spotted this while manually testing #364 - there's no reason `--nl` should crash if you feed it an empty line in between JSON objects.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/376/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1098309897, "node_id": "I_kwDOCGYnMM5BduEJ", "number": 378, "title": "analyze=True parameter for some methods", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-10T19:54:52Z", "updated_at": "2022-01-11T01:08:11Z", "closed_at": "2022-01-11T01:08:09Z", "author_association": "OWNER", "pull_request": null, "body": "This would cause `ANALYZE` to be run against the relevant table at the end of executing the method.\r\n\r\n> Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are:\r\n\r\n- [x] `table.create_index`\r\n- [x] `table.insert_all`\r\n- [x] `table.upsert_all`\r\n- [x] `table.delete_where`\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/378/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1098544628, "node_id": "I_kwDOCGYnMM5BenX0", "number": 379, "title": "CLI options for running ANALYZE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-11T01:09:16Z", "updated_at": "2022-01-11T01:38:01Z", "closed_at": "2022-01-11T01:36:48Z", "author_association": "OWNER", "pull_request": null, "body": "> The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865_\r\n\r\n- [x] `sqlite-utils analyze` command\r\n- [x] `sqlite-utils create-index --analyze` option (see #365)\r\n- [x] `sqlite-utils insert --analyze` option\r\n- [x] `sqlite-utils upsert --analyze` option\r\n\r\nIn #378 I also added `.delete_where(..., analyze=True)` but there isn't currently a `sqlite-utils delete-where` CLI command - deletions via CLI are expected to be handled using SQL queries.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/379/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1098574572, "node_id": "I_kwDOCGYnMM5Beurs", "number": 380, "title": "Release notes for 3.21", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 1, "created_at": "2022-01-11T02:12:30Z", "updated_at": "2022-01-11T02:34:26Z", "closed_at": "2022-01-11T02:34:26Z", "author_association": "OWNER", "pull_request": null, "body": "For these commits: https://github.com/simonw/sqlite-utils/compare/3.20...129141572f249ea290e2a075437e2ebaad215859", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/380/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099584685, "node_id": "I_kwDOCGYnMM5BilSt", "number": 381, "title": "`sqlite-utils rows` options `--limit` and `--offset`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-01-11T20:23:12Z", "updated_at": "2022-01-11T23:33:37Z", "closed_at": "2022-01-11T23:19:36Z", "author_association": "OWNER", "pull_request": null, "body": "Because I often want to use it just to preview a few rows from the database. Piping through `| head -n 20` works for JSON and CSV (they stream) but not for `--table`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/381/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099585611, "node_id": "I_kwDOCGYnMM5BilhL", "number": 382, "title": "`--where` option for `sqlite-rows`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-11T20:24:23Z", "updated_at": "2022-01-11T23:33:14Z", "closed_at": "2022-01-11T23:32:47Z", "author_association": "OWNER", "pull_request": null, "body": "CLI equivalent of `table.rows_where()` - should accept parameters too. Work on this at the same time as #381.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/382/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099586786, "node_id": "I_kwDOCGYnMM5Bilzi", "number": 383, "title": "Add documentation page with the output of `--help`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-11T20:25:58Z", "updated_at": "2022-01-11T22:55:05Z", "closed_at": "2022-01-11T21:44:05Z", "author_association": "OWNER", "pull_request": null, "body": "Can be maintained using `cog` from #373. Similar in purpose to the API reference page, but this is for the CLI.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/383/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099897648, "node_id": "I_kwDOCGYnMM5Bjxsw", "number": 384, "title": "Add examples to every `--help`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-12T05:31:25Z", "updated_at": "2022-01-26T03:15:02Z", "closed_at": "2022-01-26T03:15:02Z", "author_association": "OWNER", "pull_request": null, "body": "Everything on https://sqlite-utils.datasette.io/en/stable/cli-reference.html would benefit from an example.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/384/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1107557831, "node_id": "I_kwDOCGYnMM5CA_3H", "number": 386, "title": "Better \"contributing\" documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-19T02:11:48Z", "updated_at": "2022-01-19T02:15:21Z", "closed_at": "2022-01-19T02:15:21Z", "author_association": "OWNER", "pull_request": null, "body": "This page jumps straight into running the tests: https://sqlite-utils.datasette.io/en/latest/contributing.html\r\n\r\nIt should add a little more about expected collaboration styles - opening an issue before filing a pull request - and probably link to https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/386/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1111293050, "node_id": "I_kwDOCGYnMM5CPPx6", "number": 387, "title": "Python library docs should start with a self contained example", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-22T06:23:56Z", "updated_at": "2022-01-26T01:37:17Z", "closed_at": "2022-01-26T01:35:30Z", "author_association": "OWNER", "pull_request": null, "body": "You have to read a lot of stuff in a lot of different places to get started with the Python library. Add a getting started introduction to https://sqlite-utils.datasette.io/en/stable/python-api.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/387/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1123851690, "node_id": "I_kwDOCGYnMM5C_J2q", "number": 396, "title": "mypy failure, sqlite_utils/utils.py:56", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-02-04T06:08:09Z", "updated_at": "2022-02-04T06:10:33Z", "closed_at": "2022-02-04T06:10:33Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/runs/5062725880?check_suite_focus=true\r\n\r\n> `sqlite_utils/utils.py:56: error: Incompatible return value type (got \"None\", expected \"str\")`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/396/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1123849278, "node_id": "I_kwDOCGYnMM5C_JQ-", "number": 395, "title": "\"apt-get: command not found\" error on macOS", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-04T06:03:42Z", "updated_at": "2022-02-04T06:10:58Z", "closed_at": "2022-02-04T06:10:58Z", "author_association": "OWNER", "pull_request": null, "body": "Yeah, `apt-get` isn't a thing on macOS so 4a2a3e2fd0d5534f446b3f1fee34cb165e4d86d2 (to test #79 against real SpatiaLite) broke.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/395/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1123903919, "node_id": "I_kwDOCGYnMM5C_Wmv", "number": 397, "title": "Support IF NOT EXISTS for table creation", "user": {"value": 738408, "label": "rafguns"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-02-04T07:41:15Z", "updated_at": "2022-02-06T01:30:46Z", "closed_at": "2022-02-06T01:29:01Z", "author_association": "NONE", "pull_request": null, "body": "Currently, I have a bunch of code that looks like this:\r\n\r\n```python\r\nsubjects = db[\"subjects\"] if db[\"subjects\"].exists() else db[\"subjects\"].create({\r\n ...\r\n})\r\n```\r\nIt would be neat if sqlite-utils could simplify that by supporting `CREATE TABLE IF NOT EXISTS`, so that I'd be able to write, e.g.\r\n\r\n```python\r\nsubjects = db[\"subjects\"].create({...}, if_not_exists=True)\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/397/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1114543475, "node_id": "I_kwDOCGYnMM5CbpVz", "number": 388, "title": "Link to stable docs from older versions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-01-26T01:55:46Z", "updated_at": "2023-03-26T23:43:12Z", "closed_at": "2022-01-26T02:00:22Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/2.14.1/ isn't showing a link to the stable release right now.\r\n\r\nI should also apply the same fix I used for Datasette in:\r\n- https://github.com/simonw/datasette/issues/1608\r\n\r\nTIL: https://til.simonwillison.net/readthedocs/link-from-latest-to-stable", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/388/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1114544727, "node_id": "I_kwDOCGYnMM5CbppX", "number": 389, "title": "Plausible analytics for documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-01-26T01:58:35Z", "updated_at": "2022-01-26T02:07:41Z", "closed_at": "2022-01-26T02:07:41Z", "author_association": "OWNER", "pull_request": null, "body": "```html\r\n\r\n```\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/389/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1114557284, "node_id": "I_kwDOCGYnMM5Cbstk", "number": 390, "title": "`sqlite-utils upsert` should require `--pk` more elegantly", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-26T02:20:31Z", "updated_at": "2022-01-26T03:20:25Z", "closed_at": "2022-01-26T03:19:43Z", "author_association": "OWNER", "pull_request": null, "body": "Currently throws an ugly traceback:\r\n```\r\n% echo '[\r\n {\"id\": 1, \"name\": \"Lila\"},\r\n {\"id\": 1, \"name\": \"Lila\"}\r\n ]' | sqlite-utils upsert data.db chickens - \r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 1104, in upsert\r\n insert_upsert_implementation(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 906, in insert_upsert_implementation\r\n db[table].insert_all(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2615, in insert_all\r\n raise PrimaryKeyRequired(\"upsert() requires a pk\")\r\nsqlite_utils.db.PrimaryKeyRequired: upsert() requires a pk\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/390/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1114640101, "node_id": "I_kwDOCGYnMM5CcA7l", "number": 392, "title": "`sqlite-utils bulk --batch-size` option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-26T05:17:11Z", "updated_at": "2022-01-26T18:17:59Z", "closed_at": "2022-01-26T18:17:59Z", "author_association": "OWNER", "pull_request": null, "body": "> Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/392/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1114638930, "node_id": "I_kwDOCGYnMM5CcApS", "number": 391, "title": "`sqlite-utils bulk` progress bar", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-01-26T05:14:49Z", "updated_at": "2022-01-26T05:17:20Z", "closed_at": "2022-01-26T05:16:51Z", "author_association": "OWNER", "pull_request": null, "body": "It can easily have a progress bar because it works by looping through an iterator: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018\r\n\r\nShould also support the `--silent` option if I add this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/391/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1118585417, "node_id": "I_kwDOCGYnMM5CrEJJ", "number": 393, "title": "Better documentation for insert-replace", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-30T15:40:23Z", "updated_at": "2022-02-03T22:13:24Z", "closed_at": "2022-02-03T22:13:24Z", "author_association": "OWNER", "pull_request": null, "body": "Currently: https://sqlite-utils.datasette.io/en/stable/python-api.html#insert-replacing-data\r\n\r\n> If you want to insert a record or replace an existing record with the same primary key, using the replace=True argument to .insert() or .insert_all():\r\n\r\nShould describe the exception you get first, then how to use replace to avoid it.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/393/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1124237013, "node_id": "I_kwDOCGYnMM5DAn7V", "number": 398, "title": "Add SpatiaLite helpers to CLI", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-02-04T14:01:28Z", "updated_at": "2022-02-16T01:02:29Z", "closed_at": "2022-02-16T00:58:07Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Now that #385 is merged, add CLI versions of those methods.\r\n\r\n```sh\r\n# init spatialite\r\nsqlite-utils init-spatialite database.db\r\n\r\n# or maybe/also\r\nsqlite-utils create database.db --enable-wal --spatialite\r\n\r\n# add geometry columns\r\n# needs a database, table, geometry column name, type, with optional SRID and not-null\r\n# this needs to create a table if it doesn't already exist\r\nsqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null\r\n\r\n# spatial index an existing table/column\r\nsqlite-utils create-spatial-index database.db table-name geometry\r\n```\r\n\r\nShould be mostly straightforward. The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{\"rowid\": int, \"geometry\": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/398/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1125077063, "node_id": "I_kwDOCGYnMM5DD1BH", "number": 400, "title": "`sqlite-utils create-table` ... `--if-not-exists`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-06T01:32:53Z", "updated_at": "2022-02-06T01:34:53Z", "closed_at": "2022-02-06T01:34:46Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by:\r\n- #397\r\n\r\nTo match the option on `create-index`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-index\r\n\r\n```\r\n --if-not-exists Ignore if index already exists\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/400/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1125081640, "node_id": "I_kwDOCGYnMM5DD2Io", "number": 401, "title": "Update SpatiaLite example in the documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-02-06T02:02:07Z", "updated_at": "2022-02-06T02:05:03Z", "closed_at": "2022-02-06T02:03:24Z", "author_association": "OWNER", "pull_request": null, "body": "This one here: https://sqlite-utils.datasette.io/en/3.23/python-api.html#converting-column-values-using-sql-functions\r\n\r\nIt should take advantage of the new methods from:\r\n- #79", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/401/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1126692066, "node_id": "I_kwDOCGYnMM5DJ_Ti", "number": 403, "title": "Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-02-08T01:39:40Z", "updated_at": "2022-02-09T04:22:43Z", "closed_at": "2022-02-08T19:33:59Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "*Original title: Add option for adding a new, serial, primary key*\r\n\r\nsometimes we have tables that don't have primary keys, but ought to have them. we *can* use rowid for that, but it would often be nicer to have an explicit primary key. using the current value of rowid would be fine.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/403/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1128120451, "node_id": "I_kwDOCGYnMM5DPcCD", "number": 404, "title": "Add example of `--convert` to the help for `sqlite-utils insert`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-02-09T06:49:09Z", "updated_at": "2022-02-09T06:56:35Z", "closed_at": "2022-02-09T06:55:16Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/3.23/cli-reference.html#insert would be more useful if it included an example of `--convert` in action.\r\n\r\nI can maybe use an example from https://simonwillison.net/2022/Jan/11/sqlite-utils/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/404/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1128139375, "node_id": "I_kwDOCGYnMM5DPgpv", "number": 405, "title": "`Database(memory_name=\"name\")` constructor argument", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-02-09T07:15:03Z", "updated_at": "2022-02-16T01:23:16Z", "closed_at": "2022-02-16T01:23:16Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite in-memory databases can be named, in which case multiple connections can be opened to a shared in-memory database running within the same process.\r\n\r\nDatasette supports this - SQLite could support it too.\r\n\r\nhttps://docs.datasette.io/en/0.60.2/internals.html#database-ds-path-none-is-mutable-false-is-memory-false-memory-name-none", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/405/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1145882578, "node_id": "I_kwDOCGYnMM5ETMfS", "number": 408, "title": "`deterministic=True` fails on versions of SQLite prior to 3.8.3", "user": {"value": 24938923, "label": "learning4life"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2022-02-21T14:36:43Z", "updated_at": "2022-03-13T16:54:09Z", "closed_at": "2022-03-02T00:38:11Z", "author_association": "NONE", "pull_request": null, "body": "Hi, love your work.\r\n\r\nI am unable to lookup indexes in a database using sqlite-utils:\r\n\r\n`\r\nsqlite-utils indexes city_spec.db --table`\r\n\r\nor\r\n\r\n`sqlite-utils indexes city_spec.db MyTable\r\n`\r\n\r\n**Software**\r\nsqlite-utils, version 3.24\r\nsqlite3 --version: 3.36.0 \r\n\r\n**Output:**\r\n\r\nTraceback (most recent call last):\r\n File \"/opt/app-root/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/decorators.py\", line 26, in new_func\r\n return f(get_current_context(), *args, **kwargs)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py\", line 2123, in indexes\r\n ctx.invoke(\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py\", line 1624, in query\r\n db.register_fts4_bm25()\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 403, in register_fts4_bm25\r\n self.register_function(rank_bm25, deterministic=True)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 399, in register_function\r\n register(fn)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 392, in register\r\n self.conn.create_function(name, arity, fn, **kwargs)\r\nsqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/408/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1171599874, "node_id": "I_kwDOCGYnMM5F1TIC", "number": 415, "title": "Convert with `--multi` and `--dry-run` flag does not work", "user": {"value": 3976183, "label": "dotcs"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-03-16T21:59:46Z", "updated_at": "2022-03-21T04:18:24Z", "closed_at": "2022-03-21T04:18:24Z", "author_association": "NONE", "pull_request": null, "body": "It's not possible to combine `--multi` and `--dry-run` flag in the `convert` command.\r\n\r\nLet's first create a simple database from JSON string\r\n\r\n```console\r\n$ echo '[{\"foo\": \"abc\"}]' | sqlite-utils insert demo.db demo -\r\n$ sqlite-utils query demo.db \"SELECT * FROM demo\" \r\n[{\"foo\": \"abc\"}]\r\n```\r\n\r\nand then try to convert the \"foo\" column with a static value \"bar\" (see docs [Converting a column into multiple columns](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-a-column-into-multiple-columns))\r\n\r\n```console\r\n$ sqlite-utils convert demo.db demo foo '{\"foo\": \"bar\"}' --multi --dry-run\r\nTraceback (most recent call last):\r\n File \"/home/dotcs/anaconda3/envs/tools/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/sqlite_utils/cli.py\", line 2686, in convert\r\n for row in db.conn.execute(sql, where_args).fetchall():\r\nsqlite3.OperationalError: user-defined function raised exception\r\n```\r\n\r\nBut without the `--dry-run` flag it does work as expected:\r\n\r\n```console\r\n$ sqlite-utils convert demo.db demo foo '{\"foo\": \"bar\"}' --multi\r\n$ sqlite-utils query demo.db \"SELECT * FROM demo\" \r\n[{\"foo\": \"bar\"}]\r\n```\r\n\r\n```console\r\n$ sqlite-utils --version\r\nsqlite-utils, version 3.25.1\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/415/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1173023272, "node_id": "I_kwDOCGYnMM5F6uoo", "number": 416, "title": "Options for how `r.parsedate()` should handle invalid dates", "user": {"value": 638427, "label": "mattkiefer"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2022-03-17T23:29:55Z", "updated_at": "2022-05-03T21:36:49Z", "closed_at": "2022-03-21T04:01:39Z", "author_association": "NONE", "pull_request": null, "body": "Exceptions are normal expected behavior when typecasting an invalid format. However, r.parsedate() is really just re-formatting strings and keeping the type as text. So it may be better to print-and-pass on exception so the user can see a complete list of invalid values -- while also allowing for the parser to reformat the remaining valid values. \r\n```\r\nsqlite-utils convert idfpr.db license \"Expiration Date\" \"r.parsedate(value)\"\r\n [#######-----------------------------] 21% 00:01:57Traceback (most recent call last):\r\n File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/db.py\", line 2336, in convert_value\r\n return fn(v)\r\n File \"\", line 2, in fn\r\n File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/recipes.py\", line 8, in parsedate\r\n parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()\r\n File \"/usr/lib/python3/dist-packages/dateutil/parser/_parser.py\", line 1374, in parse\r\n return DEFAULTPARSER.parse(timestr, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/dateutil/parser/_parser.py\", line 652, in parse\r\n raise ParserError(\"String does not contain a date: %s\", timestr)\r\ndateutil.parser._parser.ParserError: String does not contain a date: / / \r\n```\r\nIn this case, I had just one variation of an invalid date: ' / / '. But theoretically there could be many values that would have to be fixed one at a time with the current exception handling. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/416/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1166587040, "node_id": "I_kwDOCGYnMM5FiLSg", "number": 413, "title": "Display autodoc type information more legibly", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-03-11T15:58:20Z", "updated_at": "2022-03-11T18:07:10Z", "closed_at": "2022-03-11T18:07:10Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/3.25/reference.html#sqlite_utils.db.Table.insert looks like this at the moment:\r\n\r\n\"image\"\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/413/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1166731361, "node_id": "I_kwDOCGYnMM5Fiuhh", "number": 414, "title": "I forgot to include the changelog in the 3.25.1 release", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-03-11T18:32:36Z", "updated_at": "2022-03-11T18:40:39Z", "closed_at": "2022-03-11T18:40:39Z", "author_association": "OWNER", "pull_request": null, "body": "I pushed a release for https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 but forgot to include the release notes in `docs/changelog.rst`\r\n\r\nThis means https://sqlite-utils.datasette.io/en/stable/changelog.html isn't showing them.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/414/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1175744654, "node_id": "I_kwDOCGYnMM5GFHCO", "number": 417, "title": "insert fails on JSONL with whitespace", "user": {"value": 9954, "label": "blaine"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-03-21T17:58:14Z", "updated_at": "2022-03-25T21:19:06Z", "closed_at": "2022-03-25T21:17:13Z", "author_association": "NONE", "pull_request": null, "body": "Any JSON that is newline-delimited and has whitespace (newlines) between the start of a JSON object and an attribute fails due to a parse error.\r\n\r\ne.g. given the valid JSONL:\r\n\r\n```{\r\n \"attribute\": \"value\"\r\n}\r\n{\r\n \"attribute\": \"value2\"\r\n}\r\n```\r\n\r\nI would expect that `sqlite-utils insert --nl my.db mytable file.jsonl` would properly import the data into `mytable`. However, the following error is thrown instead:\r\n\r\n`json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)`\r\n\r\nIt makes sense that since the file is intended to be newline separated, the thing being parsed is \"{\" (which obviously fails), however the default newline-separated output of `jq` isn't compact. Using `jq -c` avoids this problem, but the fix is unintuitive and undocumented.\r\n\r\nProposed solutions:\r\n1. Default to a \"loose\" newline-separated parse; this could be implemented internally as [the equivalent of] a `jq -c` filter ahead of the insert step.\r\n2. Catch the JSONDecodeError (or pre-empt it in the case of a record === \"{\\n\") and give the user a \"it looks like your json isn't _actually_ newline-delimited; try running it through `jq -c` instead\" error message.\r\n\r\nIt might just have been too early in the morning when I was playing with this, but running pipes of data through sqlite-utils without the 'knack' of it led to some false starts.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/417/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1178456794, "node_id": "I_kwDOCGYnMM5GPdLa", "number": 418, "title": "Add generated files to .gitignore", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-03-23T17:48:12Z", "updated_at": "2022-03-24T21:01:44Z", "closed_at": "2022-03-24T21:01:44Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I end up with these in my local directory:\r\n\r\n\t.hypothesis/\r\n\tPipfile\r\n\tPipfile.lock\r\n\tpyproject.toml\r\n\r\nMight as well gitignore them.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/418/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1178546862, "node_id": "I_kwDOCGYnMM5GPzKu", "number": 420, "title": "Document how to use a `--convert` function that runs initialization code first", "user": {"value": 770231, "label": "strada"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2022-03-23T19:07:36Z", "updated_at": "2022-08-28T11:34:37Z", "closed_at": "2022-03-25T20:07:33Z", "author_association": "NONE", "pull_request": null, "body": "When I have an insert command with transform like this:\r\n\r\n```\r\ncat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert '\r\nd = enchant.Dict(\"en_US\")\r\nrow[\"is_dictionary_word\"] = d.check(row[\"name\"])\r\n' --import=enchant --ignore\r\n```\r\n\r\nI noticed as the number of rows increases the operation becomes quite slow, likely due to the creation of the `d = enchant.Dict(\"en_US\")` object for each row. Is there a way to share that instance `d` between transform function calls, like a shared context?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/420/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1180427792, "node_id": "I_kwDOCGYnMM5GW-YQ", "number": 421, "title": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI", "user": {"value": 24938923, "label": "learning4life"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2022-03-25T07:12:51Z", "updated_at": "2022-04-13T22:41:59Z", "closed_at": "2022-04-13T22:41:59Z", "author_association": "NONE", "pull_request": null, "body": "This bug relates to https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1066139147\r\n\r\n**New error when using CLI: \"sqlite-utils indexes global.db --table\"**\r\n\r\n```\r\n(app-root) sqlite-utils indexes global.db --table\r\nError: near \"(\": syntax error\r\n(app-root) sqlite-utils --version\r\nsqlite-utils, version 3.25.1\r\n(app-root) sqlite3 --version\r\n3.36.0 2021-06-18 18:36:39\r\n(app-root) python --version\r\nPython 3.8.11\r\n```\r\n\r\n\r\nDockerfile\r\n```\r\nFROM centos/python-38-centos7\r\n\r\nUSER root\r\n\r\nRUN yum update -y\r\nRUN yum upgrade -y\r\n\r\n\r\n# epel\r\nRUN yum -y install epel-release && yum clean all\r\n\r\n# SQLite\r\nRUN yum -y install zlib-devel geos geos-devel proj proj-devel freexl freexl-devel libxml2-devel \r\n\r\nWORKDIR /build/\r\nCOPY sqlite-autoconf-3360000.tar.gz ./\r\nRUN tar -zxf sqlite-autoconf-3360000.tar.gz\r\nWORKDIR /build/sqlite-autoconf-3360000\r\nRUN ./configure\r\nRUN make\r\nRUN make install\r\n\r\n# \r\nRUN /opt/app-root/bin/python3.8 -m pip install --upgrade pip\r\nRUN pip install sqlite-utils\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/421/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1205687423, "node_id": "I_kwDOCGYnMM5H3VR_", "number": 426, "title": "CLI docs should link to Python docs and vice versa", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": {"value": 9599, "label": "simonw"}, "milestone": null, "comments": 1, "created_at": "2022-04-15T16:05:15Z", "updated_at": "2023-07-22T22:13:22Z", "closed_at": "2023-07-22T22:13:22Z", "author_association": "OWNER", "pull_request": null, "body": "For every command/API method there should be a link to the equivalent in the other form factor.\r\n\r\nMaybe also link to the API and CLI reference pages too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/426/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1199158210, "node_id": "I_kwDOCGYnMM5HebPC", "number": 423, "title": ".extract() doesn't set foreign key when extracted columns contain NULL value", "user": {"value": 37447552, "label": "jlieth"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-04-10T20:05:30Z", "updated_at": "2022-08-27T14:45:04Z", "closed_at": "2022-08-27T14:45:04Z", "author_association": "NONE", "pull_request": null, "body": "I've run into an issue with `extract` and I don't believe this is the intended behaviour.\r\n\r\nI'm working with a database with music listening information. Currently it has one large table `listens` that contains all information. I'm trying to normalize the database by extracting relevant columns to separate tables (`artists`, `tracks`, `albums`). Not every track has an album.\r\n\r\nA simplified demonstration with just `track_title` and `album_title` columns:\r\n```ipython\r\nIn [1]: import sqlite_utils\r\n\r\nIn [2]: db = sqlite_utils.Database(memory=True)\r\n\r\nIn [3]: db[\"listens\"].insert_all([\r\n ...: {\"id\": 1, \"track_title\": \"foo\", \"album_title\": \"bar\"},\r\n ...: {\"id\": 2, \"track_title\": \"baz\", \"album_title\": None}\r\n ...: ], pk=\"id\")\r\nOut[3]: \r\n```\r\n\r\nThe track in the first row has an album, the second track doesn't. Now I extract album information into a separate column:\r\n```ipython\r\nIn [4]: db[\"listens\"].extract(columns=[\"album_title\"], table=\"albums\", fk_column=\"album_id\")\r\nOut[4]:
\r\n\r\nIn [5]: list(db[\"albums\"].rows)\r\nOut[5]: [{'id': 1, 'album_title': 'bar'}, {'id': 2, 'album_title': None}]\r\n\r\nIn [6]: list(db[\"listens\"].rows)\r\nOut[6]: \r\n[{'id': 1, 'track_title': 'foo', 'album_id': 1},\r\n {'id': 2, 'track_title': 'baz', 'album_id': None}]\r\n```\r\n\r\nThis behaves as expected -- the `album` table contains entries for both the existing album and the NULL album. The `listens` table has a foreign key only for the first row (since the album in the second row was empty).\r\n\r\nNow I want to extract the track information as well. Album information belongs to the track so I want to extract both columns to a new table.\r\n```ipython\r\nIn [7]: db[\"listens\"].extract(columns=[\"track_title\", \"album_id\"], table=\"tracks\", fk_column=\"track_id\")\r\nOut[7]:
\r\n\r\nIn [8]: list(db[\"tracks\"].rows)\r\nOut[8]: \r\n[{'id': 1, 'track_title': 'foo', 'album_id': 1},\r\n {'id': 2, 'track_title': 'baz', 'album_id': None}]\r\n\r\nIn [9]: list(db[\"listens\"].rows)\r\nOut[9]: [{'id': 1, 'track_id': 1}, {'id': 2, 'track_id': None}]\r\n```\r\n\r\nExtracting to the `tracks` table worked fine (both tracks are present with correct columns). However, the `listens` table only has a foreign key to the newly created tracks for the first row, the foreign key in the second row is NULL.\r\n\r\nChanging the order of extracts doesn't help.\r\n\r\nI poked around in the source a bit and I believe [this line](https://github.com/simonw/sqlite-utils/blob/433813612ff9b4b501739fd7543bef0040dd51fe/sqlite_utils/db.py#L1737) (essentially comparing `NULL = NULL`) is the problem, but I don't know enough about SQL to create a reliable fix myself.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/423/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1200866134, "node_id": "I_kwDOCGYnMM5Hk8NW", "number": 424, "title": "Better error message if you try to create a table with no columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-04-12T02:43:20Z", "updated_at": "2022-04-13T22:40:15Z", "closed_at": "2022-04-13T22:40:10Z", "author_association": "OWNER", "pull_request": null, "body": "Seen here:\r\n\r\n- https://github.com/simonw/geojson-to-sqlite/issues/30\r\n\r\nAttempting to create a table with no columns produced this confusing error:\r\n\r\n```\r\nFile \"/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/geojson_to_sqlite/utils.py\", line 69, in import_features\r\n db[table].create(column_types, pk=pk)\r\n File \"/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py\", line 863, in create\r\n self.db.create_table(\r\n File \"/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py\", line 517, in create_table\r\n self.execute(sql)\r\n File \"/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py\", line 236, in execute\r\n return self.conn.execute(sql)\r\nsqlite3.OperationalError: near \")\": syntax error\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/424/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1203842656, "node_id": "I_kwDOCGYnMM5HwS5g", "number": 425, "title": "`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-04-13T22:16:53Z", "updated_at": "2023-04-15T20:14:58Z", "closed_at": "2022-04-13T22:48:57Z", "author_association": "OWNER", "pull_request": null, "body": "Got this error while investigating:\r\n- #421\r\n\r\nEven though I was using the `LD_PRELOAD` trick from https://til.simonwillison.net/sqlite/ld-preload to use a newer version of SQLite.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/425/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1212701569, "node_id": "I_kwDOCGYnMM5ISFuB", "number": 427, "title": "sqlite-utils convert date parsing recipe complains about trying to parse \"*\"", "user": {"value": 1385831, "label": "wdccdw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-04-22T19:27:10Z", "updated_at": "2022-07-02T13:59:59Z", "closed_at": "2022-07-02T13:59:32Z", "author_association": "NONE", "pull_request": null, "body": "Missing values in my dataset are denoted by a single asterisk. I am trying to parse string dates into dates. This works fine for columns without missing values, but, when the column contains \"*\", I get the following:\r\n\r\n```\r\n$ sqlite-utils convert ${dbfile} details dob 'r.parsedate(value)' \r\n [------------------------------------] 0%Traceback (most recent call last):\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2508, in convert_value\r\n return fn(v)\r\n File \"\", line 2, in fn\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/recipes.py\", line 8, in parsedate\r\n parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat()\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py\", line 1368, in parse\r\n return DEFAULTPARSER.parse(timestr, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py\", line 643, in parse\r\n raise ParserError(\"Unknown string format: %s\", timestr)\r\ndateutil.parser._parser.ParserError: Unknown string format: *\r\n\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils==3.25.1', 'console_scripts', 'sqlite-utils')())\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/cli.py\", line 2698, in convert\r\n db[table].convert(\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py\", line 2524, in convert\r\n self.db.execute(sql, where_args or [])\r\n File \"/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py\", line 458, in execute\r\n return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: user-defined function raised exception\r\n```\r\n\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/427/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1239034903, "node_id": "I_kwDOCGYnMM5J2iwX", "number": 433, "title": "CLI eats my cursor", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2022-05-17T18:52:52Z", "updated_at": "2023-11-04T00:46:30Z", "closed_at": "2023-11-04T00:46:30Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I'm not sure why this happens but `sqlite-utils` makes my terminal cursor disappear after running commands like `sqlite-utils insert`. I've only noticed this behavior in `sqlite-utils`, not in any other CLI tools\r\n\r\nI can still type commands after it runs but the text cursor is invisible", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/433/reactions\", \"total_count\": 5, \"+1\": 5, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1257724585, "node_id": "I_kwDOCGYnMM5K91qp", "number": 441, "title": "Combining `rows_where()` and `search()` to limit which rows are searched", "user": {"value": 1448859, "label": "betatim"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-06-02T06:01:55Z", "updated_at": "2022-06-14T21:57:57Z", "closed_at": "2022-06-14T21:54:38Z", "author_association": "NONE", "pull_request": null, "body": "What is the right way to limit a full text search query to some rows of a table?\r\n\r\nFor example, I have a table that contains the following columns: `title`, `content`, `owner` (each row represents a document). The `owner` column is a username. It feels right to store all documents in one table, instead of having one table per owner. In particular because I'd like to full text search all documents, only documents owned by one user and documents owned by a set of users.\r\n\r\nI tried to combine `.rows_where(\"owner = ?\", \"1234\")` and `.search()` from the `Table` class but I don't think that is meant to work. I discovered `.search_sql()` as a way to generate the FTS SQL statement. By hand I can edit it to add a `AND [original].[owner] = :owner` to the `where` clause. This seems to do what I want.\r\n\r\nMy two questions:\r\n1. is adding a `AND ...` to the `where` clause actually the right thing to do or should I be doing something else (my SQL skills are low)?\r\n2. is there a built-in to sqlite-utils way to achieve this?\r\n\r\nRight now I am thinking I will make my own version of `search_sql()` that generates a query that contains an additional `owner = :owner` for my particular use-case.\r\n\r\nBonus question: is this generally useful/something to add to sqlite-utils or too niche?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/441/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1243151184, "node_id": "I_kwDOCGYnMM5KGPtQ", "number": 434, "title": "`detect_fts()` identifies the wrong table if tables have names that are subsets of each other", "user": {"value": 559711, "label": "ryascott"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-05-20T13:28:31Z", "updated_at": "2022-06-14T23:24:09Z", "closed_at": "2022-06-14T23:24:09Z", "author_association": "NONE", "pull_request": null, "body": "Windows 10\r\nPython 3.9.6\r\n\r\nWhen I was running a full text search through the Python library, I noticed that the query was being run on a different full text search table than the one I was trying to search.\r\n\r\nI took a look at the following function\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/841ad44bacaff05ec79ef78166d12e80c82ba6d7/sqlite_utils/db.py#L2213\r\n\r\nand noticed:\r\n\r\n```python\r\nsql LIKE '%VIRTUAL TABLE%USING FTS%content=%{table}%'\r\n```\r\n\r\nMy database contains tables with similar names and %{table}% was matching another table that ended differently in its name.\r\nI have included a sample test that shows this occurring:\r\n\r\nI search for Marsupials in db[\"books\"] and The Clue of the Broken Blade is returned. \r\n\r\nThis occurs since the search for Marsupials was \"successfully\" done against db[\"booksb\"] and rowid 1 is returned. \"The Clue of the Broken Blade\" has a rowid of 1 in db[\"books\"] and this is what is returned from the search.\r\n\r\n```python\r\ndef test_fts_search_with_similar_table_names(fresh_db):\r\n db = Database(memory=True)\r\n db[\"books\"].insert_all(\r\n [\r\n {\r\n \"title\": \"The Clue of the Broken Blade\",\r\n \"author\": \"Franklin W. Dixon\",\r\n },\r\n {\r\n \"title\": \"Habits of Australian Marsupials\",\r\n \"author\": \"Marlee Hawkins\",\r\n },\r\n ]\r\n )\r\n db[\"booksb\"].insert(\r\n {\r\n \"title\": \"Habits of Australian Marsupials\",\r\n \"author\": \"Marlee Hawkins\",\r\n }\r\n )\r\n\r\n db[\"booksb\"].enable_fts([\"title\", \"author\"])\r\n db[\"books\"].enable_fts([\"title\", \"author\"])\r\n\r\n\r\n query = \"Marsupials\"\r\n\r\n assert [\r\n { \"rowid\": 1,\r\n \"title\": \"Habits of Australian Marsupials\",\r\n \"author\": \"Marlee Hawkins\",\r\n },\r\n ] == list(db[\"books\"].search(query))\r\n```\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/434/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1243704847, "node_id": "I_kwDOCGYnMM5KIW4P", "number": 435, "title": "Switch to Furo documentation theme", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-05-20T21:46:39Z", "updated_at": "2022-05-20T21:56:10Z", "closed_at": "2022-05-20T21:54:43Z", "author_association": "OWNER", "pull_request": null, "body": "As seen in:\r\n- https://github.com/simonw/datasette/issues/1746\r\n- https://github.com/simonw/shot-scraper/issues/77", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/435/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1243715381, "node_id": "I_kwDOCGYnMM5KIZc1", "number": 436, "title": "Add \"copy to clipboard\" button to code examples in documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-05-20T21:53:23Z", "updated_at": "2022-05-20T21:57:53Z", "closed_at": "2022-05-20T21:57:53Z", "author_association": "OWNER", "pull_request": null, "body": "Follows:\r\n- #435\r\n\r\nImitates:\r\n- https://github.com/simonw/datasette/issues/1748\r\n\r\nI'll use https://github.com/executablebooks/sphinx-copybutton - here's the Datasette commit: https://github.com/simonw/datasette/commit/1465fea4798599eccfe7e8f012bd8d9adfac3039", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/436/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1250161887, "node_id": "I_kwDOCGYnMM5Kg_Tf", "number": 438, "title": "illegal UTF-16 surrogate", "user": {"value": 4068, "label": "frafra"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-05-26T22:49:52Z", "updated_at": "2022-05-27T08:21:53Z", "closed_at": "2022-05-27T08:21:53Z", "author_association": "NONE", "pull_request": null, "body": "I am trying to insert `https://artsdatabanken.no/Fab2018/api/export/csv` into a SQLite database, but I have an error when using `sqlite-utils`:\r\n\r\n```\r\nsqlite-utils insert --csv --delimiter \";\" --encoding=\"utf-16-le\" --pk \"Id\" csv fremmedart test.db\r\n [------------------------------------] 0%\r\nError: 'utf-16-le' codec can't decode bytes in position 98-99: illegal UTF-16 surrogate\r\n\r\nThe input you provided uses a character encoding other than utf-8.\r\n\r\nYou can fix this by passing the --encoding= option with the encoding of the file.\r\n\r\nIf you do not know the encoding, running 'file filename.csv' may tell you.\r\n\r\nIt's often worth trying: --encoding=latin-1\r\n```\r\n\r\nI tried to convert the file using `iconv -f \"utf-16le\" -t \"utf-8\"`, but I still get a similar error (slightly different position):\r\n\r\n```\r\nsqlite-utils insert --csv --delimiter \";\" --encoding=utf-8 --pk \"Id\" csv_utf8 fremmedart test.db\r\n [------------------------------------] 0%\r\nError: 'utf-8' codec can't decode byte 0xd9 in position 99: invalid continuation byte\r\n\r\nThe input you provided uses a character encoding other than utf-8.\r\n\r\nYou can fix this by passing the --encoding= option with the encoding of the file.\r\n\r\nIf you do not know the encoding, running 'file filename.csv' may tell you.\r\n\r\nIt's often worth trying: --encoding=latin-1\r\n```\r\n\r\nI have no issues reading such file using this Python code:\r\n```python\r\ncontent = open('csv', encoding='utf-16-le').read())\r\n```\r\n\r\n`in2csv` works too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/438/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1250629388, "node_id": "I_kwDOCGYnMM5KixcM", "number": 440, "title": "CSV files with too many values in a row cause errors", "user": {"value": 4068, "label": "frafra"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 20, "created_at": "2022-05-27T10:54:44Z", "updated_at": "2022-06-14T22:23:01Z", "closed_at": "2022-06-14T20:12:46Z", "author_association": "NONE", "pull_request": null, "body": "*Original title: csv.DictReader can have None as key*\r\n\r\nIn some cases, `csv.DictReader` can have `None` as key for unnamed columns, and a list of values as value.\r\n`sqlite_utils.utils.rows_from_file` cannot handle that:\r\n\r\n```python\r\nurl=\"https://artsdatabanken.no/Fab2018/api/export/csv\"\r\ndb = sqlite_utils.Database(\":memory\")\r\n\r\nwith urlopen(url) as fab:\r\n reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding=\"utf-16le\") \r\n db[\"fab2018\"].insert_all(reader, pk=\"Id\")\r\n```\r\n\r\nResult:\r\n```\r\nTraceback (most recent call last):\r\n File \"\", line 3, in \r\n File \"/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py\", line 2924, in insert_all\r\n chunk = list(chunk)\r\n File \"/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py\", line 3454, in fix_square_braces\r\n if any(\"[\" in key or \"]\" in key for key in record.keys()):\r\n File \"/home/user/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/db.py\", line 3454, in \r\n if any(\"[\" in key or \"]\" in key for key in record.keys()):\r\nTypeError: argument of type 'NoneType' is not iterable\r\n```\r\n\r\nCode:\r\nhttps://github.com/simonw/sqlite-utils/blob/59be60c471fd7a2c4be7f75e8911163e618ff5ca/sqlite_utils/db.py#L3454\r\n\r\n`sqlite-utils insert` from command line is not affected by this issue.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/440/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1269886084, "node_id": "I_kwDOCGYnMM5LsOyE", "number": 442, "title": "`maximize_csv_field_size_limit()` utility function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-06-13T19:54:54Z", "updated_at": "2022-06-14T21:55:15Z", "closed_at": "2022-06-14T21:31:49Z", "author_association": "OWNER", "pull_request": null, "body": "This code here runs only if `cli.py` is imported: https://github.com/simonw/sqlite-utils/blob/7ddf5300886a32d6daf60cf1d71efe492b65c87e/sqlite_utils/cli.py#L50-L59\r\n\r\nI found myself needing the same fix in another library:\r\n\r\n- https://github.com/simonw/datasette-socrata/issues/13\r\n\r\nIt should be a documented utility function.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/442/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1269998342, "node_id": "I_kwDOCGYnMM5LsqMG", "number": 443, "title": "Make `utils.rows_from_file()` a documented API", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-06-13T21:53:24Z", "updated_at": "2022-06-20T19:49:37Z", "closed_at": "2022-06-14T20:12:46Z", "author_association": "OWNER", "pull_request": null, "body": "> `rows_from_file()` isn't part of the documented API but maybe it should be!\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/443/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1277295119, "node_id": "I_kwDOCGYnMM5MIfoP", "number": 445, "title": "`sqlite_utils.utils.TypeTracker` should be a documented API", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-06-20T19:08:28Z", "updated_at": "2022-06-20T19:49:02Z", "closed_at": "2022-06-20T19:46:58Z", "author_association": "OWNER", "pull_request": null, "body": "I've used it in a couple of external places now:\r\n\r\n- https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/__init__.py#L264-L280\r\n- https://github.com/simonw/datasette-lite/blob/caa8eade10f0321c64f9f65c4561186f02d57c5b/webworker.js#L55-L64\r\n\r\nRefs:\r\n- https://github.com/simonw/datasette-lite/issues/32", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/445/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1277328147, "node_id": "I_kwDOCGYnMM5MInsT", "number": 446, "title": "Use Just to automate running tests and linters locally", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-06-20T19:51:09Z", "updated_at": "2022-06-21T19:28:35Z", "closed_at": "2022-06-20T19:54:50Z", "author_association": "OWNER", "pull_request": null, "body": "I keep committing code that fails additional tests like `mypy` and `flake8` and `black`. Automate those using Just.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/446/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1278571700, "node_id": "I_kwDOCGYnMM5MNXS0", "number": 447, "title": "Incorrect syntax highlighting in docs CLI reference", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-06-21T14:53:10Z", "updated_at": "2022-06-21T18:48:47Z", "closed_at": "2022-06-21T18:48:46Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/stable/cli-reference.html#insert\r\n\r\n![CE020DDA-27FB-49C3-9EA6-37457DC4C321](https://user-images.githubusercontent.com/9599/174830380-06530537-b870-41c0-a8af-03c7fa720c6f.jpeg)\r\n\r\nIt looks like Python keywords are being incorrectly highlighted here.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/447/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1279144769, "node_id": "I_kwDOCGYnMM5MPjNB", "number": 448, "title": "Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto'", "user": {"value": 236907, "label": "mungewell"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-06-21T21:48:27Z", "updated_at": "2023-05-08T22:01:00Z", "closed_at": "2023-05-08T22:01:00Z", "author_association": "NONE", "pull_request": null, "body": "Attempting to run the example given here (without extra bracket ;-):\r\nhttps://sqlite-utils.datasette.io/en/stable/python-api.html#reading-rows-from-a-file\r\n```\r\nfrom sqlite_utils.utils import rows_from_file\r\nimport io\r\n\r\nrows, format = rows_from_file(io.StringIO(\"id,name\\n1,Cleo\"))\r\nprint(list(rows), format)\r\n# Outputs [{'id': '1', 'name': 'Cleo'}] Format.CSV\r\n```\r\n\r\nGives error\r\n```\r\n>\"c:\\Program Files\\Python37\\python.exe\" test2.py\r\nTraceback (most recent call last):\r\n File \"test2.py\", line 4, in \r\n rows, format = rows_from_file(io.StringIO(\"id,name\\n1,Cleo\"))\r\n File \"C:\\Users\\swood\\Downloads\\sqlite-utils-main-20220621\\sqlite-utils-main\\sqlite_utils\\utils.py\", line 300, in rows_from_file\r\n first_bytes = buffered.peek(2048).strip()\r\nAttributeError: '_io.StringIO' object has no attribute 'readinto'\r\n```\r\n\r\nI am running Python on Windows.\r\n```\r\n>\"c:\\Program Files\\Python37\\python.exe\"\r\nPython 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/448/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1279863844, "node_id": "I_kwDOCGYnMM5MSSwk", "number": 449, "title": "Utilities for duplicating tables and creating a table with the results of a query", "user": {"value": 1690072, "label": "davidleejy"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-06-22T09:41:43Z", "updated_at": "2022-07-15T21:46:13Z", "closed_at": "2022-07-15T21:21:36Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "is there a duplicate table functionality? Otherwise, I'd be happy to submit a PR.\r\n\r\nIn sqlite3 it would look like:\r\n\r\n```python\r\nimport sqlite3 as sl\r\n\r\ncon = sl.connect('prompt-tune.db')\r\n\r\ndef db_duplicate_table(table_name, table_name_new, con=con):\r\n # Duplicates table `table_name` to a new table `table_name_new`.\r\n try:\r\n cur = con.cursor()\r\n cur.execute(f\"\"\"CREATE TABLE {table_name_new} AS SELECT * FROM {table_name}\"\"\")\r\n except Exception as e:\r\n print(e)\r\n finally:\r\n cur.close()\r\n\r\ndb_duplicate_table('orig_table', 'new_table')\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/449/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1306548397, "node_id": "I_kwDOCGYnMM5N4Fit", "number": 454, "title": "CLI command for duplicating tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-07-15T21:31:27Z", "updated_at": "2022-07-15T21:48:23Z", "closed_at": "2022-07-15T21:45:51Z", "author_association": "OWNER", "pull_request": null, "body": "CLI equivalent of:\r\n- #449", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/454/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1292060682, "node_id": "I_kwDOCGYnMM5NA0gK", "number": 450, "title": "Add --ignore option to more commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-07-02T13:52:02Z", "updated_at": "2022-07-15T22:39:09Z", "closed_at": "2022-07-15T22:37:45Z", "author_association": "OWNER", "pull_request": null, "body": "As seen in https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key\r\n\r\nCould make this TIL trick unnecessary: https://til.simonwillison.net/bash/ignore-errors", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/450/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1298531653, "node_id": "I_kwDOCGYnMM5NZgVF", "number": 451, "title": "Make sqlite_utils.utils.chunks a documented function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-07-08T06:01:04Z", "updated_at": "2022-07-15T22:09:34Z", "closed_at": "2022-07-15T21:59:33Z", "author_association": "OWNER", "pull_request": null, "body": "I want to use it in another project: https://github.com/simonw/sqlite-utils/blob/8a9fe6498faf783a1fdeb1793e661ad194a05267/sqlite_utils/utils.py#L471-L474", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/451/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1303169663, "node_id": "I_kwDOCGYnMM5NrMp_", "number": 453, "title": "'unclosed file' warning when using insert_upsert_implementation from Python", "user": {"value": 311257, "label": "makkus"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-07-13T09:34:35Z", "updated_at": "2022-07-15T21:52:25Z", "closed_at": "2022-07-15T21:52:21Z", "author_association": "NONE", "pull_request": null, "body": "I'm using the `[insert_upsert_implementation](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py)` function directly in my Python code to import a csv file with all the bells and whistles `sqlite-utils` provides, but I'm getting a resource warning that a io.TextWrapper object is not closed.\r\n\r\nThe warning goes away when wrapping the code from [this line](https://github.com/simonw/sqlite-utils/blob/42440d6345c242ee39778045e29143fb550bd2c2/sqlite_utils/cli.py#L924) in a try/finally block like:\r\n\r\n```\r\ntry:\r\n ...\r\n ...\r\nfinally:\r\n decoded.close()\r\n```\r\n(might be that `sniff_buffer` must also be closed if non null, but I might be wrong)\r\n\r\nI suspect Python closes the reference automatically when the sqlite-utils cli run is done, but since my code doesn't exit, I'm getting the warning.\r\n\r\nAlternatively, it'd be cool if the 'import csv/tsv' functionality could be added directly to the Database class.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/453/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1320243134, "node_id": "I_kwDOCGYnMM5OsU--", "number": 458, "title": "Support custom names for registered functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8355157, "label": "3.29"}, "comments": 1, "created_at": "2022-07-28T00:13:00Z", "updated_at": "2022-08-27T03:56:01Z", "closed_at": "2022-07-28T00:13:57Z", "author_association": "OWNER", "pull_request": null, "body": "In this example:\r\n\r\n```python\r\n @db.register_function\r\n def reverse_string(s):\r\n return \"\".join(reversed(list(s)))\r\n\r\n print(db.execute('select reverse_string(\"hello\")').fetchone()[0])\r\n```\r\nThere's currently no way to over-ride the automatically selected name for the SQL function.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/458/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1338001039, "node_id": "I_kwDOCGYnMM5PwEaP", "number": 464, "title": "Link from documentation to source code", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-08-13T16:19:57Z", "updated_at": "2022-08-17T23:38:03Z", "closed_at": "2022-08-17T23:38:03Z", "author_association": "OWNER", "pull_request": null, "body": "Twitter conversation asking for ways to automate this here: https://twitter.com/simonw/status/1558260492015046656", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/464/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1348169997, "node_id": "I_kwDOCGYnMM5QW3EN", "number": 467, "title": "Mechanism for ensuring a table has all the columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8355157, "label": "3.29"}, "comments": 13, "created_at": "2022-08-23T15:50:23Z", "updated_at": "2022-08-27T23:19:41Z", "closed_at": "2022-08-27T23:17:56Z", "author_association": "OWNER", "pull_request": null, "body": "Suggested by @jefftriplett on Discord: https://discord.com/channels/823971286308356157/997738192360964156/1011655389063958600", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/467/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1352932716, "node_id": "I_kwDOCGYnMM5QpB1s", "number": 471, "title": "sqlite-utils query --functions mechanism for registering extra functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8355157, "label": "3.29"}, "comments": 12, "created_at": "2022-08-27T03:57:53Z", "updated_at": "2022-09-07T03:46:26Z", "closed_at": "2022-08-27T05:10:57Z", "author_association": "OWNER", "pull_request": null, "body": "It would be really cool if you could register additional custom SQL functions for use with the `sqlite-utils query` command - something like this:\r\n\r\n```\r\nsqlite-utils data.db 'update images set domain = extract_domain(url)' --functions '\r\nfrom urllib.parse import urlparse\r\n\r\ndef extract_domain(url):\r\n return urlparse(url).netloc\r\n'\r\n```\r\nEvery function defined in that code block would be registered with the connection, unless the name began with an underscore.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/471/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1352931464, "node_id": "I_kwDOCGYnMM5QpBiI", "number": 469, "title": "sqlite-utils rows --order option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8355157, "label": "3.29"}, "comments": 1, "created_at": "2022-08-27T03:49:51Z", "updated_at": "2022-08-27T04:30:49Z", "closed_at": "2022-08-27T04:10:32Z", "author_association": "OWNER", "pull_request": null, "body": "For consistency with `search`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#search\r\n\r\n```\r\n -o, --order TEXT Order by ('column' or 'column desc')\r\n```\r\n\r\nI wanted to run `sqlite-utils rows db.db mytable --order 'rowid desc'` to see the most recently imported rows.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/469/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1352932038, "node_id": "I_kwDOCGYnMM5QpBrG", "number": 470, "title": "Upgrade `--load-extension` to accept entrypoints like Datasette", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8355157, "label": "3.29"}, "comments": 6, "created_at": "2022-08-27T03:53:20Z", "updated_at": "2022-08-27T05:55:49Z", "closed_at": "2022-08-27T05:55:48Z", "author_association": "OWNER", "pull_request": null, "body": "Imitate:\r\n- https://github.com/simonw/datasette/pull/1789\r\n```\r\n# would load default entrypoint like before\r\ndatasette data.db --load-extension ext\r\n\r\n# loads the extensions with the \"sqlite3_foo_init\" entrpoint\r\ndatasette data.db --load-extension ext:sqlite3_foo_init\r\n\r\n# loads the extensions with the \"sqlite3_bar_init\" entrpoint\r\ndatasette data.db --load-extension ext:sqlite3_bar_init\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/470/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}