{"id": 1175744654, "node_id": "I_kwDOCGYnMM5GFHCO", "number": 417, "title": "insert fails on JSONL with whitespace", "user": {"value": 9954, "label": "blaine"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-03-21T17:58:14Z", "updated_at": "2022-03-25T21:19:06Z", "closed_at": "2022-03-25T21:17:13Z", "author_association": "NONE", "pull_request": null, "body": "Any JSON that is newline-delimited and has whitespace (newlines) between the start of a JSON object and an attribute fails due to a parse error.\r\n\r\ne.g. given the valid JSONL:\r\n\r\n```{\r\n \"attribute\": \"value\"\r\n}\r\n{\r\n \"attribute\": \"value2\"\r\n}\r\n```\r\n\r\nI would expect that `sqlite-utils insert --nl my.db mytable file.jsonl` would properly import the data into `mytable`. However, the following error is thrown instead:\r\n\r\n`json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 1 (char 2)`\r\n\r\nIt makes sense that since the file is intended to be newline separated, the thing being parsed is \"{\" (which obviously fails), however the default newline-separated output of `jq` isn't compact. Using `jq -c` avoids this problem, but the fix is unintuitive and undocumented.\r\n\r\nProposed solutions:\r\n1. Default to a \"loose\" newline-separated parse; this could be implemented internally as [the equivalent of] a `jq -c` filter ahead of the insert step.\r\n2. Catch the JSONDecodeError (or pre-empt it in the case of a record === \"{\\n\") and give the user a \"it looks like your json isn't _actually_ newline-delimited; try running it through `jq -c` instead\" error message.\r\n\r\nIt might just have been too early in the morning when I was playing with this, but running pipes of data through sqlite-utils without the 'knack' of it led to some false starts.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/417/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1181236173, "node_id": "I_kwDOCGYnMM5GaDvN", "number": 422, "title": "Reconsider not running convert functions against null values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-03-25T20:22:40Z", "updated_at": "2022-03-25T20:23:21Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I just got caught out by the fact that `None` values are not processed by the `.convert()` mechanism https://github.com/simonw/sqlite-utils/blob/0b7b80bd40fe86e4d66a04c9f607d94991c45c0b/sqlite_utils/db.py#L2504-L2510\r\n\r\nI had run this code while working on #420 and I wasn't sure why it didn't work:\r\n\r\n```\r\n$ sqlite-utils add-column content.db articles score float\r\n$ sqlite-utils convert content.db articles score '\r\nimport random\r\nrandom.seed(10)\r\n\r\ndef convert(value):\r\n global random\r\n return random.random()\r\n'\r\n```\r\nThe reason it didn't work is that the newly added `score` column was full of `null` values.\r\n\r\nI fixed it by doing this instead:\r\n\r\n $ sqlite-utils add-column content.db articles score float --not-null-default 1.0\r\n\r\nBut this indicates to me that the design of `convert()` here may be incorrect.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/422/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1178456794, "node_id": "I_kwDOCGYnMM5GPdLa", "number": 418, "title": "Add generated files to .gitignore", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-03-23T17:48:12Z", "updated_at": "2022-03-24T21:01:44Z", "closed_at": "2022-03-24T21:01:44Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I end up with these in my local directory:\r\n\r\n\t.hypothesis/\r\n\tPipfile\r\n\tPipfile.lock\r\n\tpyproject.toml\r\n\r\nMight as well gitignore them.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/418/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1178484369, "node_id": "PR_kwDOCGYnMM405rPe", "number": 419, "title": "Ignore common generated files", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-03-23T18:06:22Z", "updated_at": "2022-03-24T21:01:44Z", "closed_at": "2022-03-24T21:01:44Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/419", "body": "Closes #418 \r\n\r\nThis adds four files to `.gitignore`:\r\n\r\n\t.hypothesis/\r\n\tPipfile\r\n\tPipfile.lock\r\n\tpyproject.toml\r\n\r\nThose are all generated in the course of development and testing.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/419/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1091819089, "node_id": "I_kwDOCGYnMM5BE9ZR", "number": 360, "title": "MemoryError", "user": {"value": 559453, "label": "nzaar9"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-01T13:39:17Z", "updated_at": "2022-03-21T04:22:46Z", "closed_at": "2022-03-21T04:22:46Z", "author_association": "NONE", "pull_request": null, "body": "HI, when dealing with large json file (~170GB) i got the following error \r\n```\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 1126, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 1051, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 1657, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 1393, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/lib/python3/dist-packages/click/core.py\", line 752, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/cli.py\", line 1300, in memory\r\n rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding)\r\n File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py\", line 185, in rows_from_file\r\n return rows_from_file(buffered, format=Format.JSON)\r\n File \"/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py\", line 156, in rows_from_file\r\n decoded = json.load(fp)\r\n File \"/usr/lib/python3.9/json/__init__.py\", line 293, in load\r\n return loads(fp.read(),\r\nMemoryError\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/360/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1171599874, "node_id": "I_kwDOCGYnMM5F1TIC", "number": 415, "title": "Convert with `--multi` and `--dry-run` flag does not work", "user": {"value": 3976183, "label": "dotcs"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-03-16T21:59:46Z", "updated_at": "2022-03-21T04:18:24Z", "closed_at": "2022-03-21T04:18:24Z", "author_association": "NONE", "pull_request": null, "body": "It's not possible to combine `--multi` and `--dry-run` flag in the `convert` command.\r\n\r\nLet's first create a simple database from JSON string\r\n\r\n```console\r\n$ echo '[{\"foo\": \"abc\"}]' | sqlite-utils insert demo.db demo -\r\n$ sqlite-utils query demo.db \"SELECT * FROM demo\" \r\n[{\"foo\": \"abc\"}]\r\n```\r\n\r\nand then try to convert the \"foo\" column with a static value \"bar\" (see docs [Converting a column into multiple columns](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-a-column-into-multiple-columns))\r\n\r\n```console\r\n$ sqlite-utils convert demo.db demo foo '{\"foo\": \"bar\"}' --multi --dry-run\r\nTraceback (most recent call last):\r\n File \"/home/dotcs/anaconda3/envs/tools/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/home/dotcs/anaconda3/envs/tools/lib/python3.9/site-packages/sqlite_utils/cli.py\", line 2686, in convert\r\n for row in db.conn.execute(sql, where_args).fetchall():\r\nsqlite3.OperationalError: user-defined function raised exception\r\n```\r\n\r\nBut without the `--dry-run` flag it does work as expected:\r\n\r\n```console\r\n$ sqlite-utils convert demo.db demo foo '{\"foo\": \"bar\"}' --multi\r\n$ sqlite-utils query demo.db \"SELECT * FROM demo\" \r\n[{\"foo\": \"bar\"}]\r\n```\r\n\r\n```console\r\n$ sqlite-utils --version\r\nsqlite-utils, version 3.25.1\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/415/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688351054, "node_id": "MDU6SXNzdWU2ODgzNTEwNTQ=", "number": 140, "title": "Idea: insert-files mechanism for adding extra columns with fixed values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T20:57:36Z", "updated_at": "2022-03-20T19:45:45Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Say for example you want to populate a `file_type` column with the value `gif`. That could work like this:\r\n\r\n```\r\nsqlite-utils insert-files gifs.db images *.gif \\\r\n -c path -c md5 -c last_modified:mtime \\\r\n -c file_type:text:gif --pk=path\r\n```\r\nSo a column defined as a `text` column with a value that follows a second colon.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 675753042, "node_id": "MDU6SXNzdWU2NzU3NTMwNDI=", "number": 131, "title": "sqlite-utils insert: options for column types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-09T18:59:11Z", "updated_at": "2022-03-15T13:21:42Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs.\r\n\r\nIt would be useful if you could do the following:\r\n\r\n- automatically detects the column types based on eg the first 1000 records\r\n- explicitly state the rule for specific columns\r\n\r\n`--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types`\r\n\r\nFor specific columns maybe this:\r\n\r\n sqlite-utils insert db.db images images.tsv \\\r\n --tsv \\\r\n -c id int \\\r\n -c score float", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1145882578, "node_id": "I_kwDOCGYnMM5ETMfS", "number": 408, "title": "`deterministic=True` fails on versions of SQLite prior to 3.8.3", "user": {"value": 24938923, "label": "learning4life"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2022-02-21T14:36:43Z", "updated_at": "2022-03-13T16:54:09Z", "closed_at": "2022-03-02T00:38:11Z", "author_association": "NONE", "pull_request": null, "body": "Hi, love your work.\r\n\r\nI am unable to lookup indexes in a database using sqlite-utils:\r\n\r\n`\r\nsqlite-utils indexes city_spec.db --table`\r\n\r\nor\r\n\r\n`sqlite-utils indexes city_spec.db MyTable\r\n`\r\n\r\n**Software**\r\nsqlite-utils, version 3.24\r\nsqlite3 --version: 3.36.0 \r\n\r\n**Output:**\r\n\r\nTraceback (most recent call last):\r\n File \"/opt/app-root/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1128, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1053, in main\r\n rv = self.invoke(ctx)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1659, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1395, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/decorators.py\", line 26, in new_func\r\n return f(get_current_context(), *args, **kwargs)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py\", line 2123, in indexes\r\n ctx.invoke(\r\n File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 754, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py\", line 1624, in query\r\n db.register_fts4_bm25()\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 403, in register_fts4_bm25\r\n self.register_function(rank_bm25, deterministic=True)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 399, in register_function\r\n register(fn)\r\n File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 392, in register\r\n self.conn.create_function(name, arity, fn, **kwargs)\r\nsqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/408/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1160034488, "node_id": "I_kwDOCGYnMM5FJLi4", "number": 411, "title": "Support for generated columns", "user": {"value": 25778, "label": "eyeseast"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2022-03-04T20:41:33Z", "updated_at": "2022-03-11T22:32:43Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "This is a fairly new feature -- SQLite version 3.31.0 (2020-01-22) -- that I, admittedly, haven't gotten to work yet. But it looks _incredibly_ useful: https://dgl.cx/2020/06/sqlite-json-support\r\n\r\nI'm not sure if this is an option on `add-column` or a separate command like `add-generated-column`. Either way, it needs an argument to populate it. It could be something like this:\r\n\r\n```sh\r\nsqlite-utils add-column data.db table-name generated --as 'json_extract(data, \"$.field\")' --virtual\r\n```\r\n\r\nMore here: https://www.sqlite.org/gencol.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/411/reactions\", \"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1166731361, "node_id": "I_kwDOCGYnMM5Fiuhh", "number": 414, "title": "I forgot to include the changelog in the 3.25.1 release", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-03-11T18:32:36Z", "updated_at": "2022-03-11T18:40:39Z", "closed_at": "2022-03-11T18:40:39Z", "author_association": "OWNER", "pull_request": null, "body": "I pushed a release for https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 but forgot to include the release notes in `docs/changelog.rst`\r\n\r\nThis means https://sqlite-utils.datasette.io/en/stable/changelog.html isn't showing them.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/414/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1166587040, "node_id": "I_kwDOCGYnMM5FiLSg", "number": 413, "title": "Display autodoc type information more legibly", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-03-11T15:58:20Z", "updated_at": "2022-03-11T18:07:10Z", "closed_at": "2022-03-11T18:07:10Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/3.25/reference.html#sqlite_utils.db.Table.insert looks like this at the moment:\r\n\r\n\"image\"\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/413/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1149729902, "node_id": "PR_kwDOCGYnMM4zbaJy", "number": 410, "title": "Correct spelling mistakes (found with codespell)", "user": {"value": 3818, "label": "EdwardBetts"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-24T20:44:18Z", "updated_at": "2022-03-06T08:48:29Z", "closed_at": "2022-03-01T21:05:29Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/410", "body": null, "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/410/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1063388037, "node_id": "I_kwDOCGYnMM4_YgOF", "number": 343, "title": "Provide function to generate hash_id from specified columns", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-25T10:12:12Z", "updated_at": "2022-03-02T04:25:25Z", "closed_at": "2022-03-02T04:25:25Z", "author_association": "NONE", "pull_request": null, "body": "Hi\r\n\r\nI note that you define `_hash()` to create a `hash_id` from non-id column values in a table [here](https://github.com/simonw/sqlite-utils/blob/8f386a0d300d1b1c76132bb75972b755049fb742/sqlite_utils/db.py#L2996).\r\n\r\nIt would be useful to be able to call a complementary function to generate a corresponding `_id` from a subset of specified columns when adding items to another table, eg to support the creation of foreign keys.\r\n\r\nOr is there a better pattern for doing that?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/343/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1125297737, "node_id": "I_kwDOCGYnMM5DEq5J", "number": 402, "title": "Advanced class-based `conversions=` mechanism", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 14, "created_at": "2022-02-06T19:47:41Z", "updated_at": "2022-02-16T10:18:55Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `conversions=` parameter works like this at the moment: https://sqlite-utils.datasette.io/en/3.23/python-api.html#converting-column-values-using-sql-functions\r\n\r\n```python\r\ndb[\"places\"].insert(\r\n {\"name\": \"Wales\", \"geometry\": wkt},\r\n conversions={\"geometry\": \"GeomFromText(?, 4326)\"},\r\n)\r\n```\r\nThis proposal is to support values in that dictionary that are objects, not strings, which can represent more complex conversions - spun out from #399.\r\n\r\nNew proposed mechanism:\r\n```python\r\nfrom sqlite_utils.utils import LongitudeLatitude\r\n\r\ndb[\"places\"].insert(\r\n {\r\n \"name\": \"London\",\r\n \"point\": (-0.118092, 51.509865)\r\n },\r\n conversions={\"point\": LongitudeLatitude},\r\n)\r\n```\r\nHere `LongitudeLatitude` is a magical value which does TWO things: it sets up the `GeomFromText(?, 4326)` SQL function, and it handles converting the `(51.509865, -0.118092)` tuple into a `POINT({} {})` string.\r\n\r\nThis would involve a change to the `conversions=` contract - where it usually expects a SQL string fragment, but it can also take an object which combines that SQL string fragment with a Python conversion function.\r\n\r\nBest of all... this resolves the `lat, lon` v.s. `lon, lat` dilemma because you can use `from sqlite_utils.utils import LongitudeLatitude` OR `from sqlite_utils.utils import LatitudeLongitude` depending on which you prefer!\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030739566_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/402/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1138948786, "node_id": "PR_kwDOCGYnMM4y3yW0", "number": 407, "title": "Add SpatiaLite helpers to CLI", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-02-15T16:50:17Z", "updated_at": "2022-02-16T01:49:40Z", "closed_at": "2022-02-16T00:58:08Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/407", "body": "Closes #398 \r\n\r\nThis adds SpatiaLite helpers to the CLI. \r\n\r\n```sh\r\n# init spatialite when creating a database\r\nsqlite-utils create database.db --enable-wal --init-spatialite\r\n\r\n# add geometry columns\r\n# needs a database, table, geometry column name, type, with optional SRID and not-null\r\n# this will throw an error if the table doesn't already exist\r\nsqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null\r\n\r\n# spatial index an existing table/column\r\n# this will throw an error it the table and column don't exist\r\nsqlite-utils create-spatial-index database.db table-name geometry\r\n```\r\n\r\nDocs and tests are included. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/407/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1128139375, "node_id": "I_kwDOCGYnMM5DPgpv", "number": 405, "title": "`Database(memory_name=\"name\")` constructor argument", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-02-09T07:15:03Z", "updated_at": "2022-02-16T01:23:16Z", "closed_at": "2022-02-16T01:23:16Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite in-memory databases can be named, in which case multiple connections can be opened to a shared in-memory database running within the same process.\r\n\r\nDatasette supports this - SQLite could support it too.\r\n\r\nhttps://docs.datasette.io/en/0.60.2/internals.html#database-ds-path-none-is-mutable-false-is-memory-false-memory-name-none", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/405/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1124237013, "node_id": "I_kwDOCGYnMM5DAn7V", "number": 398, "title": "Add SpatiaLite helpers to CLI", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2022-02-04T14:01:28Z", "updated_at": "2022-02-16T01:02:29Z", "closed_at": "2022-02-16T00:58:07Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Now that #385 is merged, add CLI versions of those methods.\r\n\r\n```sh\r\n# init spatialite\r\nsqlite-utils init-spatialite database.db\r\n\r\n# or maybe/also\r\nsqlite-utils create database.db --enable-wal --spatialite\r\n\r\n# add geometry columns\r\n# needs a database, table, geometry column name, type, with optional SRID and not-null\r\n# this needs to create a table if it doesn't already exist\r\nsqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null\r\n\r\n# spatial index an existing table/column\r\nsqlite-utils create-spatial-index database.db table-name geometry\r\n```\r\n\r\nShould be mostly straightforward. The one thing worth highlighting in docs is that geometry columns can only be added to existing tables. Trying to add a geometry column to a table that doesn't exist yet might mean you have a schema like `{\"rowid\": int, \"geometry\": bytes}`. Might be worth nudging people to explicitly create a table first, then add geometry columns.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/398/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1128120451, "node_id": "I_kwDOCGYnMM5DPcCD", "number": 404, "title": "Add example of `--convert` to the help for `sqlite-utils insert`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-02-09T06:49:09Z", "updated_at": "2022-02-09T06:56:35Z", "closed_at": "2022-02-09T06:55:16Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/3.23/cli-reference.html#insert would be more useful if it included an example of `--convert` in action.\r\n\r\nI can maybe use an example from https://simonwillison.net/2022/Jan/11/sqlite-utils/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/404/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1126692066, "node_id": "I_kwDOCGYnMM5DJ_Ti", "number": 403, "title": "Document how to add a primary key to a rowid table using `sqlite-utils transform --pk`", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-02-08T01:39:40Z", "updated_at": "2022-02-09T04:22:43Z", "closed_at": "2022-02-08T19:33:59Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "*Original title: Add option for adding a new, serial, primary key*\r\n\r\nsometimes we have tables that don't have primary keys, but ought to have them. we *can* use rowid for that, but it would often be nicer to have an explicit primary key. using the current value of rowid would be fine.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/403/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1125081640, "node_id": "I_kwDOCGYnMM5DD2Io", "number": 401, "title": "Update SpatiaLite example in the documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-02-06T02:02:07Z", "updated_at": "2022-02-06T02:05:03Z", "closed_at": "2022-02-06T02:03:24Z", "author_association": "OWNER", "pull_request": null, "body": "This one here: https://sqlite-utils.datasette.io/en/3.23/python-api.html#converting-column-values-using-sql-functions\r\n\r\nIt should take advantage of the new methods from:\r\n- #79", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/401/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1125077063, "node_id": "I_kwDOCGYnMM5DD1BH", "number": 400, "title": "`sqlite-utils create-table` ... `--if-not-exists`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-06T01:32:53Z", "updated_at": "2022-02-06T01:34:53Z", "closed_at": "2022-02-06T01:34:46Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by:\r\n- #397\r\n\r\nTo match the option on `create-index`: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-index\r\n\r\n```\r\n --if-not-exists Ignore if index already exists\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/400/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1123903919, "node_id": "I_kwDOCGYnMM5C_Wmv", "number": 397, "title": "Support IF NOT EXISTS for table creation", "user": {"value": 738408, "label": "rafguns"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-02-04T07:41:15Z", "updated_at": "2022-02-06T01:30:46Z", "closed_at": "2022-02-06T01:29:01Z", "author_association": "NONE", "pull_request": null, "body": "Currently, I have a bunch of code that looks like this:\r\n\r\n```python\r\nsubjects = db[\"subjects\"] if db[\"subjects\"].exists() else db[\"subjects\"].create({\r\n ...\r\n})\r\n```\r\nIt would be neat if sqlite-utils could simplify that by supporting `CREATE TABLE IF NOT EXISTS`, so that I'd be able to write, e.g.\r\n\r\n```python\r\nsubjects = db[\"subjects\"].create({...}, if_not_exists=True)\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/397/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683805434, "node_id": "MDU6SXNzdWU2ODM4MDU0MzQ=", "number": 135, "title": "Code for finding SpatiaLite in the usual locations", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-21T20:15:34Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-08-21T20:30:13Z", "author_association": "OWNER", "pull_request": null, "body": "I built this for `shapefile-to-sqlite` but it would be useful in `sqlite-utils` too:\r\n\r\nhttps://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19\r\n\r\n```python\r\nSPATIALITE_PATHS = (\r\n \"/usr/lib/x86_64-linux-gnu/mod_spatialite.so\",\r\n \"/usr/local/lib/mod_spatialite.dylib\",\r\n)\r\n```\r\n\r\nhttps://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109\r\n\r\n```python\r\ndef find_spatialite():\r\n for path in SPATIALITE_PATHS:\r\n if os.path.exists(path):\r\n return path\r\n return None\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683812642, "node_id": "MDU6SXNzdWU2ODM4MTI2NDI=", "number": 136, "title": "--load-extension=spatialite shortcut option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-21T20:31:25Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-16T19:14:32Z", "author_association": "OWNER", "pull_request": null, "body": "In conjunction with #135 - this would do the same thing as `--load-extension=path-to-spatialite` (see #134)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 723460107, "node_id": "MDU6SXNzdWU3MjM0NjAxMDc=", "number": 187, "title": "Maybe: Utility method / CLI tool for initializing SpatiaLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-16T19:04:03Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-16T19:15:13Z", "author_association": "OWNER", "pull_request": null, "body": "> I think this should initialize SpatiaLite against the current database if it has not been initialized already.\r\n>\r\n> Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/187/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 723708310, "node_id": "MDU6SXNzdWU3MjM3MDgzMTA=", "number": 188, "title": "About loading spatialite", "user": {"value": 30607, "label": "aborruso"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-10-17T08:47:02Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-17T08:52:58Z", "author_association": "NONE", "pull_request": null, "body": "Hi @simonw ,\r\nIf I run\r\n\r\n```\r\nsqlite3\r\n.load /usr/local/lib/mod_spatialite.so\r\nselect spatialite_version();\r\n```\r\n\r\nI have `5.0.0`.\r\n\r\n![image](https://user-images.githubusercontent.com/30607/96332706-d8cd3300-1065-11eb-906b-daf99963198e.png)\r\n\r\n\r\nIf I run\r\n\r\n```\r\nsqlite-utils :memory: \"select spatialite_version()\" --load-extension=spatialite\r\n```\r\n\r\nI have\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/home/aborruso/.local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 936, in query\r\n _load_extensions(db, load_extension)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 1326, in _load_extensions\r\n db.conn.load_extension(ext)\r\nTypeError: argument 1 must be str, not None\r\n```\r\n\r\nHow to load properly spatialite extension in sqlite-utils?\r\n\r\nThank you very muc", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/188/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1102899312, "node_id": "PR_kwDOCGYnMM4w_p22", "number": 385, "title": "Add new spatialite helper methods", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 16, "created_at": "2022-01-14T03:57:30Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2022-02-04T05:55:10Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/385", "body": "Refs #79 \r\n\r\nThis PR adds three new Spatialite-related methods to Database and Table:\r\n\r\n- `Database.init_spatialite` loads the Spatialite extension and initializes it\r\n- `Table.add_geometry_column` adds a geometry column\r\n- `Table.create_spatial_index` creates a spatial index\r\n\r\nHas tests and documentation. Feedback very welcome.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/385/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 534507142, "node_id": "MDU6SXNzdWU1MzQ1MDcxNDI=", "number": 69, "title": "Feature request: enable extensions loading", "user": {"value": 30607, "label": "aborruso"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-12-08T08:06:25Z", "updated_at": "2022-02-05T00:04:25Z", "closed_at": "2020-10-16T18:42:49Z", "author_association": "NONE", "pull_request": null, "body": "Hi, it would be great to add a parameter that enables the load of a sqlite extension you need.\r\n\r\nSomething like \"-ext modspatialite\".\r\n\r\nIn this way your great tool would be even more comfortable and powerful.\r\n\r\n\r\nThank you very much", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/69/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 557825032, "node_id": "MDU6SXNzdWU1NTc4MjUwMzI=", "number": 77, "title": "Ability to insert data that is transformed by a SQL function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-01-30T23:45:55Z", "updated_at": "2022-02-05T00:04:25Z", "closed_at": "2020-01-31T00:24:32Z", "author_association": "OWNER", "pull_request": null, "body": "I want to be able to run the equivalent of this SQL insert:\r\n```python\r\n# Convert to \"Well Known Text\" format\r\nwkt = shape(geojson['geometry']).wkt\r\n# Insert and commit the record\r\nconn.execute(\"INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))\", (\r\n \"Wales\", wkt\r\n))\r\nconn.commit()\r\n```\r\nFrom the Datasette SpatiaLite docs: https://datasette.readthedocs.io/en/stable/spatialite.html\r\n\r\nTo do this, I need a way of telling `sqlite-utils` that a specific column should be wrapped in `GeomFromText(?, 4326)`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/77/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 557842245, "node_id": "MDU6SXNzdWU1NTc4NDIyNDU=", "number": 79, "title": "Helper methods for working with SpatiaLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2020-01-31T00:39:19Z", "updated_at": "2022-02-05T00:04:25Z", "closed_at": "2022-02-04T05:55:11Z", "author_association": "OWNER", "pull_request": null, "body": "As demonstrated by this piece of documentation, using SpatiaLite with sqlite-utils requires a fair bit of boilerplate:\r\nhttps://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/docs/python-api.rst#L880-L909", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/79/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1123849278, "node_id": "I_kwDOCGYnMM5C_JQ-", "number": 395, "title": "\"apt-get: command not found\" error on macOS", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-04T06:03:42Z", "updated_at": "2022-02-04T06:10:58Z", "closed_at": "2022-02-04T06:10:58Z", "author_association": "OWNER", "pull_request": null, "body": "Yeah, `apt-get` isn't a thing on macOS so 4a2a3e2fd0d5534f446b3f1fee34cb165e4d86d2 (to test #79 against real SpatiaLite) broke.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/395/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1123851690, "node_id": "I_kwDOCGYnMM5C_J2q", "number": 396, "title": "mypy failure, sqlite_utils/utils.py:56", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-02-04T06:08:09Z", "updated_at": "2022-02-04T06:10:33Z", "closed_at": "2022-02-04T06:10:33Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/runs/5062725880?check_suite_focus=true\r\n\r\n> `sqlite_utils/utils.py:56: error: Incompatible return value type (got \"None\", expected \"str\")`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/396/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1072792507, "node_id": "I_kwDOCGYnMM4_8YO7", "number": 352, "title": "`sqlite-utils insert --extract colname`", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-07T00:55:44Z", "updated_at": "2022-02-03T22:59:36Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Is there a reason I've not added `--extract` as an option for `sqlite-utils insert` next? There's a `extracts=` option for the various `table.insert()` etc methods - last line in this code block:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/213a0ff177f23a35f3b235386366ff132eb879f1/sqlite_utils/db.py#L2483-L2495", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/352/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1094981339, "node_id": "I_kwDOCGYnMM5BRBbb", "number": 363, "title": "Better error message if `--convert` code fails to return a dict", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-06T05:26:28Z", "updated_at": "2022-02-03T22:52:30Z", "closed_at": "2022-02-03T22:51:30Z", "author_association": "OWNER", "pull_request": null, "body": "Here's the traceback if your `--convert` function doesn't return a dict right now:\r\n```\r\n% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all \r\n\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 949, in insert\r\n insert_upsert_implementation(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 834, in insert_upsert_implementation\r\n db[table].insert_all(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2602, in insert_all\r\n first_record = next(records)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 3044, in fix_square_braces\r\n for record in records:\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 831, in \r\n docs = (decode_base64_values(doc) for doc in docs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 86, in decode_base64_values\r\n to_fix = [\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 89, in \r\n if isinstance(doc[k], dict)\r\nTypeError: string indices must be integers\r\n```\r\nIt would be nicer if that returned a more useful error message.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/361#issuecomment-1006295276_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/363/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1118585417, "node_id": "I_kwDOCGYnMM5CrEJJ", "number": 393, "title": "Better documentation for insert-replace", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-30T15:40:23Z", "updated_at": "2022-02-03T22:13:24Z", "closed_at": "2022-02-03T22:13:24Z", "author_association": "OWNER", "pull_request": null, "body": "Currently: https://sqlite-utils.datasette.io/en/stable/python-api.html#insert-replacing-data\r\n\r\n> If you want to insert a record or replace an existing record with the same primary key, using the replace=True argument to .insert() or .insert_all():\r\n\r\nShould describe the exception you get first, then how to use replace to avoid it.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/393/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097091527, "node_id": "I_kwDOCGYnMM5BZEnH", "number": 369, "title": "Research how much of a difference analyze / sqlite_stat1 makes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2022-01-09T03:03:36Z", "updated_at": "2022-02-03T21:07:41Z", "closed_at": "2022-02-03T21:07:35Z", "author_association": "OWNER", "pull_request": null, "body": "> Is there a downside to having a `sqlite_stat1` table if it has wildly incorrect statistics in it?\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008163050_\r\n\r\nMore generally: how much of a difference does the `sqlite_stat1` table created by `ANALYZE` make to queries?\r\n\r\nI'm particularly interested in `group by` / `count *` queries since Datasette uses those for faceting.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/369/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1122446693, "node_id": "I_kwDOCGYnMM5C5y1l", "number": 394, "title": "Test against Python 3.11-dev", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-02-02T22:21:03Z", "updated_at": "2022-02-03T21:06:35Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Same as:\r\n- https://github.com/simonw/datasette/issues/1621", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/394/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1114640101, "node_id": "I_kwDOCGYnMM5CcA7l", "number": 392, "title": "`sqlite-utils bulk --batch-size` option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-26T05:17:11Z", "updated_at": "2022-01-26T18:17:59Z", "closed_at": "2022-01-26T18:17:59Z", "author_association": "OWNER", "pull_request": null, "body": "> Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/392/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1114638930, "node_id": "I_kwDOCGYnMM5CcApS", "number": 391, "title": "`sqlite-utils bulk` progress bar", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-01-26T05:14:49Z", "updated_at": "2022-01-26T05:17:20Z", "closed_at": "2022-01-26T05:16:51Z", "author_association": "OWNER", "pull_request": null, "body": "It can easily have a progress bar because it works by looping through an iterator: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018\r\n\r\nShould also support the `--silent` option if I add this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/391/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1114557284, "node_id": "I_kwDOCGYnMM5Cbstk", "number": 390, "title": "`sqlite-utils upsert` should require `--pk` more elegantly", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-26T02:20:31Z", "updated_at": "2022-01-26T03:20:25Z", "closed_at": "2022-01-26T03:19:43Z", "author_association": "OWNER", "pull_request": null, "body": "Currently throws an ugly traceback:\r\n```\r\n% echo '[\r\n {\"id\": 1, \"name\": \"Lila\"},\r\n {\"id\": 1, \"name\": \"Lila\"}\r\n ]' | sqlite-utils upsert data.db chickens - \r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in \r\n sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n return __callback(*args, **kwargs)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 1104, in upsert\r\n insert_upsert_implementation(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 906, in insert_upsert_implementation\r\n db[table].insert_all(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2615, in insert_all\r\n raise PrimaryKeyRequired(\"upsert() requires a pk\")\r\nsqlite_utils.db.PrimaryKeyRequired: upsert() requires a pk\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/390/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099897648, "node_id": "I_kwDOCGYnMM5Bjxsw", "number": 384, "title": "Add examples to every `--help`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-12T05:31:25Z", "updated_at": "2022-01-26T03:15:02Z", "closed_at": "2022-01-26T03:15:02Z", "author_association": "OWNER", "pull_request": null, "body": "Everything on https://sqlite-utils.datasette.io/en/stable/cli-reference.html would benefit from an example.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/384/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 471818939, "node_id": "MDU6SXNzdWU0NzE4MTg5Mzk=", "number": 48, "title": "Jupyter notebook demo of the library, launchable on Binder", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-07-23T17:05:05Z", "updated_at": "2022-01-26T02:08:46Z", "closed_at": "2022-01-26T02:08:39Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/48/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1114544727, "node_id": "I_kwDOCGYnMM5CbppX", "number": 389, "title": "Plausible analytics for documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-01-26T01:58:35Z", "updated_at": "2022-01-26T02:07:41Z", "closed_at": "2022-01-26T02:07:41Z", "author_association": "OWNER", "pull_request": null, "body": "```html\r\n\r\n```\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/389/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1111293050, "node_id": "I_kwDOCGYnMM5CPPx6", "number": 387, "title": "Python library docs should start with a self contained example", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-22T06:23:56Z", "updated_at": "2022-01-26T01:37:17Z", "closed_at": "2022-01-26T01:35:30Z", "author_association": "OWNER", "pull_request": null, "body": "You have to read a lot of stuff in a lot of different places to get started with the Python library. Add a getting started introduction to https://sqlite-utils.datasette.io/en/stable/python-api.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/387/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1107557831, "node_id": "I_kwDOCGYnMM5CA_3H", "number": 386, "title": "Better \"contributing\" documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-19T02:11:48Z", "updated_at": "2022-01-19T02:15:21Z", "closed_at": "2022-01-19T02:15:21Z", "author_association": "OWNER", "pull_request": null, "body": "This page jumps straight into running the tests: https://sqlite-utils.datasette.io/en/latest/contributing.html\r\n\r\nIt should add a little more about expected collaboration styles - opening an issue before filing a pull request - and probably link to https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/386/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1067771698, "node_id": "I_kwDOCGYnMM4_pOcy", "number": 348, "title": "Command for creating an empty database", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 6, "created_at": "2021-11-30T23:24:27Z", "updated_at": "2022-01-13T07:06:59Z", "closed_at": "2022-01-09T20:33:20Z", "author_association": "OWNER", "pull_request": null, "body": "I sometimes find the need to create an empty SQLite database file - for example if I want to enable WAL on it before using it with another script. I currently do that like this:\r\n\r\n sqlite3 my.db vacuum\r\n sqlite-utils enable-wal my.db\r\n\r\nIt would be nice if `sqlite-utils` had a convenience command for doing this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/348/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099584685, "node_id": "I_kwDOCGYnMM5BilSt", "number": 381, "title": "`sqlite-utils rows` options `--limit` and `--offset`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-01-11T20:23:12Z", "updated_at": "2022-01-11T23:33:37Z", "closed_at": "2022-01-11T23:19:36Z", "author_association": "OWNER", "pull_request": null, "body": "Because I often want to use it just to preview a few rows from the database. Piping through `| head -n 20` works for JSON and CSV (they stream) but not for `--table`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/381/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099585611, "node_id": "I_kwDOCGYnMM5BilhL", "number": 382, "title": "`--where` option for `sqlite-rows`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-01-11T20:24:23Z", "updated_at": "2022-01-11T23:33:14Z", "closed_at": "2022-01-11T23:32:47Z", "author_association": "OWNER", "pull_request": null, "body": "CLI equivalent of `table.rows_where()` - should accept parameters too. Work on this at the same time as #381.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/382/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1099586786, "node_id": "I_kwDOCGYnMM5Bilzi", "number": 383, "title": "Add documentation page with the output of `--help`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-11T20:25:58Z", "updated_at": "2022-01-11T22:55:05Z", "closed_at": "2022-01-11T21:44:05Z", "author_association": "OWNER", "pull_request": null, "body": "Can be maintained using `cog` from #373. Similar in purpose to the API reference page, but this is for the CLI.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/383/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1096558279, "node_id": "I_kwDOCGYnMM5BXCbH", "number": 365, "title": "create-index should run analyze after creating index", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 16, "created_at": "2022-01-07T18:21:25Z", "updated_at": "2022-01-11T02:43:34Z", "closed_at": "2022-01-11T01:36:48Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "sqlite's query planner depends upon analyze to make good use of indices. It would be nice if analyze was run as part of the create-index command.\r\n\r\nIf data is inserted later, things can get out date, but it would still probably be a net win. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/365/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1098574572, "node_id": "I_kwDOCGYnMM5Beurs", "number": 380, "title": "Release notes for 3.21", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 1, "created_at": "2022-01-11T02:12:30Z", "updated_at": "2022-01-11T02:34:26Z", "closed_at": "2022-01-11T02:34:26Z", "author_association": "OWNER", "pull_request": null, "body": "For these commits: https://github.com/simonw/sqlite-utils/compare/3.20...129141572f249ea290e2a075437e2ebaad215859", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/380/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097251014, "node_id": "I_kwDOCGYnMM5BZrjG", "number": 375, "title": "`sqlite-utils bulk` command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T17:12:38Z", "updated_at": "2022-01-11T02:12:58Z", "closed_at": "2022-01-11T02:10:55Z", "author_association": "OWNER", "pull_request": null, "body": "The `.executemany()` method is a very efficient way to execute the same SQL query against a huge list of parameters.\r\n\r\n`sqlite-utils insert` supports a bunch of ways of loading a list of dictionaries - from CSV, TSV, JSON, newline JSON and more thanks to:\r\n- #361\r\n\r\nWhat if you could load a list of dictionaries and provide a SQL query with `:named` parameters that correspond to keys in those dictionaries instead?\r\n\r\nThis would need to be a new command - I thought about adding a `--sql` option to `insert` but that doesn't make sense as that command already requires a table name.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/375/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097477582, "node_id": "PR_kwDOCGYnMM4wtl17", "number": 377, "title": "`sqlite-utils bulk` command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-10T05:34:24Z", "updated_at": "2022-01-11T02:10:57Z", "closed_at": "2022-01-11T02:10:54Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/377", "body": "Refs #375\r\n\r\nStill needs:\r\n\r\n- [x] Refactor `@insert_upsert_options` so that it doesn't duplicate `@import_options`\r\n- [x] Tests\r\n- [x] Documentation\r\n- [x] Try it against a really big file", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/377/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1098544628, "node_id": "I_kwDOCGYnMM5BenX0", "number": 379, "title": "CLI options for running ANALYZE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-11T01:09:16Z", "updated_at": "2022-01-11T01:38:01Z", "closed_at": "2022-01-11T01:36:48Z", "author_association": "OWNER", "pull_request": null, "body": "> The Python methods are all done now, next step is the CLI options. I'll do those in a separate issue.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009508865_\r\n\r\n- [x] `sqlite-utils analyze` command\r\n- [x] `sqlite-utils create-index --analyze` option (see #365)\r\n- [x] `sqlite-utils insert --analyze` option\r\n- [x] `sqlite-utils upsert --analyze` option\r\n\r\nIn #378 I also added `.delete_where(..., analyze=True)` but there isn't currently a `sqlite-utils delete-where` CLI command - deletions via CLI are expected to be handled using SQL queries.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/379/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1096563265, "node_id": "I_kwDOCGYnMM5BXDpB", "number": 366, "title": "Python library methods for calling ANALYZE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 10, "created_at": "2022-01-07T18:28:01Z", "updated_at": "2022-01-11T01:09:33Z", "closed_at": "2022-01-11T01:09:33Z", "author_association": "OWNER", "pull_request": null, "body": "> Relevant documentation: https://www.sqlite.org/lang_analyze.html\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007633376_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/366/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1098309897, "node_id": "I_kwDOCGYnMM5BduEJ", "number": 378, "title": "analyze=True parameter for some methods", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-10T19:54:52Z", "updated_at": "2022-01-11T01:08:11Z", "closed_at": "2022-01-11T01:08:09Z", "author_association": "OWNER", "pull_request": null, "body": "This would cause `ANALYZE` to be run against the relevant table at the end of executing the method.\r\n\r\n> Having browsed the API reference I think the methods that would benefit from an `analyze=True` parameter are:\r\n\r\n- [x] `table.create_index`\r\n- [x] `table.insert_all`\r\n- [x] `table.upsert_all`\r\n- [x] `table.delete_where`\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/366#issuecomment-1009288898_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/378/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097041471, "node_id": "PR_kwDOCGYnMM4wsVM6", "number": 367, "title": "Initial prototype of .analyze() methods", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 2, "created_at": "2022-01-08T21:35:12Z", "updated_at": "2022-01-10T19:31:08Z", "closed_at": "2022-01-10T19:31:08Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/367", "body": "Refs #366", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/367/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 1, "state_reason": null} {"id": 1097436959, "node_id": "I_kwDOCGYnMM5BaY8f", "number": 376, "title": "`--nl` mode should ignore blank lines", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 0, "created_at": "2022-01-10T04:10:54Z", "updated_at": "2022-01-10T19:27:41Z", "closed_at": "2022-01-10T04:12:46Z", "author_association": "OWNER", "pull_request": null, "body": "Spotted this while manually testing #364 - there's no reason `--nl` should crash if you feed it an empty line in between JSON objects.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/376/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097129710, "node_id": "I_kwDOCGYnMM5BZN7u", "number": 372, "title": "Idea: `suffix` and `stem` file columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 1, "created_at": "2022-01-09T07:48:53Z", "updated_at": "2022-01-10T19:27:34Z", "closed_at": "2022-01-09T20:17:00Z", "author_association": "OWNER", "pull_request": null, "body": "For https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files\r\n\r\nGiven a file called `dogs.jpg` stem would be `dogs` and ext would be `jpg`.\r\n\r\nNeed to decide what happens for `dogs.and.cats.jpg.gz`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/372/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097128334, "node_id": "I_kwDOCGYnMM5BZNmO", "number": 371, "title": "Support mutating row in `--convert` without returning it", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 6, "created_at": "2022-01-09T07:38:44Z", "updated_at": "2022-01-10T19:27:30Z", "closed_at": "2022-01-09T20:06:15Z", "author_association": "OWNER", "pull_request": null, "body": "Currently you have to do this:\r\n```\r\n$ sqlite-utils insert dogs.db dogs dogs.json --convert '\r\nrow[\"is_good\"] = 1\r\nreturn row'\r\n```\r\nWould be neat if this worked too:\r\n```\r\n$ sqlite-utils insert dogs.db dogs dogs.json \\\r\n --convert 'row[\"is_good\"] = 1'\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/371/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097135860, "node_id": "I_kwDOCGYnMM5BZPb0", "number": 374, "title": "`--fmt` should imply `-t`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 4, "created_at": "2022-01-09T08:23:07Z", "updated_at": "2022-01-10T19:27:26Z", "closed_at": "2022-01-09T18:07:59Z", "author_association": "OWNER", "pull_request": null, "body": "Not sure why I didn't implement this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/374/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097135732, "node_id": "I_kwDOCGYnMM5BZPZ0", "number": 373, "title": "List `--fmt` options in the docs ", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T08:22:11Z", "updated_at": "2022-01-10T19:27:24Z", "closed_at": "2022-01-09T17:49:00Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/stable/cli.html#table-formatted-output currently cheats and tells the user to run `--help` - can fix this using `cog`. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/373/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1097087280, "node_id": "I_kwDOCGYnMM5BZDkw", "number": 368, "title": "Offer `python -m sqlite_utils` as an alternative to `sqlite-utils`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 3, "created_at": "2022-01-09T02:29:30Z", "updated_at": "2022-01-10T19:27:20Z", "closed_at": "2022-01-09T02:40:50Z", "author_association": "OWNER", "pull_request": null, "body": "> Add this to `sqlite_utils/cli.py`:\r\n>\r\n> ```python\r\n> if __name__ == \"__main__\":\r\n> cli()\r\n> ```\r\n> Now the tool can be run using `python -m sqlite_utils.cli --help`\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008214998_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/368/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1095570074, "node_id": "I_kwDOCGYnMM5BTRKa", "number": 364, "title": "`--batch-size 1` doesn't seem to commit for every item", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 16, "created_at": "2022-01-06T18:18:50Z", "updated_at": "2022-01-10T19:27:17Z", "closed_at": "2022-01-10T05:36:19Z", "author_association": "OWNER", "pull_request": null, "body": "I'm trying this, but it doesn't seem to write anything to the database file until I hit `CTRL+C`:\r\n\r\n```\r\nheroku logs --app=simonwillisonblog --tail | grep 'measure#nginx.service' | \\\r\n sqlite-utils insert /tmp/herokutail.db log - --import re --convert \"$(cat <\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/362/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1094890366, "node_id": "PR_kwDOCGYnMM4wlm3B", "number": 361, "title": "--lines and --text and --convert and --import", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 15, "created_at": "2022-01-06T01:49:44Z", "updated_at": "2022-01-06T06:37:03Z", "closed_at": "2022-01-06T06:24:54Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/361", "body": "Refs #356\r\n\r\nStill TODO:\r\n\r\n- [x] Get `--lines` working, with tests\r\n- [x] Get `--text` working, with tests\r\n- [x] Get regular JSON import working with `--convert` with tests\r\n- [x] Get `--lines` working with `--convert` with tests\r\n- [x] Get `--text` working with `--convert` with tests\r\n- [x] Get `--csv` and `--tsv` import working with `--convert` with tests\r\n- [x] Get `--nl` working with `--convert` with tests\r\n- [x] Documentation for all of the above", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/361/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1077431957, "node_id": "I_kwDOCGYnMM5AOE6V", "number": 356, "title": "`sqlite-utils insert --convert` option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2021-12-11T07:24:48Z", "updated_at": "2022-01-06T06:30:13Z", "closed_at": "2022-01-06T06:28:53Z", "author_association": "OWNER", "pull_request": null, "body": "Idea come to me while re-reading this: https://simonwillison.net/2021/Aug/6/sqlite-utils-convert/\r\n\r\nThis is a bit of a hack:\r\n```\r\ncat /tmp/log.txt | \\\r\n jq --raw-input '{line: .}' --compact-output | \\\r\n sqlite-utils insert /tmp/logs.db log - --nl\r\n```\r\nWould be great if you could pipe lines to `insert` and transform them on the way in.\r\n\r\nA `--convert python-code` option, modeled after `sqlite-utils convert`, could do this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/356/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1090798237, "node_id": "I_kwDOCGYnMM5BBEKd", "number": 359, "title": "Use RETURNING if available to populate last_pk", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-12-29T23:43:23Z", "updated_at": "2021-12-29T23:43:23Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by this: https://news.ycombinator.com/item?id=29729283\r\n\r\n> Because SQLite is effectively serializing all the writes for us, we have zero locking in our code. We used to have to lock when inserting new items (to get the LastInsertRowId), but the newer version of SQLite supports the RETURNING keyword, so we don't even have to lock on inserts now.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/359/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 807437089, "node_id": "MDU6SXNzdWU4MDc0MzcwODk=", "number": 228, "title": "--no-headers option for CSV and TSV", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2021-02-12T17:56:51Z", "updated_at": "2021-12-26T07:01:31Z", "closed_at": "2021-02-14T22:25:17Z", "author_association": "OWNER", "pull_request": null, "body": "https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c has a fascinating CSV file that doesn't have a header row - it starts like this:\r\n\r\n```csv\r\nComputation and measurement of turbulent flow through idealized turbine blade passages,,\"Loizou, Panos A.\",https://isni.org/isni/0000000136122593,,University of Manchester,https://isni.org/isni/0000000121662407,1989,Thesis (Ph.D.),,Physical Sciences,,,https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.232781,\r\n\"Prolactin and growth hormone secretion in normal, hyperprolactinaemic and acromegalic man\",,\"Prescott, R. W. G.\",https://isni.org/isni/0000000134992122,,University of Newcastle upon Tyne,https://isni.org/isni/0000000104627212,1983,Thesis (Ph.D.),,Biological Sciences,,,https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.232784,\r\n```\r\n\r\nIt would be useful if `sqlite-utils insert ... --csv` had a mechanism for importing files like this one.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/228/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706001517, "node_id": "MDU6SXNzdWU3MDYwMDE1MTc=", "number": 163, "title": "Idea: conversions= could take Python functions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-22T00:37:12Z", "updated_at": "2021-12-20T00:56:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now you use `conversions=` like this:\r\n\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": \"upper(?)\"})\r\n```\r\nHow about if you could optionally provide a Python function (or a lambda) like this?\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": lambda s: s.upper()})\r\n```\r\nThis would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1079422215, "node_id": "I_kwDOCGYnMM5AVq0H", "number": 357, "title": "pytest-runner is not required", "user": {"value": 4067843, "label": "pgajdos"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-14T07:51:24Z", "updated_at": "2021-12-16T20:43:19Z", "closed_at": "2021-12-16T20:43:13Z", "author_association": "NONE", "pull_request": null, "body": "Deprecated pytest-runner is not necessary for running the testsuite.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/357/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1077102934, "node_id": "I_kwDOCGYnMM5AM0lW", "number": 353, "title": "Allow passing a file of code to \"sqlite-utils convert\"", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2021-12-10T18:06:14Z", "updated_at": "2021-12-11T01:38:29Z", "closed_at": "2021-12-11T01:09:39Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "sqlite-utils is so nice, but the ergonomics of the multiline code in kind of tough. It's really hard (maybe impossible) to make the newlines play well with Makefiles.\r\n\r\nit would be great to write your code fragment in a separate file and direct it into the sqlite-utils\r\n\r\neither like\r\n\r\n```sqlite-utils convert my.db my_table my_column < custom_code.py```\r\n\r\nor\r\n\r\n```sqlite-utils convert my.db my_table my_column --custom-code=custom_code.py```\r\n\r\nThanks, as ever, for these great tools!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/353/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1077243232, "node_id": "I_kwDOCGYnMM5ANW1g", "number": 354, "title": "Test failure in test_rebuild_fts", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2021-12-10T21:27:55Z", "updated_at": "2021-12-11T01:08:46Z", "closed_at": "2021-12-11T01:08:46Z", "author_association": "OWNER", "pull_request": null, "body": "Not sure why this has only just started failing, but I'm getting this: https://github.com/simonw/sqlite-utils/runs/4488687639\r\n\r\n```\r\nE sqlite3.DatabaseError: database disk image is malformed\r\n\r\nsqlite_utils/db.py:425: DatabaseError\r\n_______________________ test_rebuild_fts[searchable_fts] _______________________\r\n\r\nfresh_db = >\r\ntable_to_fix = 'searchable_fts'\r\n\r\n @pytest.mark.parametrize(\"table_to_fix\", [\"searchable\", \"searchable_fts\"])\r\n def test_rebuild_fts(fresh_db, table_to_fix):\r\n table = fresh_db[\"searchable\"]\r\n table.insert(search_records[0])\r\n table.enable_fts([\"text\", \"country\"])\r\n # Run a search\r\n rows = list(table.search(\"tanuki\"))\r\n assert len(rows) == 1\r\n assert {\r\n \"rowid\": 1,\r\n \"text\": \"tanuki are running tricksters\",\r\n \"country\": \"Japan\",\r\n \"not_searchable\": \"foo\",\r\n }.items() <= rows[0].items()\r\n # Delete from searchable_fts_data\r\n fresh_db[\"searchable_fts_data\"].delete_where()\r\n # This should have broken the index\r\n with pytest.raises(sqlite3.DatabaseError):\r\n list(table.search(\"tanuki\"))\r\n # Running rebuild_fts() should fix it\r\n> fresh_db[table_to_fix].rebuild_fts()\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/354/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1066603133, "node_id": "PR_kwDOCGYnMM4vKAzW", "number": 347, "title": "Test against pysqlite3 running SQLite 3.37", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2021-11-29T23:17:57Z", "updated_at": "2021-12-11T01:02:19Z", "closed_at": null, "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/347", "body": "Refs #346 and #344.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/347/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1077322009, "node_id": "I_kwDOCGYnMM5ANqEZ", "number": 355, "title": "Allow users to pass a full convert() function definition", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-12-10T23:59:58Z", "updated_at": "2021-12-11T00:51:15Z", "closed_at": "2021-12-11T00:49:31Z", "author_association": "OWNER", "pull_request": null, "body": "> I think the fix for this is to change the rules about what code is accepted in both the `-` mode and the literal code string mode: you can pass in a Python expression, OR a fragment that gets turned into a function, OR code that implements its own `def convert(value)` function. So this would work too:\r\n> ```sh\r\n> sqlite-utils convert my.db mytable col1 '\r\n> def convert(value):\r\n> return value.upper()\r\n> '\r\n> ```\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991381679_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/355/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1072780607, "node_id": "I_kwDOCGYnMM4_8VU_", "number": 351, "title": "Support `--import xml.etree.ElementTree` in `sqlite-utils convert`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-12-07T00:40:29Z", "updated_at": "2021-12-11T00:11:25Z", "closed_at": "2021-12-11T00:11:25Z", "author_association": "OWNER", "pull_request": null, "body": "It's not possible to use a module that requires a nested import, such as `xml.etree.ElementTree`, at the moment. I found and fixed this bug in `git-history`, I should replicate that fix (and accompanying documentation) here: https://github.com/simonw/git-history/issues/39", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/351/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1071531082, "node_id": "I_kwDOCGYnMM4_3kRK", "number": 349, "title": "A way of creating indexes on newly created tables", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-12-05T18:56:12Z", "updated_at": "2021-12-07T01:04:37Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm writing code for https://github.com/simonw/git-history/issues/33 that creates a table inside a loop:\r\n\r\n```python\r\nitem_pk = db[item_table].lookup(\r\n {\"_item_id\": item_id},\r\n item_to_insert,\r\n column_order=(\"_id\", \"_item_id\"),\r\n pk=\"_id\",\r\n)\r\n```\r\nI need to look things up by `_item_id` on this table, which means I need an index on that column (the table can get very big).\r\n\r\nBut there's no mechanism in SQLite utils to detect if the table was created for the first time and add an index to it. And I don't want to run `CREATE INDEX IF NOT EXISTS` every time through the loop.\r\n\r\nThis should work like the `foreign_keys=` mechanism.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/349/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1072435124, "node_id": "I_kwDOCGYnMM4_7A-0", "number": 350, "title": "Optional caching mechanism for table.lookup()", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-12-06T17:54:25Z", "updated_at": "2021-12-06T17:56:57Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Inspired by work on `git-history` where I used this pattern:\r\n```python\r\n column_name_to_id = {}\r\n\r\n def column_id(column):\r\n if column not in column_name_to_id:\r\n id = db[\"columns\"].lookup(\r\n {\"namespace\": namespace_id, \"name\": column},\r\n foreign_keys=((\"namespace\", \"namespaces\", \"id\"),),\r\n )\r\n column_name_to_id[column] = id\r\n return column_name_to_id[column]\r\n```\r\nIf you're going to be doing a large number of `table.lookup(...)` calls and you know that no other script will be modifying the database at the same time you can presumably get a big speedup using a Python in-memory cache - maybe even a LRU one to avoid memory bloat.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/350/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1066563554, "node_id": "I_kwDOCGYnMM4_knfi", "number": 346, "title": "Way to test SQLite 3.37 (and potentially other versions) in CI", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2021-11-29T22:21:06Z", "updated_at": "2021-11-29T23:12:49Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "> Need to figure out a good pattern for testing this in CI too - it will currently skip the new tests if it doesn't have SQLite 3.37 or higher.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982076924_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/346/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 1066501534, "node_id": "I_kwDOCGYnMM4_kYWe", "number": 345, "title": "`table.strict` introspection boolean for identifying STRICT mode tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-11-29T21:05:10Z", "updated_at": "2021-11-29T22:45:26Z", "closed_at": "2021-11-29T22:44:36Z", "author_association": "OWNER", "pull_request": null, "body": "> From the STRICT docs:\r\n\r\n>> The SQLite parser accepts a comma-separated list of table options after the final close parenthesis in a CREATE TABLE statement. As of this writing (2021-08-23) only two options are recognized:\r\n>> \r\n>> - STRICT\r\n>> - [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html)\r\n> \r\n> So I think I need to read the `CREATE TABLE` statement from the `sqlite_master` table, split on the last `)`, split those tokens on `,` and see if `create` is in there (case insensitive).\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982020757_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/345/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1039037439, "node_id": "PR_kwDOCGYnMM4t0uaI", "number": 333, "title": "Add functionality to read Parquet files.", "user": {"value": 2118708, "label": "Florents-Tselai"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-10-28T23:43:19Z", "updated_at": "2021-11-25T19:47:35Z", "closed_at": "2021-11-25T19:47:35Z", "author_association": "NONE", "pull_request": "simonw/sqlite-utils/pulls/333", "body": "I needed this for a project of mine, and I thought it'd be useful to have it in sqlite-utils (It's also mentioned in #248 ).\r\nThe current implementation works (data is read & data types are inferred correctly.\r\nI've added a single straightforward test case, but @simonw please let me know if there are any non-obvious flags/combinations I should test too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/333/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1058196641, "node_id": "I_kwDOCGYnMM4_Esyh", "number": 342, "title": "Extra options to `lookup()` which get passed to `insert()`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2021-11-19T06:53:03Z", "updated_at": "2021-11-19T07:26:54Z", "closed_at": "2021-11-19T07:26:54Z", "author_association": "OWNER", "pull_request": null, "body": "For https://github.com/simonw/git-history/issues/12 I found myself wanting to pass extra options to `lookup()` to set the column order, primary key etc.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/342/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1025726600, "node_id": "PR_kwDOCGYnMM4tKxHD", "number": 330, "title": "Test against Python 3.10", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-10-13T21:50:22Z", "updated_at": "2021-11-15T02:59:29Z", "closed_at": "2021-10-13T22:25:05Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/330", "body": null, "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/330/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1005891028, "node_id": "I_kwDOCGYnMM479K3U", "number": 329, "title": "Rethink approach to [ and ] in column names (currently throws error)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 12, "created_at": "2021-09-23T22:14:24Z", "updated_at": "2021-11-15T02:57:51Z", "closed_at": "2021-11-15T02:57:51Z", "author_association": "OWNER", "pull_request": null, "body": "> I think it's best to still keep `[` and `]` out of column names though. Transforming them into `(` and `)` seems reasonable - but should that happen here or in `sqlite-utils`? I think in `sqlite-utils`.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/datasette-app/issues/121#issuecomment-926200398_\r\n\r\nThis is a rethinking of the solution to:\r\n\r\n- https://github.com/simonw/sqlite-utils/issues/86", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/329/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1053136495, "node_id": "I_kwDOCGYnMM4-xZZv", "number": 341, "title": "`hash_id: Optional[Any]` should be `hash_id: Optional[str]`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-15T02:12:39Z", "updated_at": "2021-11-15T02:19:31Z", "closed_at": "2021-11-15T02:19:31Z", "author_association": "OWNER", "pull_request": null, "body": "In a few places:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L642\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L751\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L1049\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L1230\r\n\r\nBut it's correct here:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/sqlite_utils/db.py#L2470", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/341/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1053122092, "node_id": "I_kwDOCGYnMM4-xV4s", "number": 339, "title": "`table.lookup()` option to populate additional columns when creating a record", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2021-11-15T01:41:17Z", "updated_at": "2021-11-15T02:02:34Z", "closed_at": "2021-11-15T02:02:00Z", "author_association": "OWNER", "pull_request": null, "body": "> For the commits table I feel like I want a version of `table.lookup()` that can be passed additional columns to populate only if the record does not exist yet.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/git-history/issues/12#issuecomment-967455017_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/339/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1053087862, "node_id": "I_kwDOCGYnMM4-xNh2", "number": 338, "title": "dict, list, tuple should all map to TEXT", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-15T00:28:01Z", "updated_at": "2021-11-15T00:36:03Z", "closed_at": "2021-11-15T00:36:03Z", "author_association": "OWNER", "pull_request": null, "body": "> This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947\r\n>\r\n> So the `COLUMN_TYPE_MAPPING` should include those too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/322#issuecomment-968401459_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/338/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 979612115, "node_id": "MDExOlB1bGxSZXF1ZXN0NzE5OTk4MjI1", "number": 322, "title": "Add dict type to be mapped as TEXT in sqllite", "user": {"value": 2496189, "label": "minaeid90"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-08-25T20:54:26Z", "updated_at": "2021-11-15T00:27:40Z", "closed_at": "2021-11-15T00:27:40Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/322", "body": "the library deal with Postgres type jsonb as dictionary, add dict type as a TEXT for mapping to sqlite\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/322/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 988013247, "node_id": "MDExOlB1bGxSZXF1ZXN0NzI3MDEyOTk2", "number": 324, "title": "Use python-dateutil package instead of dateutils", "user": {"value": 191622, "label": "meatcar"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-09-03T18:31:19Z", "updated_at": "2021-11-14T23:25:40Z", "closed_at": "2021-11-14T23:25:40Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/324", "body": "While working on updating `sqlite-utils` for NixOS/Nixpkgs, I came a cross the following:\r\n\r\nIn 5ec6686153e29ae10d4921a1ad4c841f192f20e2, a new dependency was added on `dateutils` (https://pypi.org/project/dateutils/). \r\n\r\nI believe this is unintentional, and instead `python-dateutil` (https://pypi.org/project/python-dateutil/) was intended.\r\n\r\nMy reasoning is:\r\n- `python-dateutil` is imported here in [recipes.py](https://github.com/simonw/sqlite-utils/blob/5ec6686153e29ae10d4921a1ad4c841f192f20e2/sqlite_utils/recipes.py#L1) \r\n- The `mypy` `type-python-dateutil` dependency in [setup.py](https://github.com/simonw/sqlite-utils/blob/5ec6686153e29ae10d4921a1ad4c841f192f20e2/setup.py#L36)\r\n- `python-dateutil` is a dependency of `dateutils` as seen in the output in [docs/tutorial.ipynb](https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/docs/tutorial.ipynb#L43)\r\n\r\nSeems like the trailing \"s\" seems to be the source of confusion \ud83d\ude05\r\n\r\nI've swapped the dependencies out, hope this helps.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/324/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1026794056, "node_id": "I_kwDOCGYnMM49M6JI", "number": 331, "title": "Mypy error: found module but no type hints or library stubs", "user": {"value": 53032010, "label": "andreaslongo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2021-10-14T20:29:50Z", "updated_at": "2021-11-14T23:21:08Z", "closed_at": "2021-11-14T23:21:08Z", "author_association": "NONE", "pull_request": null, "body": "```\r\nPython 3.9.5\r\nmypy 0.910\r\nsqlite-utils 3.17.1\r\n```\r\n\r\nWhile using sqlite-utils as a library, when I use mypy for static type checking, it throws an error:\r\n\r\n```\r\nmypy .\r\nsrc/etl.py:5: error: Skipping analyzing \"sqlite_utils\": found module but no type hints or library stubs\r\n import sqlite_utils\r\n ^\r\nsrc/etl.py:5: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports\r\ntest/test_etl.py:4: error: Skipping analyzing \"sqlite_utils\": found module but no type hints or library stubs\r\n import sqlite_utils\r\n ^\r\nFound 2 errors in 2 files (checked 7 source files)\r\n```\r\n\r\n\r\nWhen I add a `py.typed` file to the sqlite-utils package to mark it as PEP 561 compatible, the error goes away.\r\n\r\n```\r\nal@nbal ..b/python3.9/site-packages/sqlite_utils (git)-[main] % la\r\ntotal 200\r\ndrwx------ 3 al al 4096 Oct 14 22:00 .\r\ndrwx------ 117 al al 4096 Oct 12 21:12 ..\r\n-rw------- 1 al al 64409 Oct 12 21:11 cli.py\r\n-rw------- 1 al al 109092 Oct 12 21:11 db.py\r\n-rw------- 1 al al 0 Oct 14 22:00 py.typed\r\n-rw------- 1 al al 684 Oct 12 21:11 recipes.py\r\n-rw------- 1 al al 7988 Oct 12 21:11 utils.py\r\n-rw------- 1 al al 113 Oct 12 21:11 __init__.py\r\n```\r\n\r\nI would like to suggest adding a `py.typed` file to the repository.\r\n\r\nSee also the mypy docs on creating PEP 561 compatible packages:\r\nhttps://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/331/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1028056713, "node_id": "I_kwDOCGYnMM49RuaJ", "number": 332, "title": "`sqlite-utils memory --flatten` option to flatten nested JSON", "user": {"value": 22523840, "label": "rdtq"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-10-16T14:04:42Z", "updated_at": "2021-11-14T23:05:05Z", "closed_at": "2021-11-14T23:05:05Z", "author_association": "NONE", "pull_request": null, "body": "currently --flatten option works only for `insert` command, it would be cool if it worked for `memory` as well to query nested json", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/332/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1042569687, "node_id": "I_kwDOCGYnMM4-JFnX", "number": 335, "title": "sqlite-utils index-foreign-keys fails due to pre-existing index", "user": {"value": 596279, "label": "zaneselvans"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 11, "created_at": "2021-11-02T16:22:11Z", "updated_at": "2021-11-14T22:55:56Z", "closed_at": "2021-11-14T22:55:56Z", "author_association": "NONE", "pull_request": null, "body": "While running the command:\r\n```sh\r\nsqlite-utils index-foreign-keys $SQLITE_DIR/pudl.sqlite\r\n```\r\n\r\nI got the following error:\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/home/zane/miniconda3/envs/pudl-dev/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/cli.py\", line 454, in index_foreign_keys\r\n db.index_foreign_keys()\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py\", line 902, in index_foreign_keys\r\n table.create_index([fk.column])\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1563, in create_index\r\n self.db.execute(sql)\r\n File \"/home/zane/miniconda3/envs/pudl-dev/lib/python3.9/site-packages/sqlite_utils/db.py\", line 421, in execute\r\n return self.conn.execute(sql)\r\nsqlite3.OperationalError: index idx_generators_eia860_report_date already exists\r\n```\r\n\r\nThis DB was created with the foreign key constraint `PRAGMA` enabled and a bunch of column-level `CHECK` constraints. Is this an expected behavior? Should one not try to index foreign keys if FK constraints are already being enforced within the DB?\r\n\r\nI'm also noticing that the size of the DB after FK indexes have been added went from 483MB to 835MB, which seems like a much bigger jump than when I've done this previously.\r\n\r\nSoftware versions...\r\n* sqlite-utils 3.17.1\r\n* sqlite 3.36.0\r\n* SQLAlchemy 1.4.26 (used to create the DB)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/335/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1041778507, "node_id": "I_kwDOCGYnMM4-GEdL", "number": 334, "title": "Filter by datetime objects using rows_where()", "user": {"value": 11642379, "label": "viseshrp"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-11-02T00:44:08Z", "updated_at": "2021-11-13T19:23:21Z", "closed_at": "2021-11-13T19:23:21Z", "author_association": "NONE", "pull_request": null, "body": "Firstly, thanks for this nice utility. \r\nIt would be nice to have an example in the docs on how to filter by date range using `rows_where()`. \r\nThis doesn't seem to work:\r\n```\r\ntable.rows_where('datetime(created) between datetime(\"2021-10-31T17:29:59.277428-04:00\") AND datetime(\"2021-11-01T03:44:04.544651+00:00\")')\r\n```\r\n\r\n\r\nI could probably just use `db.query()`, which works for the above, but it would be nice if I could pass in `datetime` objects in `rows_where()`.\r\nThanks.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/334/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1046271107, "node_id": "PR_kwDOCGYnMM4uK5z2", "number": 337, "title": "Default values for `--attach` and `--param` options", "user": {"value": 771193, "label": "urbas"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-11-05T21:57:53Z", "updated_at": "2021-11-05T22:33:03Z", "closed_at": "2021-11-05T22:33:02Z", "author_association": "NONE", "pull_request": "simonw/sqlite-utils/pulls/337", "body": "It seems that `click` 8.x uses `None` as the default value for `multiple=True` options.\r\n\r\nThis change makes the code forward-compatible with `click` 8.x.\r\n\r\nSee this build failure for more info: https://hydra.nixos.org/build/156926608", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/337/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 707478649, "node_id": "MDU6SXNzdWU3MDc0Nzg2NDk=", "number": 173, "title": "Progress bar for sqlite-utils insert", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-09-23T15:43:56Z", "updated_at": "2021-11-01T08:42:24Z", "closed_at": "2020-10-27T18:16:04Z", "author_association": "OWNER", "pull_request": null, "body": "It would be nice if `sqlite-utils insert` had a progress bar, for when it's churning through huge CSV files.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/173/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 761915790, "node_id": "MDU6SXNzdWU3NjE5MTU3OTA=", "number": 206, "title": "sqlite-utils should suggest --csv if JSON parsing fails", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-12-11T05:17:56Z", "updated_at": "2021-10-30T15:52:17Z", "closed_at": "2021-01-03T18:42:22Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n~ % gsutil cat gs://ossf-criticality-score/python_top_200.csv | sqlite-utils insert /tmp/crit.db crit -\r\n...\r\n File \"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py\", line 337, in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n File \"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py\", line 355, in raw_decode\r\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```\r\nA nicer error message here would be one that says the JSON is invalid but suggests that maybe you could try `--csv`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/206/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 836829560, "node_id": "MDU6SXNzdWU4MzY4Mjk1NjA=", "number": 248, "title": "support for Apache Arrow / parquet files I/O", "user": {"value": 649467, "label": "mhalle"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-03-20T14:59:30Z", "updated_at": "2021-10-28T23:46:48Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I just started looking at Apache Arrow using pyarrow for import and export of tabular datasets, and it looks quite compelling. It might be worth looking at for sqlite-utils and/or datasette.\r\n\r\nAs a test, I took a random jsonl data dump of a dataset I have with floats, strings, and ints and converted it to arrow's parquet format using the naive `pyarrow.parquet.write_file()` command, which has automatic type inferrence. It compressed down to 7% of the original size. Conversion of a 26MB JSON file and serializing it to parquet was eyeblink instantaneous. Parquet files are portable and can be directly imported into pandas and other analytics software. \r\n\r\nThe only hangup is the automatic type inference of the naive reader. It's great for general laziness and for parsing JSON columns (it correctly interpreted a table of mine with a JSON array). However, I did get an exception for a string column where most entries looked integer-like but had a couple values that weren't -- the reader tried to coerce all of them for some reason, even though the JSON type is string. Since the writer optionally takes a schema, it shouldn't be too hard to grab the sqlite header types. With some additional hinting, you might get datetime columns and JSON, which are native Arrow types. \r\n\r\nSomewhat tangentially, someone even wrote an sqlite vfs extension for Parquet: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/248/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 817989436, "node_id": "MDU6SXNzdWU4MTc5ODk0MzY=", "number": 242, "title": "Async support", "user": {"value": 25778, "label": "eyeseast"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 13, "created_at": "2021-02-27T18:29:38Z", "updated_at": "2021-10-28T14:37:56Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Following our conversation last week, want to note this here before I forget.\r\n\r\nI've had a couple situations where I'd like to do a bunch of updates in an async event loop, but I run into SQLite's issues with concurrent writes. This feels like something sqlite-utils could help with.\r\n\r\nPeeWee ORM has a [SQLite write queue](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqliteq) that might be a good model. It's using threads or gevent, but I _think_ that approach would translate well enough to asyncio. \r\n\r\nHappy to help with this, too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/242/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 964400482, "node_id": "MDU6SXNzdWU5NjQ0MDA0ODI=", "number": 310, "title": "`sqlite-utils insert --flatten` option to flatten nested JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2021-08-09T21:23:08Z", "updated_at": "2021-10-16T13:54:56Z", "closed_at": "2021-08-09T21:44:06Z", "author_association": "OWNER", "pull_request": null, "body": "I had to do this with a `jq` recipe today: https://til.simonwillison.net/cloudrun/tailing-cloud-run-request-logs\r\n\r\n```\r\ncat log.json | jq -c '[leaf_paths as $path | {\r\n \"key\": $path | join(\"_\"), \"value\": getpath($path)\r\n}] | from_entries' \\\r\n| sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1\r\n```\r\nThat was to turn something like this:\r\n```json\r\n {\r\n \"httpRequest\": {\r\n \"latency\": \"0.112114537s\",\r\n \"requestMethod\": \"GET\",\r\n \"requestSize\": \"534\",\r\n \"status\": 200,\r\n },\r\n \"insertId\": \"6111722f000b5b4c4d4071e2\",\r\n \"labels\": {\r\n \"service\": \"datasette-io\"\r\n }\r\n }\r\n```\r\nInto this instead:\r\n```json\r\n{\r\n \"httpRequest_latency\": \"0.112114537s\",\r\n \"httpRequest_requestMethod\": \"GET\",\r\n \"httpRequest_requestSize\": \"534\",\r\n \"httpRequest_status\": 200,\r\n \"insertId\": \"6111722f000b5b4c4d4071e2\",\r\n \"labels_service\": \"datasette-io\"\r\n}\r\n```\r\nI have to do this often enough that I think it should be an option, `--flatten` - so I can do this instead:\r\n```\r\ncat log.json | sqlite-utils insert /tmp/logs.db logs - --flatten\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/310/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}