{"id": 1886771493, "node_id": "I_kwDOCGYnMM5wddkl", "number": 592, "title": "`table.transform()` should preserve `rowid` values", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-09-08T00:42:38Z", "updated_at": "2023-09-10T17:46:41Z", "closed_at": "2023-09-09T00:45:32Z", "author_association": "OWNER", "pull_request": null, "body": "I just spotted a bug when using https://datasette.io/plugins/datasette-configure-fts and https://datasette.io/plugins/datasette-edit-schema at the same time.\r\n\r\nSteps to reproduce:\r\n\r\n- Configure FTS for a table, then run a test search\r\n- Edit the schema for that table and change the order of columns\r\n- Run the test search again\r\n\r\nI got the wrong search results, which I think is because the `_fts` table pointed to the first table by `rowid` but those `rowid` values were entirely rewritten as a consequence of running `table.transform()` on the table.\r\n\r\nReconfiguring FTS on the table fixed the problem.\r\n\r\nI think `table.transform()` should be able to preserve `rowid` values.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/592/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1786258502, "node_id": "I_kwDOCGYnMM5qeCRG", "number": 565, "title": "Table renaming: db.rename_table() and sqlite-utils rename-table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-07-03T14:07:42Z", "updated_at": "2023-07-22T22:12:40Z", "closed_at": "2023-07-22T22:12:40Z", "author_association": "OWNER", "pull_request": null, "body": "> I find myself wanting two new features in `sqlite-utils`:\r\n> - The ability to have the new transformed table set to a specific name, while keeping the old table around\r\n> - The ability to rename a table (`sqlite-utils` doesn't have a table rename function at all right now)\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618375042_\r\n            ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/565/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1718607907, "node_id": "I_kwDOCGYnMM5mb-Aj", "number": 551, "title": "Make as many examples in the CLI docs as possible copy-and-pastable", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-05-21T19:04:10Z", "updated_at": "2023-05-21T21:04:04Z", "closed_at": "2023-05-21T20:57:24Z", "author_association": "OWNER", "pull_request": null, "body": "e.g. in this section:\r\n\r\nhttps://sqlite-utils.datasette.io/en/stable/cli.html#running-queries-directly-against-csv-or-json\r\n\r\n<img width=\"771\" alt=\"image\" src=\"https://github.com/simonw/sqlite-utils/assets/9599/3935111d-441e-4edb-9100-ce210117dbe5\">\r\n\r\nThe little copy button will also copy the `$ ` which breaks the examples when copied.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/551/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1718517882, "node_id": "I_kwDOCGYnMM5mboB6", "number": 545, "title": "Try out Trogon for a tui interface", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-05-21T14:08:25Z", "updated_at": "2023-05-21T19:33:13Z", "closed_at": "2023-05-21T18:41:58Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/Textualize/trogon", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/545/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1348294436, "node_id": "PR_kwDOCGYnMM49qP2V", "number": 468, "title": "db[table].create(..., transform=True) and create-table --transform", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8355157, "label": "3.29"}, "comments": 6, "created_at": "2022-08-23T17:27:58Z", "updated_at": "2022-08-27T23:17:55Z", "closed_at": "2022-08-27T23:17:55Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/468", "body": "Work in progress. Still needs documentation and tests (and to cover more cases of things that might have changed).\r\n\r\nRefs:\r\n- #467\r\n\r\n<!-- readthedocs-preview sqlite-utils start -->\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--468.org.readthedocs.build/en/468/\r\n\r\n<!-- readthedocs-preview sqlite-utils end -->", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/468/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 1352932038, "node_id": "I_kwDOCGYnMM5QpBrG", "number": 470, "title": "Upgrade `--load-extension` to accept entrypoints like Datasette", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 8355157, "label": "3.29"}, "comments": 6, "created_at": "2022-08-27T03:53:20Z", "updated_at": "2022-08-27T05:55:49Z", "closed_at": "2022-08-27T05:55:48Z", "author_association": "OWNER", "pull_request": null, "body": "Imitate:\r\n- https://github.com/simonw/datasette/pull/1789\r\n```\r\n# would load default entrypoint like before\r\ndatasette data.db --load-extension ext\r\n\r\n# loads the extensions with the \"sqlite3_foo_init\" entrpoint\r\ndatasette data.db --load-extension ext:sqlite3_foo_init\r\n\r\n# loads the extensions with the \"sqlite3_bar_init\" entrpoint\r\ndatasette data.db --load-extension ext:sqlite3_bar_init\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/470/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1145882578, "node_id": "I_kwDOCGYnMM5ETMfS", "number": 408, "title": "`deterministic=True` fails on versions of SQLite prior to 3.8.3", "user": {"value": 24938923, "label": "learning4life"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2022-02-21T14:36:43Z", "updated_at": "2022-03-13T16:54:09Z", "closed_at": "2022-03-02T00:38:11Z", "author_association": "NONE", "pull_request": null, "body": "Hi, love your work.\r\n\r\nI am unable to lookup indexes in a database using sqlite-utils:\r\n\r\n`\r\nsqlite-utils indexes city_spec.db --table`\r\n\r\nor\r\n\r\n`sqlite-utils indexes city_spec.db MyTable\r\n`\r\n\r\n**Software**\r\nsqlite-utils, version 3.24\r\nsqlite3 --version: 3.36.0 \r\n\r\n**Output:**\r\n\r\nTraceback (most recent call last):\r\n  File \"/opt/app-root/bin/sqlite-utils\", line 8, in <module>\r\n    sys.exit(cli())\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1128, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1053, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1659, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 1395, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 754, in invoke\r\n    return __callback(*args, **kwargs)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/click/decorators.py\", line 26, in new_func\r\n    return f(get_current_context(), *args, **kwargs)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py\", line 2123, in indexes\r\n    ctx.invoke(\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/click/core.py\", line 754, in invoke\r\n    return __callback(*args, **kwargs)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py\", line 1624, in query\r\n    db.register_fts4_bm25()\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 403, in register_fts4_bm25\r\n    self.register_function(rank_bm25, deterministic=True)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 399, in register_function\r\n    register(fn)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 392, in register\r\n    self.conn.create_function(name, arity, fn, **kwargs)\r\nsqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/408/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1067771698, "node_id": "I_kwDOCGYnMM4_pOcy", "number": 348, "title": "Command for creating an empty database", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 6, "created_at": "2021-11-30T23:24:27Z", "updated_at": "2022-01-13T07:06:59Z", "closed_at": "2022-01-09T20:33:20Z", "author_association": "OWNER", "pull_request": null, "body": "I sometimes find the need to create an empty SQLite database file - for example if I want to enable WAL on it before using it with another script. I currently do that like this:\r\n\r\n    sqlite3 my.db vacuum\r\n    sqlite-utils enable-wal my.db\r\n\r\nIt would be nice if `sqlite-utils` had a convenience command for doing this.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/348/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1097128334, "node_id": "I_kwDOCGYnMM5BZNmO", "number": 371, "title": "Support mutating row in `--convert` without returning it", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 6, "created_at": "2022-01-09T07:38:44Z", "updated_at": "2022-01-10T19:27:30Z", "closed_at": "2022-01-09T20:06:15Z", "author_association": "OWNER", "pull_request": null, "body": "Currently you have to do this:\r\n```\r\n$ sqlite-utils insert dogs.db dogs dogs.json --convert '\r\nrow[\"is_good\"] = 1\r\nreturn row'\r\n```\r\nWould be neat if this worked too:\r\n```\r\n$ sqlite-utils insert dogs.db dogs dogs.json \\\r\n  --convert 'row[\"is_good\"] = 1'\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/371/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 707478649, "node_id": "MDU6SXNzdWU3MDc0Nzg2NDk=", "number": 173, "title": "Progress bar for sqlite-utils insert", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-09-23T15:43:56Z", "updated_at": "2021-11-01T08:42:24Z", "closed_at": "2020-10-27T18:16:04Z", "author_association": "OWNER", "pull_request": null, "body": "It would be nice if `sqlite-utils insert` had a progress bar, for when it's churning through huge CSV files.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/173/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 465815372, "node_id": "MDU6SXNzdWU0NjU4MTUzNzI=", "number": 37, "title": "Experiment with type hints", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2019-07-09T14:30:34Z", "updated_at": "2021-08-18T21:48:57Z", "closed_at": "2021-08-18T21:48:57Z", "author_association": "OWNER", "pull_request": null, "body": "Since it's designed to be used in Jupyter or for rapid prototyping in an IDE (and it's still pretty small) `sqlite-utils` feels like a great candidate for me to finally try out Python type hints.\r\n\r\nhttps://veekaybee.github.io/2019/07/08/python-type-hints/ is good.\r\n\r\nIt suggests the mypy docs for getting started: https://mypy.readthedocs.io/en/latest/existing_code.html plus this tutorial: https://pymbook.readthedocs.io/en/latest/typehinting.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/37/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 944326512, "node_id": "MDU6SXNzdWU5NDQzMjY1MTI=", "number": 296, "title": "`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option", "user": {"value": 32427188, "label": "deafmute1"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-07-14T11:26:47Z", "updated_at": "2021-08-18T20:13:12Z", "closed_at": "2021-08-18T20:10:48Z", "author_association": "NONE", "pull_request": null, "body": "Hi,\r\nRecently got this error:\r\n```\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"/home/ethan/git/music-metadata-indexer/src/mmindexer/__init__.py\", line 38, in <module>\r\n    start(\"/home/ethan/git/music-metadata-indexer/sample\", \"/home/ethan/git/music-metadata-indexer/test.db\")\r\n  File \"/home/ethan/git/music-metadata-indexer/src/mmindexer/__init__.py\", line 23, in start\r\n    scanner.build_database()\r\n  File \"/home/ethan/git/music-metadata-indexer/src/mmindexer/scan.py\", line 79, in build_database\r\n    _import_song(self.db, Path(dirpath).joinpath(f), self.logger) \r\n  File \"/home/ethan/git/music-metadata-indexer/src/mmindexer/scan.py\", line 23, in _import_song\r\n    db.add_song(filepath)\r\n  File \"/home/ethan/git/music-metadata-indexer/src/mmindexer/index.py\", line 166, in add_song\r\n    for match in self.search(\"albums\", album): \r\n  File \"/home/ethan/git/music-metadata-indexer/env/lib/python3.9/site-packages/sqlite_utils/db.py\", line 1625, in search\r\n    cursor = self.db.execute(\r\n  File \"/home/ethan/git/music-metadata-indexer/env/lib/python3.9/site-packages/sqlite_utils/db.py\", line 243, in execute\r\n    return self.conn.execute(sql, parameters)\r\nsqlite3.OperationalError: fts5: syntax error near \".\" \r\n```\r\nSo, the error seems to suggest there was a \".\" character somewhere in the SQL command that was causing the error. I did a little digging and found this in the docs: https://www.sqlite.org/fts5.html#fts5_strings. \".\" is one of the many prohibited characters.\r\n\r\nMy solution was to just strip these out of the query using this line\r\n`query = query.translate({e: None for e in itertools.chain(range(0,26), range(27, 48), range(58,65), range(91,95), [96], range(123,128))})`\r\n\r\nPerhaps this could be included into the `table.search()` function?\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/296/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 963897111, "node_id": "MDU6SXNzdWU5NjM4OTcxMTE=", "number": 309, "title": "sqlite-utils insert errors should show SQL and parameters, if possible", "user": {"value": 16622642, "label": "scaleoutsean"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-08-09T11:24:14Z", "updated_at": "2021-08-09T23:40:29Z", "closed_at": "2021-08-09T22:25:58Z", "author_association": "NONE", "pull_request": null, "body": "I've tried several approaches, but this is the current one:\r\n\r\n```sh\r\necho $json-line | sqlite-utils insert json.db jsontable --truncate --alter --detect-types -\r\n```\r\nIn all cases, I get this error:\r\n\r\n```sh\r\nOverflowError: Python int too large to convert to SQLite INTEGER\r\nTraceback (most recent call last):\r\n  File \"/home/sean/.local/bin/sqlite-utils\", line 8, in <module>\r\n    sys.exit(cli())\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 764, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 717, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 1137, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 956, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/usr/lib/python3/dist-packages/click/core.py\", line 555, in invoke\r\n    return callback(*args, **kwargs)\r\n  File \"/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 841, in insert\r\n    insert_upsert_implementation(\r\n  File \"/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 780, in insert_upsert_implementation\r\n    db[table].insert_all(\r\n  File \"/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py\", line 2145, in insert_all\r\n    self.insert_chunk(\r\n  File \"/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py\", line 1957, in insert_chunk\r\n    result = self.db.execute(query, params)\r\n  File \"/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py\", line 257, in execute\r\n    return self.conn.execute(sql, parameters)\r\n```\r\n\r\nI googled the error and checked SO answers and advice, all good. I changed my JSON file to not use integers so I no longer get this error. Of course, that makes using the database a bit harder, so I also tried to solve the problem by modifying DB structure (while using integers in JSON).\r\n\r\nIf change all `INTEGER` Data Types to something else (`STRING`, `TEXT`) and try to import again using `--truncate`, I still get this error. I suppose I should tell sqlite-utils which columns should use non-INTEGER Data Type rather than rely on it to check my SQL table configuration.\r\n\r\nIf that is the case, can this error be a bit more specific for easier troubleshooting - maybe tell us which which record caused the problem when that error is thrown? \r\n\r\nMy table has 60+ columns, many of which use 64-bit integers (not all records are large or known in advance), so while I can modify JSON to use strings instead of integers, it decreases usability and finding out which records have values for which SQLite integers aren't sufficient requires some work (I'm thinking about parsing all integers with `jq` and sorting output by length to identify those columns, but I'd prefer if sqlite-utils could tell me that). \r\n\r\nMy environment:\r\n\r\n- Python 3.8.10\r\n  - sqlite-utils            3.14 \r\n  - pandas                  1.3.1               \r\n  - numpy                   1.21.1              \r\n  - sqlite-fts4             1.0.1\r\n- sqlite 3.31.1-4ubuntu0.2\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/309/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 926777310, "node_id": "MDU6SXNzdWU5MjY3NzczMTA=", "number": 290, "title": "`db.query()` method (renamed `db.execute_returning_dicts()`)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-06-22T03:03:54Z", "updated_at": "2021-06-24T23:17:38Z", "closed_at": "2021-06-24T22:54:43Z", "author_association": "OWNER", "pull_request": null, "body": "Most of this library deals with lists of Python dictionaries - `.insert_all()`, `.rows`, `.rows_where()`, `.search()`.\r\n\r\nThe `db.execute()` method is the only thing that returns a `sqlite3` cursor.\r\n\r\nThere is a clumsily named `db.execute_returning_dicts(sql)` method but it's not currently mentioned in the documentation.\r\n\r\nIt needs a better name, and needs to be properly documented.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/290/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 906356331, "node_id": "MDU6SXNzdWU5MDYzNTYzMzE=", "number": 263, "title": "`sqlite-utils indexes` command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2021-05-29T04:52:34Z", "updated_at": "2021-06-03T04:34:38Z", "closed_at": "2021-06-03T04:34:38Z", "author_association": "OWNER", "pull_request": null, "body": "While working on #260 I realized there's no command to show indexes in a database, even though there is one for showing tables and one for triggers.\r\n\r\nI should implement #261 first.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/263/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 763320133, "node_id": "MDExOlB1bGxSZXF1ZXN0NTM3NzkxNjc1", "number": 208, "title": "sqlite-utils analyze-tables command and table.analyze_column() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-12-12T05:27:49Z", "updated_at": "2020-12-13T07:20:16Z", "closed_at": "2020-12-13T07:20:12Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/208", "body": "Refs #207\r\n\r\n- [x] Improve design of CLI output\r\n- [x] Truncate long values in least/most common\r\n- [x] Add a `-c` column selection option\r\n- [x] Tests\r\n- [x] Documentation", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/208/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 737476423, "node_id": "MDU6SXNzdWU3Mzc0NzY0MjM=", "number": 198, "title": "Support order by relevance against FTS4", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-11-06T05:36:31Z", "updated_at": "2020-11-06T18:30:44Z", "closed_at": "2020-11-06T18:30:44Z", "author_association": "OWNER", "pull_request": null, "body": "For #192 and #197 I've decided I want to be able to order by relevance in FTS4 as well as FTS5.\r\n\r\nThis means I need to port over my work on bm25() from https://github.com/simonw/sqlite-fts4 (since I don't want to add a full dependency).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/198/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 613755043, "node_id": "MDU6SXNzdWU2MTM3NTUwNDM=", "number": 110, "title": "Support decimal.Decimal type", "user": {"value": 134771, "label": "dvhthomas"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-05-07T03:57:19Z", "updated_at": "2020-05-11T01:58:20Z", "closed_at": "2020-05-11T01:50:11Z", "author_association": "NONE", "pull_request": null, "body": "Decimal types in Postgres cause a failure in db.py data type selection\r\n---\r\nI have a Django app using a MoneyField, which uses a `numeric(14,0)` data type in Postgres (https://www.postgresql.org/docs/9.3/datatype-numeric.html). When attempting to export that table I get the following error:\r\n\r\n```bash\r\n$ db-to-sqlite --table isaweb_proposal \"postgres://connection\" test.db\r\n....\r\n    column_type=COLUMN_TYPE_MAPPING[column_type],\r\nKeyError: <class 'decimal.Decimal'>\r\n```\r\n\r\nLooking at `sql_utils.db.py` at 292-ish it's clear that there is no matching type for what I assume SQLAlchemy interprets as Python decimal.Decimal.\r\n\r\nFrom the [SQLite docs](https://www.sqlite.org/datatype3.html#affinity_name_examples) it looks like DECIMAL in other DBs are considered numeric.\r\n\r\nI'm not quite sure if it's as simple as adding a data type to that list or if there are repercussions beyond it.\r\n\r\nThanks for a great tool!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/110/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 545407916, "node_id": "MDU6SXNzdWU1NDU0MDc5MTY=", "number": 73, "title": "upsert_all() throws issue when upserting to empty table", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-01-05T11:58:57Z", "updated_at": "2020-01-31T14:21:09Z", "closed_at": "2020-01-05T17:20:18Z", "author_association": "NONE", "pull_request": null, "body": "If I try to add a list of `dict`s to an empty table using `upsert_all`, I get an error:\r\n\r\n```python\r\nimport sqlite3\r\nfrom sqlite_utils import Database\r\nimport pandas as pd\r\n\r\nconx = sqlite3.connect(':memory')\r\ncx = conx.cursor()\r\ncx.executescript('CREATE TABLE \"test\" (\"Col1\" TEXT);')\r\n\r\nq=\"SELECT * FROM test;\"\r\npd.read_sql(q, conx) #shows empty table\r\n\r\ndb = Database(conx)\r\ndb['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\n\r\n---------------------------------------------------------------------------\r\nTypeError                                 Traceback (most recent call last)\r\n<ipython-input-74-8c26d93d7587> in <module>\r\n      1 db = Database(conx)\r\n----> 2 db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)\r\n   1157             alter=alter,\r\n   1158             extracts=extracts,\r\n-> 1159             upsert=True,\r\n   1160         )\r\n   1161 \r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)\r\n   1040                     sql = \"INSERT OR IGNORE INTO [{table}]({pks}) VALUES({pk_placeholders});\".format(\r\n   1041                         table=self.name,\r\n-> 1042                         pks=\", \".join([\"[{}]\".format(p) for p in pks]),\r\n   1043                         pk_placeholders=\", \".join([\"?\" for p in pks]),\r\n   1044                     )\r\n\r\nTypeError: 'NoneType' object is not iterable\r\n\r\n```\r\n\r\nA hacky workaround in use is:\r\n\r\n```python\r\ntry:\r\n  db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])\r\nexcept:\r\n  db['test'].insert_all([{'Col1':'a'},{'Col1':'b'}])\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/73/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 449818897, "node_id": "MDU6SXNzdWU0NDk4MTg4OTc=", "number": 24, "title": "Additional Column Constraints?", "user": {"value": 98555, "label": "IgnoredAmbience"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2019-05-29T13:47:03Z", "updated_at": "2019-06-13T06:47:17Z", "closed_at": "2019-06-13T06:30:26Z", "author_association": "NONE", "pull_request": null, "body": "I'm looking to import data from XML with a pre-defined schema that maps fairly closely to a relational database.\r\nIn particular, it has explicit annotations for when fields are required, optional, or when a default value should be inferred.\r\n\r\nWould there be value in adding the ability to define `NOT NULL` and `DEFAULT` column constraints to sqlite-utils?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}