{"id": 1094981339, "node_id": "I_kwDOCGYnMM5BRBbb", "number": 363, "title": "Better error message if `--convert` code fails to return a dict", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-01-06T05:26:28Z", "updated_at": "2022-02-03T22:52:30Z", "closed_at": "2022-02-03T22:51:30Z", "author_association": "OWNER", "pull_request": null, "body": "Here's the traceback if your `--convert` function doesn't return a dict right now:\r\n```\r\n% sqlite-utils insert /tmp/all.db blah /tmp/log.log --convert 'all.upper()' --all         \r\n\r\nTraceback (most recent call last):\r\n  File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/bin/sqlite-utils\", line 33, in <module>\r\n    sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())\r\n  File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1137, in __call__\r\n    return self.main(*args, **kwargs)\r\n  File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1062, in main\r\n    rv = self.invoke(ctx)\r\n  File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1668, in invoke\r\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n  File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 1404, in invoke\r\n    return ctx.invoke(self.callback, **ctx.params)\r\n  File \"/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.8/site-packages/click/core.py\", line 763, in invoke\r\n    return __callback(*args, **kwargs)\r\n  File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 949, in insert\r\n    insert_upsert_implementation(\r\n  File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 834, in insert_upsert_implementation\r\n    db[table].insert_all(\r\n  File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2602, in insert_all\r\n    first_record = next(records)\r\n  File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 3044, in fix_square_braces\r\n    for record in records:\r\n  File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py\", line 831, in <genexpr>\r\n    docs = (decode_base64_values(doc) for doc in docs)\r\n  File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 86, in decode_base64_values\r\n    to_fix = [\r\n  File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/utils.py\", line 89, in <listcomp>\r\n    if isinstance(doc[k], dict)\r\nTypeError: string indices must be integers\r\n```\r\nIt would be nicer if that returned a more useful error message.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/361#issuecomment-1006295276_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/363/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1095570074, "node_id": "I_kwDOCGYnMM5BTRKa", "number": 364, "title": "`--batch-size 1` doesn't seem to commit for every item", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 7558727, "label": "3.21"}, "comments": 16, "created_at": "2022-01-06T18:18:50Z", "updated_at": "2022-01-10T19:27:17Z", "closed_at": "2022-01-10T05:36:19Z", "author_association": "OWNER", "pull_request": null, "body": "I'm trying this, but it doesn't seem to write anything to the database file until I hit `CTRL+C`:\r\n\r\n```\r\nheroku logs --app=simonwillisonblog --tail | grep 'measure#nginx.service' | \\\r\n  sqlite-utils insert /tmp/herokutail.db log - --import re --convert \"$(cat <<EOD\r\n    r = re.compile(r'([^\\s=]+)=(?:\"(.*?)\"|(\\S+))')\r\n    pairs = {}\r\n    for key, value1, value2 in r.findall(line):\r\n        pairs[key] = value1 or value2\r\n    return pairs\r\nEOD\r\n)\" --lines --batch-size 1\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/364/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1094974713, "node_id": "I_kwDOCGYnMM5BQ_z5", "number": 362, "title": "upsert --detect-types is broken", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-01-06T05:12:10Z", "updated_at": "2022-01-06T06:54:45Z", "closed_at": "2022-01-06T06:28:34Z", "author_association": "OWNER", "pull_request": null, "body": "Noticed this thanks to syntax highlighting in VS Code showing an unused variable - need to fix it and add a test.\r\n\r\n<img src=\"https://user-images.githubusercontent.com/9599/148331804-6295bcf5-01fd-4159-b8ad-60c82acb97b4.png\" width=\"200\">\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/362/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1094890366, "node_id": "PR_kwDOCGYnMM4wlm3B", "number": 361, "title": "--lines and --text and --convert and --import", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 15, "created_at": "2022-01-06T01:49:44Z", "updated_at": "2022-01-06T06:37:03Z", "closed_at": "2022-01-06T06:24:54Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/361", "body": "Refs #356\r\n\r\nStill TODO:\r\n\r\n- [x] Get `--lines` working, with tests\r\n- [x] Get `--text` working, with tests\r\n- [x] Get regular JSON import working with `--convert` with tests\r\n- [x] Get `--lines` working with `--convert` with tests\r\n- [x] Get `--text` working with `--convert` with tests\r\n- [x] Get `--csv` and `--tsv` import working with `--convert` with tests\r\n- [x] Get `--nl` working with `--convert` with tests\r\n- [x] Documentation for all of the above", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/361/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}