{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248527646", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248527646, "node_id": "IC_kwDOCGYnMM5KawUe", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2022-09-15T19:34:59Z", "updated_at": "2022-09-15T20:23:12Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nBase: **96.47**% // Head: **96.52**% // Increases project coverage by **`+0.04%`** :tada:\n> Coverage data is based on head [(`0acbc68`)](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`d9b9e07`)](https://codecov.io/gh/simonw/sqlite-utils/commit/d9b9e075f07a20f1137cd2e34ed5d3f1a3db4ad8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n> Patch coverage: 100.00% of modified lines in pull request are covered.\n\n> :exclamation: Current head 0acbc68 differs from pull request most recent head d5db749. Consider uploading reports for the commit d5db749 to get more accurate results\n\n<details><summary>Additional details and impacted files</summary>\n\n\n```diff\n@@            Coverage Diff             @@\n##             main     #486      +/-   ##\n==========================================\n+ Coverage   96.47%   96.52%   +0.04%     \n==========================================\n  Files           6        6              \n  Lines        2642     2646       +4     \n==========================================\n+ Hits         2549     2554       +5     \n+ Misses         93       92       -1     \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/486/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.86% <100.00%> (\u00f8)` | |\n| [sqlite\\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/486/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.98% <100.00%> (+0.47%)` | :arrow_up: |\n\nHelp us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)\n\n</details>\n\n[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/486?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   \n:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248565396, "node_id": "IC_kwDOCGYnMM5Ka5iU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:12:50Z", "updated_at": "2022-09-15T20:12:50Z", "author_association": "OWNER", "body": "Annoying `mypy` test failure:\r\n\r\n```\r\n/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636:\r\nerror: Positional-only parameters are only supported in Python 3.8 and greater\r\n```\r\nLooks like this:\r\n- https://github.com/python/mypy/issues/13627", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248567323, "node_id": "IC_kwDOCGYnMM5Ka6Ab", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:14:45Z", "updated_at": "2022-09-15T20:14:45Z", "author_association": "OWNER", "body": "There's a fix for `mypy` that has landed but isn't out in a release yet:\r\n- https://github.com/python/mypy/issues/13385\r\n\r\nFor the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248568775, "node_id": "IC_kwDOCGYnMM5Ka6XH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:16:14Z", "updated_at": "2022-09-15T20:16:14Z", "author_association": "OWNER", "body": "https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version:\r\n\r\n```\r\n- uses: actions/setup-python@v4\r\n  with:\r\n    python-version: \"3.10.6\" \r\n```\r\nI'll try that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248582147, "node_id": "IC_kwDOCGYnMM5Ka9oD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:29:17Z", "updated_at": "2022-09-15T20:29:17Z", "author_association": "OWNER", "body": "This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248591268, "node_id": "IC_kwDOCGYnMM5Ka_2k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:36:02Z", "updated_at": "2022-09-15T20:40:03Z", "author_association": "OWNER", "body": "I had a big CSV file lying around, I converted it to other formats like this:\r\n\r\n    sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv\r\n    sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl\r\n    sqlite-utils rows /tmp/t.db t > /tmp/big.json\r\n\r\nThen tested the progress bar like this:\r\n\r\n    sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl\r\n\r\nOutput:\r\n\r\n```\r\nsqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl\r\n  [------------------------------------]    0%\r\n  [#######-----------------------------]   20%  00:00:20\r\n```\r\nWith `--silent` it is silent.\r\n\r\nAnd for regular JSON:\r\n\r\n```\r\nsqlite-utils insert /tmp/t3.db t /tmp/big.json \r\n  [####################################]  100%\r\n```\r\nThis is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248593835, "node_id": "IC_kwDOCGYnMM5KbAer", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:37:14Z", "updated_at": "2022-09-15T20:37:14Z", "author_association": "OWNER", "body": "I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}