{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248591268, "node_id": "IC_kwDOCGYnMM5Ka_2k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:36:02Z", "updated_at": "2022-09-15T20:40:03Z", "author_association": "OWNER", "body": "I had a big CSV file lying around, I converted it to other formats like this:\r\n\r\n    sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv\r\n    sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl\r\n    sqlite-utils rows /tmp/t.db t > /tmp/big.json\r\n\r\nThen tested the progress bar like this:\r\n\r\n    sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl\r\n\r\nOutput:\r\n\r\n```\r\nsqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl\r\n  [------------------------------------]    0%\r\n  [#######-----------------------------]   20%  00:00:20\r\n```\r\nWith `--silent` it is silent.\r\n\r\nAnd for regular JSON:\r\n\r\n```\r\nsqlite-utils insert /tmp/t3.db t /tmp/big.json \r\n  [####################################]  100%\r\n```\r\nThis is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248593835, "node_id": "IC_kwDOCGYnMM5KbAer", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:37:14Z", "updated_at": "2022-09-15T20:37:14Z", "author_association": "OWNER", "body": "I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248582147, "node_id": "IC_kwDOCGYnMM5Ka9oD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:29:17Z", "updated_at": "2022-09-15T20:29:17Z", "author_association": "OWNER", "body": "This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248568775, "node_id": "IC_kwDOCGYnMM5Ka6XH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:16:14Z", "updated_at": "2022-09-15T20:16:14Z", "author_association": "OWNER", "body": "https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version:\r\n\r\n```\r\n- uses: actions/setup-python@v4\r\n  with:\r\n    python-version: \"3.10.6\" \r\n```\r\nI'll try that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248567323, "node_id": "IC_kwDOCGYnMM5Ka6Ab", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:14:45Z", "updated_at": "2022-09-15T20:14:45Z", "author_association": "OWNER", "body": "There's a fix for `mypy` that has landed but isn't out in a release yet:\r\n- https://github.com/python/mypy/issues/13385\r\n\r\nFor the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486", "id": 1248565396, "node_id": "IC_kwDOCGYnMM5Ka5iU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-15T20:12:50Z", "updated_at": "2022-09-15T20:12:50Z", "author_association": "OWNER", "body": "Annoying `mypy` test failure:\r\n\r\n```\r\n/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636:\r\nerror: Positional-only parameters are only supported in Python 3.8 and greater\r\n```\r\nLooks like this:\r\n- https://github.com/python/mypy/issues/13627", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1366512990, "label": "progressbar for inserts/upserts of all fileformats, closes #485"}, "performed_via_github_app": null}