github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248591268 | https://api.github.com/repos/simonw/sqlite-utils/issues/486 | 1248591268 | IC_kwDOCGYnMM5Ka_2k | 9599 | 2022-09-15T20:36:02Z | 2022-09-15T20:40:03Z | OWNER | I had a big CSV file lying around, I converted it to other formats like this: sqlite-utils insert /tmp/t.db t /tmp/en.openfoodfacts.org.products.csv --csv sqlite-utils rows /tmp/t.db t --nl > /tmp/big.nl sqlite-utils rows /tmp/t.db t > /tmp/big.json Then tested the progress bar like this: sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl Output: ``` sqlite-utils insert /tmp/t2.db t /tmp/big.nl --nl [------------------------------------] 0% [#######-----------------------------] 20% 00:00:20 ``` With `--silent` it is silent. And for regular JSON: ``` sqlite-utils insert /tmp/t3.db t /tmp/big.json [####################################] 100% ``` This is actually not doing the right thing. The problem is that `sqlite-utils` doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1366512990 | |
https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248593835 | https://api.github.com/repos/simonw/sqlite-utils/issues/486 | 1248593835 | IC_kwDOCGYnMM5KbAer | 9599 | 2022-09-15T20:37:14Z | 2022-09-15T20:37:14Z | OWNER | I'm going to land this anyway. The lack of a streaming JSON parser is a separate issue, I don't think it should block landing this improvement. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1366512990 | |
https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248582147 | https://api.github.com/repos/simonw/sqlite-utils/issues/486 | 1248582147 | IC_kwDOCGYnMM5Ka9oD | 9599 | 2022-09-15T20:29:17Z | 2022-09-15T20:29:17Z | OWNER | This looks good to me. I need to run some manual tests before merging (it's a good sign that the automated tests pass though). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1366512990 | |
https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248568775 | https://api.github.com/repos/simonw/sqlite-utils/issues/486 | 1248568775 | IC_kwDOCGYnMM5Ka6XH | 9599 | 2022-09-15T20:16:14Z | 2022-09-15T20:16:14Z | OWNER | https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#using-the-python-version-input says can set the full version: ``` - uses: actions/setup-python@v4 with: python-version: "3.10.6" ``` I'll try that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1366512990 | |
https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248567323 | https://api.github.com/repos/simonw/sqlite-utils/issues/486 | 1248567323 | IC_kwDOCGYnMM5Ka6Ab | 9599 | 2022-09-15T20:14:45Z | 2022-09-15T20:14:45Z | OWNER | There's a fix for `mypy` that has landed but isn't out in a release yet: - https://github.com/python/mypy/issues/13385 For the moment looks like pinning to Python 3.10.6 could help. Need to figure out how to do that in GitHub Actions though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1366512990 | |
https://github.com/simonw/sqlite-utils/pull/486#issuecomment-1248565396 | https://api.github.com/repos/simonw/sqlite-utils/issues/486 | 1248565396 | IC_kwDOCGYnMM5Ka5iU | 9599 | 2022-09-15T20:12:50Z | 2022-09-15T20:12:50Z | OWNER | Annoying `mypy` test failure: ``` /Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/site-packages/numpy/__init__.pyi:636: error: Positional-only parameters are only supported in Python 3.8 and greater ``` Looks like this: - https://github.com/python/mypy/issues/13627 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1366512990 |