github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-714758139 | https://api.github.com/repos/simonw/sqlite-utils/issues/173 | 714758139 | MDEyOklzc3VlQ29tbWVudDcxNDc1ODEzOQ== | 9599 | 2020-10-22T20:57:56Z | 2020-10-22T20:57:56Z | OWNER | I could use `ijson` to provide a progress bar for JSON arrays too. I'd prefer to keep that as an optional dependency though, since `sqlite-utils` is a library dependency for many other projects and it would be using `ijson` purely for the CLI component. Here's how to iterate through a list of objects being read from a file: ```python import json parser = ijson.items(open( "/tmp/list.json" ), "item") for object in parser: # ... ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
707478649 | |
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698577508 | https://api.github.com/repos/simonw/sqlite-utils/issues/173 | 698577508 | MDEyOklzc3VlQ29tbWVudDY5ODU3NzUwOA== | 9599 | 2020-09-24T20:41:18Z | 2020-09-24T20:41:18Z | OWNER | I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen. Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
707478649 | |
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698578959 | https://api.github.com/repos/simonw/sqlite-utils/issues/173 | 698578959 | MDEyOklzc3VlQ29tbWVudDY5ODU3ODk1OQ== | 9599 | 2020-09-24T20:44:35Z | 2020-09-24T20:50:19Z | OWNER | I'm using a `click.File()` at the moment: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L496 I'll need to change that to be something that I can easily measure progress through. Also I should change its name - `json_file` is a bad name when it sometimes handles `csv` or `tsv` instead. It looks like the argument provided by `click.File` doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
707478649 | |
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698579389 | https://api.github.com/repos/simonw/sqlite-utils/issues/173 | 698579389 | MDEyOklzc3VlQ29tbWVudDY5ODU3OTM4OQ== | 9599 | 2020-09-24T20:45:29Z | 2020-09-24T20:45:29Z | OWNER | Relevant code: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L550-L560 Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
707478649 | |
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-697577646 | https://api.github.com/repos/simonw/sqlite-utils/issues/173 | 697577646 | MDEyOklzc3VlQ29tbWVudDY5NzU3NzY0Ng== | 9599 | 2020-09-23T15:48:51Z | 2020-09-23T15:48:51Z | OWNER | This can only work when it's reading from a file, not when it's reading from standard input. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
707478649 |