{"html_url": "https://github.com/simonw/sqlite-utils/issues/173#issuecomment-956041692", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/173", "id": 956041692, "node_id": "IC_kwDOCGYnMM44_Anc", "user": {"value": 2118708, "label": "Florents-Tselai"}, "created_at": "2021-11-01T08:42:24Z", "updated_at": "2021-11-01T08:42:24Z", "author_association": "NONE", "body": "> I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.\r\n> \r\n> Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.\r\n\r\nI was thinking about this, while inserting a stream of ~40M line-delimited json docs. Wouldn't a `--total-expected` flag work ? \r\n\r\nThat's [how tqdm does it](https://github.com/tqdm/tqdm/blob/fc69d5dcf578f7c7986fa76841a6b793f813df35/tqdm/std.py#L366)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707478649, "label": "Progress bar for sqlite-utils insert"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/173#issuecomment-714758139", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/173", "id": 714758139, "node_id": "MDEyOklzc3VlQ29tbWVudDcxNDc1ODEzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-22T20:57:56Z", "updated_at": "2020-10-22T20:57:56Z", "author_association": "OWNER", "body": "I could use `ijson` to provide a progress bar for JSON arrays too. I'd prefer to keep that as an optional dependency though, since `sqlite-utils` is a library dependency for many other projects and it would be using `ijson` purely for the CLI component.\r\n\r\nHere's how to iterate through a list of objects being read from a file:\r\n\r\n```python\r\nimport json\r\nparser = ijson.items(open(\r\n    \"/tmp/list.json\"\r\n), \"item\")\r\nfor object in parser:\r\n   # ...\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707478649, "label": "Progress bar for sqlite-utils insert"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698578959", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/173", "id": 698578959, "node_id": "MDEyOklzc3VlQ29tbWVudDY5ODU3ODk1OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-24T20:44:35Z", "updated_at": "2020-09-24T20:50:19Z", "author_association": "OWNER", "body": "I'm using a `click.File()` at the moment: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L496\r\n\r\nI'll need to change that to be something that I can easily measure progress through. Also I should change its name - `json_file` is a bad name when it sometimes handles `csv` or `tsv` instead.\r\n\r\nIt looks like the argument provided by `click.File` doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707478649, "label": "Progress bar for sqlite-utils insert"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698579389", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/173", "id": 698579389, "node_id": "MDEyOklzc3VlQ29tbWVudDY5ODU3OTM4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-24T20:45:29Z", "updated_at": "2020-09-24T20:45:29Z", "author_association": "OWNER", "body": "Relevant code: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L550-L560\r\n\r\nChanging that to track progress through NL-JSON, CSV and TSV shouldn't be too hard.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707478649, "label": "Progress bar for sqlite-utils insert"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698577508", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/173", "id": 698577508, "node_id": "MDEyOklzc3VlQ29tbWVudDY5ODU3NzUwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-24T20:41:18Z", "updated_at": "2020-09-24T20:41:18Z", "author_association": "OWNER", "body": "I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.\r\n\r\nNot sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707478649, "label": "Progress bar for sqlite-utils insert"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/173#issuecomment-697577646", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/173", "id": 697577646, "node_id": "MDEyOklzc3VlQ29tbWVudDY5NzU3NzY0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-23T15:48:51Z", "updated_at": "2020-09-23T15:48:51Z", "author_association": "OWNER", "body": "This can only work when it's reading from a file, not when it's reading from standard input.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 707478649, "label": "Progress bar for sqlite-utils insert"}, "performed_via_github_app": null}