html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-956041692,https://api.github.com/repos/simonw/sqlite-utils/issues/173,956041692,IC_kwDOCGYnMM44_Anc,2118708,Florents-Tselai,2021-11-01T08:42:24Z,2021-11-01T08:42:24Z,NONE,"> I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.
> 
> Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.

I was thinking about this, while inserting a stream of ~40M line-delimited json docs. Wouldn't a `--total-expected` flag work ? 

That's [how tqdm does it](https://github.com/tqdm/tqdm/blob/fc69d5dcf578f7c7986fa76841a6b793f813df35/tqdm/std.py#L366)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-714758139,https://api.github.com/repos/simonw/sqlite-utils/issues/173,714758139,MDEyOklzc3VlQ29tbWVudDcxNDc1ODEzOQ==,9599,simonw,2020-10-22T20:57:56Z,2020-10-22T20:57:56Z,OWNER,"I could use `ijson` to provide a progress bar for JSON arrays too. I'd prefer to keep that as an optional dependency though, since `sqlite-utils` is a library dependency for many other projects and it would be using `ijson` purely for the CLI component.

Here's how to iterate through a list of objects being read from a file:

```python
import json
parser = ijson.items(open(
    ""/tmp/list.json""
), ""item"")
for object in parser:
   # ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698578959,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698578959,MDEyOklzc3VlQ29tbWVudDY5ODU3ODk1OQ==,9599,simonw,2020-09-24T20:44:35Z,2020-09-24T20:50:19Z,OWNER,"I'm using a `click.File()` at the moment: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L496

I'll need to change that to be something that I can easily measure progress through. Also I should change its name - `json_file` is a bad name when it sometimes handles `csv` or `tsv` instead.

It looks like the argument provided by `click.File` doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698579389,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698579389,MDEyOklzc3VlQ29tbWVudDY5ODU3OTM4OQ==,9599,simonw,2020-09-24T20:45:29Z,2020-09-24T20:45:29Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L550-L560

Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698577508,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698577508,MDEyOklzc3VlQ29tbWVudDY5ODU3NzUwOA==,9599,simonw,2020-09-24T20:41:18Z,2020-09-24T20:41:18Z,OWNER,"I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.

Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-697577646,https://api.github.com/repos/simonw/sqlite-utils/issues/173,697577646,MDEyOklzc3VlQ29tbWVudDY5NzU3NzY0Ng==,9599,simonw,2020-09-23T15:48:51Z,2020-09-23T15:48:51Z,OWNER,"This can only work when it's reading from a file, not when it's reading from standard input.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,