html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008557414,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008557414,IC_kwDOCGYnMM48HV1m,9599,2022-01-10T05:36:19Z,2022-01-10T05:36:19Z,OWNER,That did the trick.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008546573,IC_kwDOCGYnMM48HTMN,9599,2022-01-10T05:05:15Z,2022-01-10T05:05:15Z,OWNER,"Bit nasty but it might work:
```python
    def try_until(expected):
        tries = 0
        while True:
            rows = list(Database(db_path)[""rows""].rows)
            if rows == expected:
                return
            tries += 1
            if tries > 10:
                assert False, ""Expected {}, got {}"".format(expected, rows)
            time.sleep(tries * 0.1)

    try_until([{""name"": ""Azi""}])
    proc.stdin.write(b'{""name"": ""Suna""}\n')
    proc.stdin.flush()
    try_until([{""name"": ""Azi""}, {""name"": ""Suna""}])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008545140,IC_kwDOCGYnMM48HS10,9599,2022-01-10T05:01:34Z,2022-01-10T05:01:34Z,OWNER,"Urgh, tests are still failing intermittently - for example:
```
        time.sleep(0.4)
>       assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}]
E       AssertionError: assert [] == [{'name': 'Azi'}]
E         Right contains one more item: {'name': 'Azi'}
E         Full diff:
E         - [{'name': 'Azi'}]
E         + []
```
I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008537194,IC_kwDOCGYnMM48HQ5q,9599,2022-01-10T04:29:53Z,2022-01-10T04:31:29Z,OWNER,"After a bunch of debugging with `print()` statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time.

I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770

The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736,https://api.github.com/repos/simonw/sqlite-utils/issues/364,1008526736,IC_kwDOCGYnMM48HOWQ,9599,2022-01-10T04:07:29Z,2022-01-10T04:07:29Z,OWNER,"I think this test is right:
```python
def test_insert_streaming_batch_size_1(db_path):
    # https://github.com/simonw/sqlite-utils/issues/364
    # Streaming with --batch-size 1 should commit on each record
    # Can't use CliRunner().invoke() here bacuse we need to
    # run assertions in between writing to process stdin
    proc = subprocess.Popen(
        [
            sys.executable,
            ""-m"",
            ""sqlite_utils"",
            ""insert"",
            db_path,
            ""rows"",
            ""-"",
            ""--nl"",
            ""--batch-size"",
            ""1"",
        ],
        stdin=subprocess.PIPE,
    )
    proc.stdin.write(b'{""name"": ""Azi""}')
    proc.stdin.flush()
    assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}]
    proc.stdin.write(b'{""name"": ""Suna""}')
    proc.stdin.flush()
    assert list(Database(db_path)[""rows""].rows) == [{""name"": ""Azi""}, {""name"": ""Suna""}]
    proc.stdin.close()
    proc.wait()
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1095570074,