github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008557414 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008557414 | IC_kwDOCGYnMM48HV1m | 9599 | 2022-01-10T05:36:19Z | 2022-01-10T05:36:19Z | OWNER | That did the trick. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1095570074 | |
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008546573 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008546573 | IC_kwDOCGYnMM48HTMN | 9599 | 2022-01-10T05:05:15Z | 2022-01-10T05:05:15Z | OWNER | Bit nasty but it might work: ```python def try_until(expected): tries = 0 while True: rows = list(Database(db_path)["rows"].rows) if rows == expected: return tries += 1 if tries > 10: assert False, "Expected {}, got {}".format(expected, rows) time.sleep(tries * 0.1) try_until([{"name": "Azi"}]) proc.stdin.write(b'{"name": "Suna"}\n') proc.stdin.flush() try_until([{"name": "Azi"}, {"name": "Suna"}]) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1095570074 | |
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008545140 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008545140 | IC_kwDOCGYnMM48HS10 | 9599 | 2022-01-10T05:01:34Z | 2022-01-10T05:01:34Z | OWNER | Urgh, tests are still failing intermittently - for example: ``` time.sleep(0.4) > assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}] E AssertionError: assert [] == [{'name': 'Azi'}] E Right contains one more item: {'name': 'Azi'} E Full diff: E - [{'name': 'Azi'}] E + [] ``` I'm going to change this code to keep on trying up to 10 seconds - that should get the tests to pass faster on most machines. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1095570074 | |
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008537194 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008537194 | IC_kwDOCGYnMM48HQ5q | 9599 | 2022-01-10T04:29:53Z | 2022-01-10T04:31:29Z | OWNER | After a bunch of debugging with `print()` statements it's clear that the problem isn't with when things are committed or the size of the batches - it's that the data sent to standard input is all being processed in one go, not a line at a time. I think that's because it is being buffered by this: https://github.com/simonw/sqlite-utils/blob/d2a79d200f9071a86027365fa2a576865b71064f/sqlite_utils/cli.py#L759-L770 The buffering is there so that we can sniff the first few bytes to detect if it's a CSV file - added in 99ff0a288c08ec2071139c6031eb880fa9c95310 for #230. So maybe for non-CSV inputs we should disable buffering? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1095570074 | |
https://github.com/simonw/sqlite-utils/issues/364#issuecomment-1008526736 | https://api.github.com/repos/simonw/sqlite-utils/issues/364 | 1008526736 | IC_kwDOCGYnMM48HOWQ | 9599 | 2022-01-10T04:07:29Z | 2022-01-10T04:07:29Z | OWNER | I think this test is right: ```python def test_insert_streaming_batch_size_1(db_path): # https://github.com/simonw/sqlite-utils/issues/364 # Streaming with --batch-size 1 should commit on each record # Can't use CliRunner().invoke() here bacuse we need to # run assertions in between writing to process stdin proc = subprocess.Popen( [ sys.executable, "-m", "sqlite_utils", "insert", db_path, "rows", "-", "--nl", "--batch-size", "1", ], stdin=subprocess.PIPE, ) proc.stdin.write(b'{"name": "Azi"}') proc.stdin.flush() assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}] proc.stdin.write(b'{"name": "Suna"}') proc.stdin.flush() assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}, {"name": "Suna"}] proc.stdin.close() proc.wait() ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1095570074 |