{"html_url": "https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011906", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/7", "id": 458011906, "node_id": "MDEyOklzc3VlQ29tbWVudDQ1ODAxMTkwNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-01-28T06:25:55Z", "updated_at": "2019-01-28T06:25:55Z", "author_association": "OWNER", "body": "I tested this with a script called `churn_em_out.py`\r\n```\r\ni = 0\r\nwhile True:\r\n    i += 1\r\n    print(\r\n        '{\"id\": I, \"another\": \"row\", \"number\": J}'.replace(\"I\", str(i)).replace(\r\n            \"J\", str(i + 1)\r\n        )\r\n    )\r\n```\r\n\r\nThen I ran this:\r\n```\r\npython churn_em_out.py | \\\r\n    sqlite-utils insert /tmp/getbig.db stats - \\\r\n        --nl --batch-size=10000\r\n```\r\nAnd used `watch 'ls -lah /tmp/getbig.db'` to watch the file growing as it had 10,000 lines of junk committed in batches. The memory used by the process never grew about around 50MB.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403625674, "label": ".insert_all() should accept a generator and process it efficiently"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/7#issuecomment-458011885", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/7", "id": 458011885, "node_id": "MDEyOklzc3VlQ29tbWVudDQ1ODAxMTg4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-01-28T06:25:48Z", "updated_at": "2019-01-28T06:25:48Z", "author_association": "OWNER", "body": "Re-opening for the second bit involving the cli tool.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403625674, "label": ".insert_all() should accept a generator and process it efficiently"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/7#issuecomment-457980966", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/7", "id": 457980966, "node_id": "MDEyOklzc3VlQ29tbWVudDQ1Nzk4MDk2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-01-28T02:29:32Z", "updated_at": "2019-01-28T02:29:32Z", "author_association": "OWNER", "body": "Remember to remove this TODO (and turn the `[]` into `()` on this line) as part of this task:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/5309c5c7755818323a0f5353bad0de98ecc866be/sqlite_utils/cli.py#L78-L80", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 403625674, "label": ".insert_all() should accept a generator and process it efficiently"}, "performed_via_github_app": null}