{"html_url": "https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622587177", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103", "id": 622587177, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjU4NzE3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T22:07:51Z", "updated_at": "2020-05-01T22:07:51Z", "author_association": "OWNER", "body": "This is my failed attempt to recreate the bug (plus some extra debugging output):\r\n```diff\r\n % git diff\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex dd49d5c..ea42aea 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -1013,7 +1013,11 @@ class Table(Queryable):\r\n         assert (\r\n             num_columns <= SQLITE_MAX_VARS\r\n         ), \"Rows can have a maximum of {} columns\".format(SQLITE_MAX_VARS)\r\n+        print(\"default batch_size = \", batch_size)\r\n         batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))\r\n+        print(\"new batch_size = {},num_columns = {}, MAX_VARS // num_columns = {}\".format(\r\n+            batch_size, num_columns, SQLITE_MAX_VARS // num_columns\r\n+        ))\r\n         self.last_rowid = None\r\n         self.last_pk = None\r\n         for chunk in chunks(itertools.chain([first_record], records), batch_size):\r\n@@ -1124,6 +1128,9 @@ class Table(Queryable):\r\n                 )\r\n                 flat_values = list(itertools.chain(*values))\r\n                 queries_and_params = [(sql, flat_values)]\r\n+                print(sql.count(\"?\"), len(flat_values))\r\n+\r\n+            # print(json.dumps(queries_and_params, indent=4))\r\n \r\n             with self.db.conn:\r\n                 for query, params in queries_and_params:\r\ndiff --git a/tests/test_create.py b/tests/test_create.py\r\nindex 5290cd8..52940df 100644\r\n--- a/tests/test_create.py\r\n+++ b/tests/test_create.py\r\n@@ -853,3 +853,33 @@ def test_create_with_nested_bytes(fresh_db):\r\n     record = {\"id\": 1, \"data\": {\"foo\": b\"bytes\"}}\r\n     fresh_db[\"t\"].insert(record)\r\n     assert [{\"id\": 1, \"data\": '{\"foo\": \"b\\'bytes\\'\"}'}] == list(fresh_db[\"t\"].rows)\r\n+\r\n+\r\n+def test_create_throws_useful_error_with_increasing_number_of_columns(fresh_db):\r\n+    # https://github.com/simonw/sqlite-utils/issues/103\r\n+    def rows():\r\n+        yield {\"name\": 0}\r\n+        for i in range(1, 1001):\r\n+            yield {\r\n+                \"name\": i,\r\n+                \"age\": i,\r\n+                \"size\": i,\r\n+                \"name2\": i,\r\n+                \"age2\": i,\r\n+                \"size2\": i,\r\n+                \"name3\": i,\r\n+                \"age3\": i,\r\n+                \"size3\": i,\r\n+                \"name4\": i,\r\n+                \"age4\": i,\r\n+                \"size4\": i,\r\n+                \"name5\": i,\r\n+                \"age5\": i,\r\n+                \"size5\": i,\r\n+                \"name6\": i,\r\n+                \"age6\": i,\r\n+                \"size6\": i,\r\n+            }\r\n+\r\n+    fresh_db[\"t\"].insert_all(rows())\r\n+    assert 1001 == fresh_db[\"t\"].count\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610517472, "label": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622584433", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103", "id": 622584433, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjU4NDQzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T21:57:52Z", "updated_at": "2020-05-01T21:57:52Z", "author_association": "OWNER", "body": "@b0b5h4rp13 I'm having trouble creating a test that triggers this bug. Could you share a chunk of code that replicates what you're seeing here?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610517472, "label": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622565276", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103", "id": 622565276, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjU2NTI3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T20:57:16Z", "updated_at": "2020-05-01T20:57:16Z", "author_association": "OWNER", "body": "I'm reconsidering this: I think this is going to happen ANY time someone has at least one row that is wider than the first row. So at the very least I should show a more understandable error message.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610517472, "label": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622563188", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103", "id": 622563188, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjU2MzE4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T20:51:24Z", "updated_at": "2020-05-01T20:51:29Z", "author_association": "OWNER", "body": "Hopefully anyone who runs into this problem in the future will search for and find this issue thread!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610517472, "label": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622563059", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103", "id": 622563059, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjU2MzA1OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T20:51:01Z", "updated_at": "2020-05-01T20:51:01Z", "author_association": "OWNER", "body": "I'm not sure what to do about this.\r\n\r\nI was thinking the solution would be to look at ALL of the rows in a batch before deciding on the maximum number of columns, but that doesn't work because we calculate batch size based on the number of columns!\r\n\r\nI think my recommendation here is to manually pass a `batch_size=` argument to `.insert_all()` if you run into this error.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610517472, "label": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622561944", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103", "id": 622561944, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjU2MTk0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T20:47:51Z", "updated_at": "2020-05-01T20:47:51Z", "author_association": "OWNER", "body": "Yup we only take the number of columns in the first record into account at the moment:\r\nhttps://github.com/simonw/sqlite-utils/blob/d56029549acae0b0ea94c5a0f783e3b3895d9218/sqlite_utils/db.py#L1007-L1016", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610517472, "label": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622561585", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/103", "id": 622561585, "node_id": "MDEyOklzc3VlQ29tbWVudDYyMjU2MTU4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-01T20:46:50Z", "updated_at": "2020-05-01T20:46:50Z", "author_association": "OWNER", "body": "The varying number of columns thing is interesting - I don't think the tests cover that case much if at all.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 610517472, "label": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns"}, "performed_via_github_app": null}