{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-1303660293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 1303660293, "node_id": "IC_kwDOCGYnMM5NtEcF", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-04T14:38:36Z", "updated_at": "2022-11-04T14:38:36Z", "author_association": "CONTRIBUTOR", "body": "where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515751719", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 515751719, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTc1MTcxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-28T10:40:11Z", "updated_at": "2019-07-28T10:40:11Z", "author_association": "OWNER", "body": "I think the fix here is for me to switch to using `executemany()` - example from the Python docs: https://docs.python.org/3/library/sqlite3.html\r\n```python\r\npurchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),\r\n             ('2006-04-05', 'BUY', 'MSFT', 1000, 72.00),\r\n             ('2006-04-06', 'SELL', 'IBM', 500, 53.00),\r\n            ]\r\nc.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752129", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 515752129, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTc1MjEyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-28T10:46:49Z", "updated_at": "2019-07-28T10:46:49Z", "author_association": "OWNER", "body": "The problem with `.executemany()` is it breaks `lastrowid`:\r\n\r\n> This read-only attribute provides the rowid of the last modified row. It is only set if you issued an INSERT or a REPLACE statement using the execute() method. For operations other than INSERT or REPLACE or when executemany() is called, lastrowid is set to None.\r\n\r\nSo I think I need to continue to use my existing way of executing bulk inserts (with a giant repeated `INSERT INTO ... VALUES` block) but ensure that I calculate the chunk size such that I don't ever try to pass more than 999 values at once.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-515752204", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 515752204, "node_id": "MDEyOklzc3VlQ29tbWVudDUxNTc1MjIwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-07-28T10:48:14Z", "updated_at": "2019-07-28T10:48:14Z", "author_association": "OWNER", "body": "Here's the diff where I tried to use `.executemany()` and ran into the `lastrowid` problem:\r\n```diff\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex ef55976..7f85759 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -881,13 +881,10 @@ class Table:\r\n                 or_what=or_what,\r\n                 table=self.name,\r\n                 columns=\", \".join(\"[{}]\".format(c) for c in all_columns),\r\n-                rows=\", \".join(\r\n-                    \"\"\"\r\n+                rows=\"\"\"\r\n                     ({placeholders})\r\n                 \"\"\".format(\r\n-                        placeholders=\", \".join([\"?\"] * len(all_columns))\r\n-                    )\r\n-                    for record in chunk\r\n+                    placeholders=\", \".join([\"?\"] * len(all_columns))\r\n                 ),\r\n             )\r\n             values = []\r\n@@ -902,15 +899,15 @@ class Table:\r\n                         extract_table = extracts[key]\r\n                         value = self.db[extract_table].lookup({\"value\": value})\r\n                     record_values.append(value)\r\n-                values.extend(record_values)\r\n+                values.append(record_values)\r\n             with self.db.conn:\r\n                 try:\r\n-                    result = self.db.conn.execute(sql, values)\r\n+                    result = self.db.conn.executemany(sql, values)\r\n                 except sqlite3.OperationalError as e:\r\n                     if alter and (\" has no column \" in e.args[0]):\r\n                         # Attempt to add any missing columns, then try again\r\n                         self.add_missing_columns(chunk)\r\n-                        result = self.db.conn.execute(sql, values)\r\n+                        result = self.db.conn.executemany(sql, values)\r\n                     else:\r\n                         raise\r\n                 self.last_rowid = result.lastrowid\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}