{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655653292, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T17:26:02Z", "updated_at": "2020-07-08T17:26:02Z", "author_association": "OWNER", "body": "Awesome, thank you very much.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655643078, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA==", "user": {"value": 79913, "label": "tsibley"}, "created_at": "2020-07-08T17:05:59Z", "updated_at": "2020-07-08T17:05:59Z", "author_association": "CONTRIBUTOR", "body": "> The only thing missing from this PR is updates to the documentation.\r\n\r\nAh, yes, thanks for this reminder! I've repushed with doc bits added.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655286864, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T05:05:27Z", "updated_at": "2020-07-08T05:05:36Z", "author_association": "OWNER", "body": "The only thing missing from this PR is updates to the documentation. Those need to go in two places:\r\n\r\n- In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts\r\n- In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data\r\n\r\nHere's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655284168, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T04:58:00Z", "updated_at": "2020-07-08T04:58:00Z", "author_association": "OWNER", "body": "Oops didn't mean to click \"close\" there.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655284054, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T04:57:38Z", "updated_at": "2020-07-08T04:57:38Z", "author_association": "OWNER", "body": "Thoughts on transactions would be much appreciated in #121 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655283393, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-07-08T04:55:18Z", "updated_at": "2020-07-08T04:55:18Z", "author_association": "OWNER", "body": "This is a really good idea - and thank you for the detailed discussion in the pull request.\r\n\r\nI'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:\r\n\r\n    with db.conn:\r\n        db[\"table\"].insert(...)\r\n\r\nBut it's not documented and I've not though very hard about it!\r\n\r\nI like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.\r\n\r\nLots to discuss here. I'll start a new issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655239728, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA==", "user": {"value": 79913, "label": "tsibley"}, "created_at": "2020-07-08T02:16:42Z", "updated_at": "2020-07-08T02:16:42Z", "author_association": "CONTRIBUTOR", "body": "I fixed my original oops by moving the `DELETE FROM $table` out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next.\r\n\r\nI wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making `Table.insert_all` (or other operations) consistent and robust in the face of errors.\r\n\r\nFor example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted):\r\n\r\n```diff\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex d6b9ecf..4107ceb 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -1028,6 +1028,11 @@ class Table(Queryable):\r\n         batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))\r\n         self.last_rowid = None\r\n         self.last_pk = None\r\n+        with self.db.conn:\r\n+            # Explicit BEGIN is necessary because Python's sqlite3 doesn't\r\n+            # issue implicit BEGINs for DDL, only DML.  We mix DDL and DML\r\n+            # below and might execute DDL first, e.g. for table creation.\r\n+            self.db.conn.execute(\"BEGIN\")\r\n             if truncate and self.exists():\r\n                 self.db.conn.execute(\"DELETE FROM [{}];\".format(self.name))\r\n             for chunk in chunks(itertools.chain([first_record], records), batch_size):\r\n@@ -1038,7 +1043,11 @@ class Table(Queryable):\r\n                         # Use the first batch to derive the table names\r\n                         column_types = suggest_column_types(chunk)\r\n                         column_types.update(columns or {})\r\n-                    self.create(\r\n+                        # Not self.create() because that is wrapped in its own\r\n+                        # transaction and Python's sqlite3 doesn't support\r\n+                        # nested transactions.\r\n+                        self.db.create_table(\r\n+                            self.name,\r\n                             column_types,\r\n                             pk,\r\n                             foreign_keys,\r\n@@ -1139,7 +1148,6 @@ class Table(Queryable):\r\n                     flat_values = list(itertools.chain(*values))\r\n                     queries_and_params = [(sql, flat_values)]\r\n \r\n-            with self.db.conn:\r\n                 for query, params in queries_and_params:\r\n                     try:\r\n                         result = self.db.conn.execute(query, params)\r\n```\r\n\r\nbut that fails in tests because other methods call `insert/upsert/insert_all/upsert_all` in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed).\r\n\r\nStepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the `sqlite_utils/db.py` code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make `sqlite_utils/cli.py` begin and commit a transaction in each `@cli.command` function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to _require_ callers to begin a transaction rather than just immediately calling methods.\r\n\r\nThere is also the caveat that for each transaction, an explicit `BEGIN` is also necessary so that DDL as well as DML (as well as `SELECT`s) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655052451, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ==", "user": {"value": 79913, "label": "tsibley"}, "created_at": "2020-07-07T18:45:23Z", "updated_at": "2020-07-07T18:45:23Z", "author_association": "CONTRIBUTOR", "body": "Ah, I see the problem. The truncate is inside a loop I didn't realize was there.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655018966", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/118", "id": 655018966, "node_id": "MDEyOklzc3VlQ29tbWVudDY1NTAxODk2Ng==", "user": {"value": 79913, "label": "tsibley"}, "created_at": "2020-07-07T17:41:06Z", "updated_at": "2020-07-07T17:41:06Z", "author_association": "CONTRIBUTOR", "body": "Hmm, while tests pass, this may not work as intended on larger datasets. Looking into it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 651844316, "label": "Add insert --truncate option"}, "performed_via_github_app": null}