{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1313128913", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1313128913, "node_id": "IC_kwDOBm6k_c5ORMHR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-14T05:48:22Z", "updated_at": "2022-11-14T05:48:22Z", "author_association": "OWNER", "body": "I changed my mind about the `\"return_rows\": true` option - I'm going to rename it to `\"return\": true`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1295200988", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1295200988, "node_id": "IC_kwDOBm6k_c5NMzLc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-28T16:29:55Z", "updated_at": "2022-10-28T16:29:55Z", "author_association": "OWNER", "body": "I wonder if there's something clever I could do here within a transaction?\r\n\r\nStart a transaction. Write out a temporary in-memory table with all of the existing primary keys in the table. Run the bulk insert. Then run `select pk from table where pk not in (select pk from old_pks)` to see what has changed.\r\n\r\nI don't think that's going to work well for large tables.\r\n\r\nI'm going to go with not returning inserted rows by default, unless you pass a special option requesting that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1294316640", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1294316640, "node_id": "IC_kwDOBm6k_c5NJbRg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-28T01:51:40Z", "updated_at": "2022-10-28T01:51:40Z", "author_association": "OWNER", "body": "This needs to support the following:\r\n- Rows do not include a primary key - one is assigned by the database\r\n- Rows provide their own primary key, any clashes are errors\r\n- Rows provide their own primary key, clashes are silently ignored\r\n- Rows provide their own primary key, replacing any existing records", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1294306071", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1294306071, "node_id": "IC_kwDOBm6k_c5NJYsX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-28T01:37:14Z", "updated_at": "2022-10-28T01:37:59Z", "author_association": "OWNER", "body": "Quick crude benchmark:\r\n```python\r\nimport sqlite3\r\n\r\ndb = sqlite3.connect(\":memory:\")\r\n\r\ndef create_table(db, name):\r\n    db.execute(f\"create table {name} (id integer primary key, title text)\")\r\n\r\ncreate_table(db, \"single\")\r\ncreate_table(db, \"multi\")\r\ncreate_table(db, \"bulk\")\r\n\r\ndef insert_singles(titles):\r\n    inserted = []\r\n    for title in titles:\r\n        cursor = db.execute(f\"insert into single (title) values (?)\", [title])\r\n        inserted.append((cursor.lastrowid, title))\r\n    return inserted\r\n\r\n\r\ndef insert_many(titles):\r\n    db.executemany(f\"insert into multi (title) values (?)\", ((t,) for t in titles))\r\n\r\n\r\ndef insert_bulk(titles):\r\n    db.execute(\"insert into bulk (title) values {}\".format(\r\n        \", \".join(\"(?)\" for _ in titles)\r\n    ), titles)\r\n\r\ntitles = [\"title {}\".format(i) for i in range(1, 10001)]\r\n```\r\nThen in iPython I ran these:\r\n```\r\nIn [14]: %timeit insert_singles(titles)\r\n23.8 ms \u00b1 535 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 10 loops each)\r\n\r\nIn [13]: %timeit insert_many(titles)\r\n12 ms \u00b1 520 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 100 loops each)\r\n\r\nIn [12]: %timeit insert_bulk(titles)\r\n2.59 ms \u00b1 25 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 100 loops each)\r\n```\r\nSo the bulk insert really is a lot faster - 3ms compared to 24ms for single inserts, so ~8x faster.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1294296767", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1294296767, "node_id": "IC_kwDOBm6k_c5NJWa_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-28T01:22:25Z", "updated_at": "2022-10-28T01:23:09Z", "author_association": "OWNER", "body": "Nasty catch on this one: I wanted to return the IDs of the freshly inserted rows. But... the `insert_all()` method I was planning to use from `sqlite-utils` doesn't appear to have a way of doing that:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611/sqlite_utils/db.py#L2813-L2835\r\n\r\nSQLite itself added a `RETURNING` statement which might help, but that is only available from version 3.35 released in March 2021: https://www.sqlite.org/lang_returning.html - which isn't commonly available yet. https://latest.datasette.io/-/versions right now shows 3.34, and https://lite.datasette.io/#/-/versions shows 3.27.2 (from Feb 2019).\r\n\r\nTwo options then:\r\n\r\n1. Even for bulk inserts do one insert at a time so I can use `cursor.lastrowid` to get the ID of the inserted record. This isn't terrible since SQLite is very fast, but it may still be a big performance hit for large inserts.\r\n2. Don't return the list of inserted rows for bulk inserts\r\n3. Default to not returning the list of inserted rows for bulk inserts, but allow the user to request that - in which case we use the slower path\r\n\r\nThat third option might be the way to go here.\r\n\r\nI should benchmark first to figure out how much of a difference this actually makes.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1294282263", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1294282263, "node_id": "IC_kwDOBm6k_c5NJS4X", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-28T01:00:42Z", "updated_at": "2022-10-28T01:00:42Z", "author_association": "OWNER", "body": "I'm going to set the limit at 1,000 rows inserted at a time. I'll make this configurable using a new `max_insert_rows` setting (for consistency with `max_returned_rows`).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1293893789", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1293893789, "node_id": "IC_kwDOBm6k_c5NH0Cd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-27T18:13:00Z", "updated_at": "2022-10-27T18:13:00Z", "author_association": "OWNER", "body": "If people care about that kind of thing they could always push all of their inserts to a table called `_tablename` and then atomically rename that once they've uploaded all of the data (assuming I provide an atomic-rename-this-table mechanism).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1293892818", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1293892818, "node_id": "IC_kwDOBm6k_c5NHzzS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-27T18:12:02Z", "updated_at": "2022-10-27T18:12:02Z", "author_association": "OWNER", "body": "There's one catch with batched inserts: if your CLI tool fails half way through you could end up with a partially populated table - since a bunch of batches will have succeeded first.\r\n\r\nI think that's OK. In the future I may want to come up with a way to run multiple batches of inserts inside a single transaction, but I can ignore that for the first release of this feature.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1293891876", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1293891876, "node_id": "IC_kwDOBm6k_c5NHzkk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-27T18:11:05Z", "updated_at": "2022-10-27T18:11:05Z", "author_association": "OWNER", "body": "Likewise for newline-delimited JSON. While it's tempting to want to accept that as an ingest format (because it's nice to generate and stream) I think it's better to have a client application that can turn a stream of newline-delimited JSON into batched JSON inserts.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1293891191", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1293891191, "node_id": "IC_kwDOBm6k_c5NHzZ3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-27T18:10:22Z", "updated_at": "2022-10-27T18:10:22Z", "author_association": "OWNER", "body": "So for the moment I'm just going to concentrate on the JSON API. I can consider CSV variants later on, or as plugins, or both.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1293890684", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1293890684, "node_id": "IC_kwDOBm6k_c5NHzR8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-27T18:09:52Z", "updated_at": "2022-10-27T18:09:52Z", "author_association": "OWNER", "body": "Should this API accept CSV/TSV etc in addition to JSON?\r\n\r\nI'm torn on this one. My initial instinct is that it should not - and there should instead be a Datasette client library / CLI tool you can use that knows how to turn CSV into batches of JSON calls for when you want to upload a CSV file.\r\n\r\nI don't think the usability of `curl https://datasette/db/table  -F 'data=@path/to/file.csv' -H 'Authentication: Bearer xxx'` is particularly great compared to  something like`datasette client insert https://datasette/ db table file.csv --csv` (where the command version could store API tokens for you too).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1866#issuecomment-1293887808", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1866", "id": 1293887808, "node_id": "IC_kwDOBm6k_c5NHylA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-27T18:07:02Z", "updated_at": "2022-10-27T18:07:02Z", "author_association": "OWNER", "body": "Error handling is really important here.\r\n\r\nWhat should happen if you submit 100 records and one of them has some kind of validation error? How should that error be reported back to you?\r\n\r\nI'm inclined to say that it defaults to all-or-nothing in a transaction - but there should be a `\"continue_on_error\": true` option (or similar) which causes it to insert the ones that are valid while reporting back the ones that are invalid.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1426001541, "label": "API for bulk inserting records into a table"}, "performed_via_github_app": null}