{"id": 455996809, "node_id": "MDU6SXNzdWU0NTU5OTY4MDk=", "number": 28, "title": "Rearrange the docs by area, not CLI vs Python", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-06-13T23:33:35Z", "updated_at": "2019-07-15T02:37:20Z", "closed_at": "2019-07-15T02:37:20Z", "author_association": "OWNER", "pull_request": null, "body": "The docs for eg inserting data should live on the same page, rather than being split across the API and CLI pages.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/28/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 462423839, "node_id": "MDU6SXNzdWU0NjI0MjM4Mzk=", "number": 33, "title": "index_foreign_keys / index-foreign-keys utilities", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-06-30T16:42:03Z", "updated_at": "2019-06-30T23:54:11Z", "closed_at": "2019-06-30T23:50:55Z", "author_association": "OWNER", "pull_request": null, "body": "Sometimes it's good to have indices on all columns that are foreign keys, to allow for efficient reverse lookups.\r\n\r\nThis would be a useful utility:\r\n\r\n    $ sqlite-utils index-foreign-keys database.db\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/33/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 462423972, "node_id": "MDExOlB1bGxSZXF1ZXN0MjkzMTE3MTgz", "number": 34, "title": "sqlite-utils index-foreign-keys / db.index_foreign_keys()", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-06-30T16:43:40Z", "updated_at": "2019-06-30T23:50:55Z", "closed_at": "2019-06-30T23:50:55Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/34", "body": "Refs #33\r\n\r\n- [x] `sqlite-utils index-foreign-keys` command\r\n- [x] `db.index_foreign_keys()` method\r\n- [x] unit tests\r\n- [x] documentation", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/34/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 461237618, "node_id": "MDU6SXNzdWU0NjEyMzc2MTg=", "number": 31, "title": "Mechanism for adding multiple foreign key constraints at once", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-06-27T00:04:30Z", "updated_at": "2019-06-29T06:27:40Z", "closed_at": "2019-06-29T06:27:40Z", "author_association": "OWNER", "pull_request": null, "body": "Needed by [db-to-sqlite](https://github.com/simonw/db-to-sqlite). It currently works by collecting all of the foreign key relationships it can find and then applying them at the end of the process.\r\n\r\nThe problem is, the `add_foreign_key()` method looks like this:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/86bd2bba689e25f09551d611ccfbee1e069e5b66/sqlite_utils/db.py#L498-L516\r\n\r\nThat means it's doing a full `VACUUM` for every single relationship it sets up - and if you have hundreds of foreign key relationships in your database this can take hours.\r\n\r\nI think the right solution is to have a `.add_foreign_keys(list_of_args)` method which does the bulk operation and then a single `VACUUM`. `.add_foreign_key(...)` can then call the bulk action with a single list item.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/31/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 462094937, "node_id": "MDExOlB1bGxSZXF1ZXN0MjkyODc5MjA0", "number": 32, "title": "db.add_foreign_keys() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-06-28T15:40:33Z", "updated_at": "2019-06-29T06:27:39Z", "closed_at": "2019-06-29T06:27:39Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/32", "body": "Refs #31. Still TODO:\r\n\r\n- [x] Unit tests\r\n- [x] Documentation", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/32/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 458941203, "node_id": "MDU6SXNzdWU0NTg5NDEyMDM=", "number": 29, "title": "Prevent accidental add-foreign-key with invalid column", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-06-20T23:57:24Z", "updated_at": "2019-06-20T23:58:26Z", "closed_at": "2019-06-20T23:58:26Z", "author_association": "OWNER", "pull_request": null, "body": "You can corrupt your database by running:\r\n\r\n    $ sqlite-utils add-foreign-key my.db table non_existent_column other_table other_column\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/29/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 449818897, "node_id": "MDU6SXNzdWU0NDk4MTg4OTc=", "number": 24, "title": "Additional Column Constraints?", "user": {"value": 98555, "label": "IgnoredAmbience"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2019-05-29T13:47:03Z", "updated_at": "2019-06-13T06:47:17Z", "closed_at": "2019-06-13T06:30:26Z", "author_association": "NONE", "pull_request": null, "body": "I'm looking to import data from XML with a pre-defined schema that maps fairly closely to a relational database.\r\nIn particular, it has explicit annotations for when fields are required, optional, or when a default value should be inferred.\r\n\r\nWould there be value in adding the ability to define `NOT NULL` and `DEFAULT` column constraints to sqlite-utils?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/24/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 449848803, "node_id": "MDU6SXNzdWU0NDk4NDg4MDM=", "number": 25, "title": "Allow .insert(..., foreign_keys=()) to auto-detect table and primary key", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-05-29T14:39:22Z", "updated_at": "2019-06-13T05:32:32Z", "closed_at": "2019-06-13T05:32:32Z", "author_association": "OWNER", "pull_request": null, "body": "The `foreign_keys=` argument currently takes a list of triples:\r\n```python\r\ndb[\"usages\"].insert_all(\r\n    usages_to_insert,\r\n    foreign_keys=(\r\n        (\"line_id\", \"lines\", \"id\"),\r\n        (\"definition_id\", \"definitions\", \"id\"),\r\n    ),\r\n)\r\n```\r\nAs of #16 we have a mechanism for detecting the primary key column (the third item in this triple) - we should use that here too, so foreign keys can be optionally defined as a list of pairs.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/25/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 413867537, "node_id": "MDU6SXNzdWU0MTM4Njc1Mzc=", "number": 16, "title": "add_column() should support REFERENCES {other_table}({other_column})", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-02-24T21:00:45Z", "updated_at": "2019-05-29T05:17:59Z", "closed_at": "2019-05-29T04:56:18Z", "author_association": "OWNER", "pull_request": null, "body": "Related to #2 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 448391492, "node_id": "MDU6SXNzdWU0NDgzOTE0OTI=", "number": 21, "title": "Option to ignore inserts if primary key exists already", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-05-25T00:17:12Z", "updated_at": "2019-05-29T05:09:01Z", "closed_at": "2019-05-29T04:18:26Z", "author_association": "OWNER", "pull_request": null, "body": "> I've just noticed that SQLite lets you IGNORE inserts that collide with a pre-existing key. This can be quite handy if you have a dataset that keeps changing in part, and you don't want to upsert and replace pre-existing PK rows but you do want to ignore collisions to existing PK rows.\r\n> \r\n> Do `sqlite_utils` support such (cavalier!) behaviour?\r\n\r\n_Originally posted by @psychemedia in https://github.com/simonw/sqlite-utils/issues/18#issuecomment-480621924_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/21/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 448395665, "node_id": "MDU6SXNzdWU0NDgzOTU2NjU=", "number": 22, "title": "Release notes for 1.0", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 2, "created_at": "2019-05-25T00:58:03Z", "updated_at": "2019-05-25T01:18:27Z", "closed_at": "2019-05-25T01:06:52Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/compare/0.14...251e473", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/22/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 432727685, "node_id": "MDU6SXNzdWU0MzI3Mjc2ODU=", "number": 20, "title": "JSON column values get extraneously quoted ", "user": {"value": 649467, "label": "mhalle"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 1, "created_at": "2019-04-12T20:15:30Z", "updated_at": "2019-05-25T00:57:19Z", "closed_at": "2019-05-25T00:57:19Z", "author_association": "NONE", "pull_request": null, "body": "If the input to `sqlite-utils insert` includes a column that is a JSON array or object, `sqlite-utils query` will introduce an extra level of quoting on output:\r\n\r\n```\r\n# echo '[{\"key\": [\"one\", \"two\", \"three\"]}]' | sqlite-utils insert t.db t -\r\n\r\n# sqlite-utils t.db 'select * from t'\r\n[{\"key\": \"[\\\"one\\\", \\\"two\\\", \\\"three\\\"]\"}]\r\n\r\n# sqlite3 t.db 'select * from t'\r\n[\"one\", \"two\", \"three\"]\r\n```\r\n\r\nThis might require an imperfect solution, since sqlite3 doesn't have a JSON type. Perhaps fields that start with `[\"` or `{\"` and end with `\"]` or `\"}` could be detected, with a flag to turn off that behavior for weird text fields (or vice versa).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/20/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 432217625, "node_id": "MDU6SXNzdWU0MzIyMTc2MjU=", "number": 19, "title": "Incorrect help text for enable-fts command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 0, "created_at": "2019-04-11T19:46:44Z", "updated_at": "2019-05-25T00:44:31Z", "closed_at": "2019-05-25T00:44:31Z", "author_association": "OWNER", "pull_request": null, "body": "I clearly copied-and-pasted this from the `tables` command without updating it:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/0b1af42ead3b3902347951180b3364ce1942da6e/sqlite_utils/cli.py#L216-L222", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/19/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 413871266, "node_id": "MDU6SXNzdWU0MTM4NzEyNjY=", "number": 18, "title": ".insert/.upsert/.insert_all/.upsert_all should add missing columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 2, "created_at": "2019-02-24T21:36:11Z", "updated_at": "2019-05-25T00:42:11Z", "closed_at": "2019-05-25T00:42:11Z", "author_association": "OWNER", "pull_request": null, "body": "This is a larger change, but it would be incredibly useful: if you attempt to insert or update a document with a field that does not currently exist in the underlying table, sqlite-utils should add the appropriate column for you.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 403922644, "node_id": "MDU6SXNzdWU0MDM5MjI2NDQ=", "number": 8, "title": "Problems handling column names containing spaces or - ", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-01-28T17:23:28Z", "updated_at": "2019-04-14T15:29:33Z", "closed_at": "2019-02-23T21:09:03Z", "author_association": "NONE", "pull_request": null, "body": "Irrrespective of whether using column names containing a space or - character is good practice, SQLite does allow it, but `sqlite-utils` throws an error in the following cases:\r\n\r\n```python\r\nfrom sqlite_utils import Database\r\n\r\ndbname = 'test.db'\r\nDB = Database(sqlite3.connect(dbname))\r\n\r\nimport pandas as pd\r\ndf = pd.DataFrame({'col1':range(3), 'col2':range(3)})\r\n\r\n#Convert pandas dataframe to appropriate list/dict format\r\nDB['test1'].insert_all( df.to_dict(orient='records') )\r\n#Works fine\r\n```\r\n\r\nHowever:\r\n\r\n```python\r\ndf = pd.DataFrame({'col 1':range(3), 'col2':range(3)})\r\nDB['test1'].insert_all(df.to_dict(orient='records'))\r\n```\r\n\r\nthrows:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nOperationalError                          Traceback (most recent call last)\r\n<ipython-input-27-070b758f4f92> in <module>()\r\n      1 import pandas as pd\r\n      2 df = pd.DataFrame({'col 1':range(3), 'col2':range(3)})\r\n----> 3 DB['test1'].insert_all(df.to_dict(orient='records'))\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n    327                     jsonify_if_needed(record.get(key, None)) for key in all_columns\r\n    328                 )\r\n--> 329             result = self.db.conn.execute(sql, values)\r\n    330             self.db.conn.commit()\r\n    331             self.last_id = result.lastrowid\r\n\r\nOperationalError: near \"1\": syntax error\r\n```\r\n\r\nand:\r\n\r\n```python\r\ndf = pd.DataFrame({'col-1':range(3), 'col2':range(3)})\r\nDB['test1'].upsert_all(df.to_dict(orient='records'))\r\n```\r\n\r\nresults in:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nOperationalError                          Traceback (most recent call last)\r\n<ipython-input-28-654523549d20> in <module>()\r\n      1 import pandas as pd\r\n      2 df = pd.DataFrame({'col-1':range(3), 'col2':range(3)})\r\n----> 3 DB['test1'].insert_all(df.to_dict(orient='records'))\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n    327                     jsonify_if_needed(record.get(key, None)) for key in all_columns\r\n    328                 )\r\n--> 329             result = self.db.conn.execute(sql, values)\r\n    330             self.db.conn.commit()\r\n    331             self.last_id = result.lastrowid\r\n\r\nOperationalError: near \"-\": syntax error\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/8/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 413868452, "node_id": "MDU6SXNzdWU0MTM4Njg0NTI=", "number": 17, "title": "Improve and document foreign_keys=... argument to insert/create/etc", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2019-02-24T21:09:11Z", "updated_at": "2019-02-24T23:45:48Z", "closed_at": "2019-02-24T23:45:48Z", "author_association": "OWNER", "pull_request": null, "body": "The `foreign_keys=` argument to `table.insert_all()` and friends can be used to specify foreign key relationships that should be created.\r\n\r\nIt is not yet documented. It also requires you to specify the SQLite type of each column, even though this can be detected by introspecting the referenced table:\r\n\r\n    cols = [c for c in self.db[other_table].columns if c.name == other_column]\r\n    cols[0].type\r\n\r\nRelates to #2 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 349850687, "node_id": "MDU6SXNzdWUzNDk4NTA2ODc=", "number": 2, "title": "Mechanism for adding foreign keys to an existing table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2018-08-12T22:50:56Z", "updated_at": "2019-02-24T21:34:41Z", "closed_at": "2019-02-24T21:34:41Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite does not have ALTER TABLE support for adding new foreign keys... but it turns out it's possible to make these changes without having to duplicate the entire table by carefully running `UPDATE sqlite_master SET sql=... WHERE type='table' AND name='X';`\r\n\r\nHere's how Django does it: https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/db/backends/sqlite3/schema.py#L103-L125\r\n\r\nAnd here's the official documentation about this: https://sqlite.org/lang_altertable.html#otheralter (scroll to the very bottom of the page)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 413857257, "node_id": "MDU6SXNzdWU0MTM4NTcyNTc=", "number": 15, "title": "Ability to add columns to tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-02-24T19:20:51Z", "updated_at": "2019-02-24T20:04:40Z", "closed_at": "2019-02-24T20:04:40Z", "author_association": "OWNER", "pull_request": null, "body": "Makes sense to do this before foreign keys in #2\r\n\r\nPython:\r\n\r\n    db[\"table\"].add_column(\"new_column\", int)\r\n\r\nCLI:\r\n\r\n    $ sqlite-utils add-column table new_column INTEGER\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/15/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 413842611, "node_id": "MDU6SXNzdWU0MTM4NDI2MTE=", "number": 14, "title": "Utilities for adding indexes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-02-24T16:57:28Z", "updated_at": "2019-02-24T19:11:28Z", "closed_at": "2019-02-24T19:11:28Z", "author_association": "OWNER", "pull_request": null, "body": "Both in the Python API and the CLI tool. For the CLI tool this should work:\r\n\r\n    $ sqlite-utils create-index mydb.db mytable col1 col2\r\n\r\nThis will create a compound index across col1 and col2. The name of the index will be automatically chosen unless you use the `--name=...` option.\r\n\r\nSupport a `--unique` option too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 411066700, "node_id": "MDU6SXNzdWU0MTEwNjY3MDA=", "number": 10, "title": "Error in upsert if column named 'order'", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-02-16T12:05:18Z", "updated_at": "2019-02-24T16:55:38Z", "closed_at": "2019-02-24T16:55:37Z", "author_association": "NONE", "pull_request": null, "body": "The following works fine:\r\n```\r\nconnX = sqlite3.connect('DELME.db', timeout=10)\r\n\r\ndfX=pd.DataFrame({'col1':range(3),'col2':range(3)})\r\nDBX = Database(connX)\r\nDBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n```\r\n\r\nBut if a column is named `order`:\r\n```\r\nconnX = sqlite3.connect('DELME.db', timeout=10)\r\n\r\ndfX=pd.DataFrame({'order':range(3),'col2':range(3)})\r\nDBX = Database(connX)\r\nDBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n```\r\n\r\nit throws an error:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nOperationalError                          Traceback (most recent call last)\r\n<ipython-input-130-7dba33cd806c> in <module>\r\n      3 dfX=pd.DataFrame({'order':range(3),'col2':range(3)})\r\n      4 DBX = Database(connX)\r\n----> 5 DBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order)\r\n    347             foreign_keys=foreign_keys,\r\n    348             upsert=True,\r\n--> 349             column_order=column_order,\r\n    350         )\r\n    351 \r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n    327                     jsonify_if_needed(record.get(key, None)) for key in all_columns\r\n    328                 )\r\n--> 329             result = self.db.conn.execute(sql, values)\r\n    330             self.db.conn.commit()\r\n    331             self.last_id = result.lastrowid\r\n\r\nOperationalError: near \"order\": syntax error\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 413779210, "node_id": "MDU6SXNzdWU0MTM3NzkyMTA=", "number": 13, "title": "Ability to automatically create IDs from content hash of row", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-02-24T04:07:08Z", "updated_at": "2019-02-24T04:36:48Z", "closed_at": "2019-02-24T04:36:48Z", "author_association": "OWNER", "pull_request": null, "body": "Sometimes when you are importing data the underlying source provides records without IDs that can be uniquely identified by their contents.\r\n\r\nA utility mechanism for calculating a sha1 hash of the contents and using that as a unique ID would be useful.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 413740684, "node_id": "MDU6SXNzdWU0MTM3NDA2ODQ=", "number": 11, "title": "Detect numpy types when creating tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-02-23T21:09:35Z", "updated_at": "2019-02-24T04:02:20Z", "closed_at": "2019-02-24T04:02:20Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by #8", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 413778585, "node_id": "MDExOlB1bGxSZXF1ZXN0MjU1NjU4MTEy", "number": 12, "title": "Support for numpy types, closes #11", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-02-24T03:57:32Z", "updated_at": "2019-02-24T04:02:20Z", "closed_at": "2019-02-24T04:02:20Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/12", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 405801771, "node_id": "MDExOlB1bGxSZXF1ZXN0MjQ5NjgwOTQ0", "number": 9, "title": ":pencil: Updates my_database.py to my_database.db", "user": {"value": 50527, "label": "jefftriplett"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-02-01T17:35:43Z", "updated_at": "2019-02-24T03:55:04Z", "closed_at": "2019-02-24T03:55:04Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/9", "body": "I noticed that both `.py` and `.db` were used in the docs and assumed you'd prefer `.db`. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/9/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 403625674, "node_id": "MDU6SXNzdWU0MDM2MjU2NzQ=", "number": 7, "title": ".insert_all() should accept a generator and process it efficiently", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-01-28T02:11:58Z", "updated_at": "2019-01-28T06:26:53Z", "closed_at": "2019-01-28T06:26:53Z", "author_association": "OWNER", "pull_request": null, "body": "Right now you have to load every record into memory before passing the list to `.insert_all()` and friends.\r\n\r\nIf you want to process millions of rows, this is inefficient. Python has generators - we should use them!\r\n\r\nThe only catch here is that part of the magic of `sqlite-utils` is that it guesses the column types and creates the table for you. This code will need to be updated to notice if the table needs creating and, if it does, create it using the first X (where x=1,000 but can be customized) records.\r\n\r\nIf a record outside of those first 1,000 has a rogue column, we can crash with an error.\r\n\r\nThis will free us up to make the `--nl` option added in #6 much more efficient.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/7/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 403624090, "node_id": "MDU6SXNzdWU0MDM2MjQwOTA=", "number": 6, "title": "\"sqlite-utils insert\" should support newline-delimited JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-01-28T02:00:02Z", "updated_at": "2019-01-28T02:17:45Z", "closed_at": "2019-01-28T02:17:45Z", "author_association": "OWNER", "pull_request": null, "body": "We can already export newline delimited JSON. We should learn to import it as well.\r\n\r\nThe neat thing about importing it is that you can import GBs of data without having to read the whole lot into memory in order to decode the wrapping JSON array.\r\n\r\nDatasette can export it now: https://github.com/simonw/datasette/issues/405\r\n\r\nDemo: https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on\r\n\r\nIt should be possible to do this:\r\n\r\n    $ curl \"https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on\" \\\r\n        | sqlite-utils insert data.db facetable - --nl\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 403396009, "node_id": "MDExOlB1bGxSZXF1ZXN0MjQ3ODYxNDE5", "number": 5, "title": "Run Travis tests against Python 3.8-dev", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-01-26T02:30:55Z", "updated_at": "2019-01-26T02:37:54Z", "closed_at": "2019-01-26T02:37:54Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/5", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/5/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 403028630, "node_id": "MDExOlB1bGxSZXF1ZXN0MjQ3NTc2OTQy", "number": 4, "title": "Fts5", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-01-25T06:54:05Z", "updated_at": "2019-01-25T06:54:33Z", "closed_at": "2019-01-25T06:54:33Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/4", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/4/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}
{"id": 347058326, "node_id": "MDExOlB1bGxSZXF1ZXN0MjA1NzcwOTk2", "number": 1, "title": "Make .indexes compatible with older SQLite versions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-08-02T15:17:05Z", "updated_at": "2018-08-02T15:17:30Z", "closed_at": "2018-08-02T15:17:30Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/1", "body": "Older SQLite versions return a different set of columns from the PRAGMA we are using.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}