{"id": 347058326, "node_id": "MDExOlB1bGxSZXF1ZXN0MjA1NzcwOTk2", "number": 1, "title": "Make .indexes compatible with older SQLite versions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-08-02T15:17:05Z", "updated_at": "2018-08-02T15:17:30Z", "closed_at": "2018-08-02T15:17:30Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/1", "body": "Older SQLite versions return a different set of columns from the PRAGMA we are using.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 411066700, "node_id": "MDU6SXNzdWU0MTEwNjY3MDA=", "number": 10, "title": "Error in upsert if column named 'order'", "user": {"value": 82988, "label": "psychemedia"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-02-16T12:05:18Z", "updated_at": "2019-02-24T16:55:38Z", "closed_at": "2019-02-24T16:55:37Z", "author_association": "NONE", "pull_request": null, "body": "The following works fine:\r\n```\r\nconnX = sqlite3.connect('DELME.db', timeout=10)\r\n\r\ndfX=pd.DataFrame({'col1':range(3),'col2':range(3)})\r\nDBX = Database(connX)\r\nDBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n```\r\n\r\nBut if a column is named `order`:\r\n```\r\nconnX = sqlite3.connect('DELME.db', timeout=10)\r\n\r\ndfX=pd.DataFrame({'order':range(3),'col2':range(3)})\r\nDBX = Database(connX)\r\nDBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n```\r\n\r\nit throws an error:\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nOperationalError Traceback (most recent call last)\r\n in \r\n 3 dfX=pd.DataFrame({'order':range(3),'col2':range(3)})\r\n 4 DBX = Database(connX)\r\n----> 5 DBX['test'].upsert_all(dfX.to_dict(orient='records'))\r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order)\r\n 347 foreign_keys=foreign_keys,\r\n 348 upsert=True,\r\n--> 349 column_order=column_order,\r\n 350 )\r\n 351 \r\n\r\n/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order)\r\n 327 jsonify_if_needed(record.get(key, None)) for key in all_columns\r\n 328 )\r\n--> 329 result = self.db.conn.execute(sql, values)\r\n 330 self.db.conn.commit()\r\n 331 self.last_id = result.lastrowid\r\n\r\nOperationalError: near \"order\": syntax error\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/10/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 601358649, "node_id": "MDU6SXNzdWU2MDEzNTg2NDk=", "number": 100, "title": "Mechanism for forcing column-type, over-riding auto-detection", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-04-16T19:12:52Z", "updated_at": "2020-04-17T23:53:32Z", "closed_at": "2020-04-17T23:53:32Z", "author_association": "OWNER", "pull_request": null, "body": "As seen in https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406 - there's a problem where you insert a record with a `None` value for a column and that column is created as `TEXT` - but actually you intended it to be an `INT` (as later examples will demonstrate).\r\n\r\nSome kind of mechanism for over-riding the detected types of columns would be useful here.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/100/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 601392318, "node_id": "MDU6SXNzdWU2MDEzOTIzMTg=", "number": 101, "title": "README should include an example of CLI data insertion", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-16T19:45:37Z", "updated_at": "2020-04-17T23:59:49Z", "closed_at": "2020-04-17T23:59:49Z", "author_association": "OWNER", "pull_request": null, "body": "Maybe using `curl` from the GitHub API.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/101/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602569315, "node_id": "MDU6SXNzdWU2MDI1NjkzMTU=", "number": 102, "title": "Can't store an array or dictionary containing a bytes value", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-18T22:49:21Z", "updated_at": "2020-05-01T20:45:45Z", "closed_at": "2020-05-01T20:45:45Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\nIn [1]: import sqlite_utils \r\n\r\nIn [2]: db = sqlite_utils.Database(memory=True) \r\n\r\nIn [3]: db[\"t\"].insert({\"id\": 1, \"data\": {\"foo\": b\"bytes\"}}) \r\n---------------------------------------------------------------------------\r\nTypeError Traceback (most recent call last)\r\n in \r\n----> 1 db[\"t\"].insert({\"id\": 1, \"data\": {\"foo\": b\"bytes\"}})\r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert(self, record, pk, foreign_keys, column_order, not_null, defaults, hash_id, alter, ignore, replace, extracts, conversions, columns)\r\n 950 extracts=extracts,\r\n 951 conversions=conversions,\r\n--> 952 columns=columns,\r\n 953 )\r\n 954 \r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, conversions, columns, upsert)\r\n 1052 for key in all_columns:\r\n 1053 value = jsonify_if_needed(\r\n-> 1054 record.get(key, None if key != hash_id else _hash(record))\r\n 1055 )\r\n 1056 if key in extracts:\r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in jsonify_if_needed(value)\r\n 1318 def jsonify_if_needed(value):\r\n 1319 if isinstance(value, (dict, list, tuple)):\r\n-> 1320 return json.dumps(value)\r\n 1321 elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):\r\n 1322 return value.isoformat()\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)\r\n 229 cls is None and indent is None and separators is None and\r\n 230 default is None and not sort_keys and not kw):\r\n--> 231 return _default_encoder.encode(obj)\r\n 232 if cls is None:\r\n 233 cls = JSONEncoder\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in encode(self, o)\r\n 197 # exceptions aren't as detailed. The list call should be roughly\r\n 198 # equivalent to the PySequence_Fast that ''.join() would do.\r\n--> 199 chunks = self.iterencode(o, _one_shot=True)\r\n 200 if not isinstance(chunks, (list, tuple)):\r\n 201 chunks = list(chunks)\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in iterencode(self, o, _one_shot)\r\n 255 self.key_separator, self.item_separator, self.sort_keys,\r\n 256 self.skipkeys, _one_shot)\r\n--> 257 return _iterencode(o, 0)\r\n 258 \r\n 259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in default(self, o)\r\n 177 \r\n 178 \"\"\"\r\n--> 179 raise TypeError(f'Object of type {o.__class__.__name__} '\r\n 180 f'is not JSON serializable')\r\n 181 \r\n\r\nTypeError: Object of type bytes is not JSON serializable\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/102/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610517472, "node_id": "MDU6SXNzdWU2MTA1MTc0NzI=", "number": 103, "title": "sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns", "user": {"value": 32605365, "label": "b0b5h4rp13"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 8, "created_at": "2020-05-01T02:26:14Z", "updated_at": "2020-05-14T00:18:57Z", "closed_at": "2020-05-14T00:18:57Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "If using insert_all to put in 1000 rows of data with varying number of columns, it comes up with this message `sqlite3.OperationalError: too many SQL variables` if the number of columns is larger in later records (past the first row)\r\n\r\nI've reduced `SQLITE_MAX_VARS` by 100 to 899 at the top of `db.py` to add wiggle room, so that if the column count increases it wont go past SQLite's batch limit as calculated by this line of code based on the count of the first row's dict keys\r\n\r\n batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/103/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610853393, "node_id": "MDU6SXNzdWU2MTA4NTMzOTM=", "number": 104, "title": "--schema option to \"sqlite-utils tables\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-05-01T16:55:49Z", "updated_at": "2020-05-01T17:12:37Z", "closed_at": "2020-05-01T17:12:37Z", "author_association": "OWNER", "pull_request": null, "body": "Adds output showing the table schema.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/104/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610853576, "node_id": "MDU6SXNzdWU2MTA4NTM1NzY=", "number": 105, "title": "\"sqlite-utils views\" command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-01T16:56:11Z", "updated_at": "2020-05-01T20:40:07Z", "closed_at": "2020-05-01T20:38:36Z", "author_association": "OWNER", "pull_request": null, "body": "Similar to `sqlite-utils tables`. See also #104.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/105/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611216862, "node_id": "MDU6SXNzdWU2MTEyMTY4NjI=", "number": 106, "title": "create_view(..., ignore=True, replace=True) parameters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-02T15:45:21Z", "updated_at": "2020-05-02T16:04:51Z", "closed_at": "2020-05-02T16:02:10Z", "author_association": "OWNER", "pull_request": null, "body": "Two new parameters which specify what should happen if the view already exists. I want this for https://github.com/dogsheep/github-to-sqlite/issues/37\r\n\r\nHere's the current `create_view()` implementation:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/b4d953d3ccef28bb81cea40ca165a647b59971fa/sqlite_utils/db.py#L325-L332\r\n\r\n`ignore=True` will not do anything if the view exists already.\r\n\r\n`replace=True` will drop and redefine the view - but only if its SQL definition differs, otherwise it will be left alone.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/106/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611222968, "node_id": "MDU6SXNzdWU2MTEyMjI5Njg=", "number": 107, "title": "sqlite-utils create-view CLI command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-02T16:15:13Z", "updated_at": "2020-05-03T15:36:58Z", "closed_at": "2020-05-03T15:36:37Z", "author_association": "OWNER", "pull_request": null, "body": "Can go with #27 - `sqlite-utils create-table`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611326701, "node_id": "MDU6SXNzdWU2MTEzMjY3MDE=", "number": 108, "title": "Documentation unit tests for CLI commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-03T03:58:42Z", "updated_at": "2020-05-03T04:13:57Z", "closed_at": "2020-05-03T04:13:57Z", "author_association": "OWNER", "pull_request": null, "body": "Have a test that ensures all CLI commands are documented.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/108/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 612658444, "node_id": "MDU6SXNzdWU2MTI2NTg0NDQ=", "number": 109, "title": "table.create_index(..., ignore=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-05T14:44:21Z", "updated_at": "2020-05-05T14:46:53Z", "closed_at": "2020-05-05T14:46:53Z", "author_association": "OWNER", "pull_request": null, "body": "Option to silently do nothing if the index already exists.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/109/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413740684, "node_id": "MDU6SXNzdWU0MTM3NDA2ODQ=", "number": 11, "title": "Detect numpy types when creating tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-02-23T21:09:35Z", "updated_at": "2019-02-24T04:02:20Z", "closed_at": "2019-02-24T04:02:20Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by #8", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 613755043, "node_id": "MDU6SXNzdWU2MTM3NTUwNDM=", "number": 110, "title": "Support decimal.Decimal type", "user": {"value": 134771, "label": "dvhthomas"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-05-07T03:57:19Z", "updated_at": "2020-05-11T01:58:20Z", "closed_at": "2020-05-11T01:50:11Z", "author_association": "NONE", "pull_request": null, "body": "Decimal types in Postgres cause a failure in db.py data type selection\r\n---\r\nI have a Django app using a MoneyField, which uses a `numeric(14,0)` data type in Postgres (https://www.postgresql.org/docs/9.3/datatype-numeric.html). When attempting to export that table I get the following error:\r\n\r\n```bash\r\n$ db-to-sqlite --table isaweb_proposal \"postgres://connection\" test.db\r\n....\r\n column_type=COLUMN_TYPE_MAPPING[column_type],\r\nKeyError: \r\n```\r\n\r\nLooking at `sql_utils.db.py` at 292-ish it's clear that there is no matching type for what I assume SQLAlchemy interprets as Python decimal.Decimal.\r\n\r\nFrom the [SQLite docs](https://www.sqlite.org/datatype3.html#affinity_name_examples) it looks like DECIMAL in other DBs are considered numeric.\r\n\r\nI'm not quite sure if it's as simple as adding a data type to that list or if there are repercussions beyond it.\r\n\r\nThanks for a great tool!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/110/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 615477131, "node_id": "MDU6SXNzdWU2MTU0NzcxMzE=", "number": 111, "title": "sqlite-utils drop-table and drop-view commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-10T21:10:42Z", "updated_at": "2020-05-11T01:58:36Z", "closed_at": "2020-05-11T00:44:26Z", "author_association": "OWNER", "pull_request": null, "body": "Would be useful to be able to drop views and tables from the CLI.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 616271236, "node_id": "MDU6SXNzdWU2MTYyNzEyMzY=", "number": 112, "title": "add_foreign_key(...., ignore=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 4, "created_at": "2020-05-12T00:24:00Z", "updated_at": "2020-09-20T22:17:34Z", "closed_at": "2020-09-20T22:17:34Z", "author_association": "OWNER", "pull_request": null, "body": "When using this library I often find myself wanting to \"add this foreign key, but only if it doesn't exist yet\". The `ignore=True` parameter is increasingly being used for this else where in the library (e.g. in `create_view()`).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/112/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 621286870, "node_id": "MDU6SXNzdWU2MjEyODY4NzA=", "number": 113, "title": "Syntactic sugar for ATTACH DATABASE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-19T21:10:00Z", "updated_at": "2021-02-19T05:09:12Z", "closed_at": "2021-02-19T04:56:36Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/lang_attach.html\r\n\r\nMaybe something like this:\r\n```python\r\ndb.attach(\"other_db\", \"other_db.db\")\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/113/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 621989740, "node_id": "MDU6SXNzdWU2MjE5ODk3NDA=", "number": 114, "title": "table.transform() method for advanced alter table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 26, "created_at": "2020-05-20T18:20:46Z", "updated_at": "2020-09-22T07:51:37Z", "closed_at": "2020-09-22T04:20:02Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite's `ALTER TABLE` can only do the following:\r\n\r\n* Rename a table\r\n* Rename a column\r\n* Add a column\r\n\r\nNotably, it cannot drop columns - so tricks like \"add a float version of this text column, populate it, then drop the old one and rename\" won't work.\r\n\r\nThe docs here https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes describe a way of implementing full alters safely within a transaction, but it's fiddly.\r\n\r\n1. Create new table\r\n2. Copy data\r\n3. Drop old table\r\n4. Rename new into old \r\n\r\nIt would be great if `sqlite-utils` provided an abstraction to help make these kinds of changes safely.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/114/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 637889964, "node_id": "MDU6SXNzdWU2Mzc4ODk5NjQ=", "number": 115, "title": "Ability to execute insert/update statements with the CLI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-06-12T17:01:17Z", "updated_at": "2020-06-12T17:51:11Z", "closed_at": "2020-06-12T17:41:10Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n$ sqlite-utils github.db \"update stars set starred_at = ''\"\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 673, in query\r\n headers = [c[0] for c in cursor.description]\r\nTypeError: 'NoneType' object is not iterable\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/115/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 644122661, "node_id": "MDU6SXNzdWU2NDQxMjI2NjE=", "number": 116, "title": "Documentation for table.pks introspection property", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-23T20:27:24Z", "updated_at": "2020-06-23T21:21:33Z", "closed_at": "2020-06-23T21:03:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L535-L540", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/116/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 644161221, "node_id": "MDU6SXNzdWU2NDQxNjEyMjE=", "number": 117, "title": "Support for compound (composite) foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-06-23T21:33:42Z", "updated_at": "2020-06-23T21:40:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite\r\n\r\nTheir example looks like this:\r\n```sql\r\nCREATE TABLE album(\r\n albumartist TEXT,\r\n albumname TEXT,\r\n albumcover BINARY,\r\n PRIMARY KEY(albumartist, albumname)\r\n);\r\n\r\nCREATE TABLE song(\r\n songid INTEGER,\r\n songartist TEXT,\r\n songalbum TEXT,\r\n songname TEXT,\r\n FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname)\r\n);\r\n```\r\n\r\nHere's what that looks like in sqlite-utils:\r\n\r\n```\r\nIn [1]: import sqlite_utils \r\n\r\nIn [2]: import sqlite3 \r\n\r\nIn [3]: conn = sqlite3.connect(\":memory:\") \r\n\r\nIn [4]: conn \r\nOut[4]: \r\n\r\nIn [5]: conn.executescript(\"\"\" \r\n ...: CREATE TABLE album( \r\n ...: albumartist TEXT, \r\n ...: albumname TEXT, \r\n ...: albumcover BINARY, \r\n ...: PRIMARY KEY(albumartist, albumname) \r\n ...: ); \r\n ...: \r\n ...: CREATE TABLE song( \r\n ...: songid INTEGER, \r\n ...: songartist TEXT, \r\n ...: songalbum TEXT, \r\n ...: songname TEXT, \r\n ...: FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) \r\n ...: ); \r\n ...: \"\"\") \r\nOut[5]: \r\n\r\nIn [6]: db = sqlite_utils.Database(conn) \r\n\r\nIn [7]: db.tables \r\nOut[7]: \r\n[,\r\n
]\r\n\r\nIn [8]: db.tables[0].foreign_keys \r\nOut[8]: []\r\n\r\nIn [9]: db.tables[1].foreign_keys \r\nOut[9]: \r\n[ForeignKey(table='song', column='songartist', other_table='album', other_column='albumartist'),\r\n ForeignKey(table='song', column='songalbum', other_table='album', other_column='albumname')]\r\n```\r\nThe table appears to have two separate foreign keys, when actually it has a single compound composite foreign key.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/117/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 651844316, "node_id": "MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2", "number": 118, "title": "Add insert --truncate option", "user": {"value": 79913, "label": "tsibley"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-07-06T21:58:40Z", "updated_at": "2020-07-08T17:26:21Z", "closed_at": "2020-07-08T17:26:21Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/118", "body": "\r\n\r\nDeletes all rows in the table (if it exists) before inserting new rows.\r\nSQLite doesn't implement a TRUNCATE TABLE statement but does optimize an\r\nunqualified DELETE FROM.\r\n\r\nThis can be handy if you want to refresh the entire contents of a table\r\nbut a) don't have a PK (so can't use --replace), b) don't want the table\r\nto disappear (even briefly) for other connections, and c) have to handle\r\nrecords that used to exist being deleted.\r\n\r\nIdeally the replacement of rows would appear instantaneous to other\r\nconnections by putting the DELETE + INSERT in a transaction, but this is\r\nvery difficult without breaking other code as the current transaction\r\nhandling is inconsistent and non-systematic. There exists the\r\npossibility for the DELETE to succeed but the INSERT to fail, leaving an\r\nempty table. This is not much worse, however, than the current\r\npossibility of one chunked INSERT succeeding and being committed while\r\nthe next chunked INSERT fails, leaving a partially complete operation.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/118/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 652700770, "node_id": "MDU6SXNzdWU2NTI3MDA3NzA=", "number": 119, "title": "Ability to remove a foreign key", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-07T22:31:37Z", "updated_at": "2020-09-24T20:36:59Z", "closed_at": "2020-09-24T20:36:59Z", "author_association": "OWNER", "pull_request": null, "body": "Useful if you add one but make a mistake and need to undo it without recreating the database from scratch.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/119/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413778585, "node_id": "MDExOlB1bGxSZXF1ZXN0MjU1NjU4MTEy", "number": 12, "title": "Support for numpy types, closes #11", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-02-24T03:57:32Z", "updated_at": "2019-02-24T04:02:20Z", "closed_at": "2019-02-24T04:02:20Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/12", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 652816158, "node_id": "MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4", "number": 120, "title": "Fix query command's support for DML", "user": {"value": 79913, "label": "tsibley"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-08T01:36:34Z", "updated_at": "2020-07-08T05:14:04Z", "closed_at": "2020-07-08T05:14:04Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/120", "body": "See commit messages for details. I ran into this while investigating another feature/issue.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/120/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 652961907, "node_id": "MDU6SXNzdWU2NTI5NjE5MDc=", "number": 121, "title": "Improved (and better documented) support for transactions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-08T04:56:51Z", "updated_at": "2020-09-24T20:36:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_\r\n\r\nWe should put some thought into how this library supports and encourages smart use of transactions.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/121/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 665700495, "node_id": "MDU6SXNzdWU2NjU3MDA0OTU=", "number": 122, "title": "CLI utility for inserting binary files into SQLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-07-26T03:27:39Z", "updated_at": "2020-07-27T07:10:41Z", "closed_at": "2020-07-27T07:09:03Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite BLOB columns can store entire binary files. The challenge is inserting them, since they don't neatly fit into JSON objects.\r\n\r\nIt would be great if the `sqlite-utils` CLI had a trick for helping with this.\r\n\r\nInspired by https://github.com/simonw/datasette-media/issues/14", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/122/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665701216, "node_id": "MDU6SXNzdWU2NjU3MDEyMTY=", "number": 123, "title": "--raw option for outputting binary content", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-07-26T03:35:39Z", "updated_at": "2020-07-26T16:44:11Z", "closed_at": "2020-07-26T16:44:11Z", "author_association": "OWNER", "pull_request": null, "body": "Related to the `insert-files` work in #122 - it should be easy to get binary data back out of the database again.\r\n\r\nOne way to do that could be:\r\n\r\n sqlite-utils files.db \"select content from files where key = 'foo.jpg'\" --raw\r\n\r\nThe `--raw` option would cause just the contents of the first column to be output directly to stdout.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/123/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665802405, "node_id": "MDU6SXNzdWU2NjU4MDI0MDU=", "number": 124, "title": "sqlite-utils query should support named parameters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-26T15:25:10Z", "updated_at": "2020-07-30T22:57:51Z", "closed_at": "2020-07-27T03:53:58Z", "author_association": "OWNER", "pull_request": null, "body": "To help out with escaping - so you can run this:\r\n\r\n sqlite-utils query \"insert into foo (blah) values (:blah)\" --param blah `something here`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665817570, "node_id": "MDU6SXNzdWU2NjU4MTc1NzA=", "number": 125, "title": "Output binary columns in \"sqlite-utils query\" JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-07-26T16:47:02Z", "updated_at": "2020-07-27T00:49:41Z", "closed_at": "2020-07-27T00:48:45Z", "author_association": "OWNER", "pull_request": null, "body": "You get an error if you try to run a query that returns data from a BLOB.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/125/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665819048, "node_id": "MDU6SXNzdWU2NjU4MTkwNDg=", "number": 126, "title": "Ability to insert binary data on the CLI using JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-26T16:54:14Z", "updated_at": "2020-07-27T04:00:33Z", "closed_at": "2020-07-27T03:59:45Z", "author_association": "OWNER", "pull_request": null, "body": "> I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says \"these columns are base64 encoded, store the decoded data in a BLOB\".\r\n>\r\n> That would solve inserting binary data using JSON too.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 666040390, "node_id": "MDU6SXNzdWU2NjYwNDAzOTA=", "number": 127, "title": "Ability to insert files piped to insert-files stdin", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-27T07:09:33Z", "updated_at": "2020-07-30T03:08:52Z", "closed_at": "2020-07-30T03:08:18Z", "author_association": "OWNER", "pull_request": null, "body": "> Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.\r\n>\r\n> cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif\r\n>\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/127/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 666639051, "node_id": "MDU6SXNzdWU2NjY2MzkwNTE=", "number": 128, "title": "Support UUID and memoryview types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-27T23:08:34Z", "updated_at": "2020-07-30T01:10:43Z", "closed_at": "2020-07-30T01:10:43Z", "author_association": "OWNER", "pull_request": null, "body": "`psycopg2` can return data from PostgreSQL as `uuid.UUID` or `memoryview` objects. These should to be supported by `sqlite-utils` - mainly for https://github.com/simonw/db-to-sqlite", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/128/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 668308777, "node_id": "MDU6SXNzdWU2NjgzMDg3Nzc=", "number": 129, "title": "\"insert-files --sqlar\" for creating SQLite archives", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-30T02:28:29Z", "updated_at": "2020-07-30T22:41:01Z", "closed_at": "2020-07-30T22:40:55Z", "author_association": "OWNER", "pull_request": null, "body": "A `--sqlar` option could cause `insert-files` to behave in the same way as SQLite's own sqlar mechanism.\r\n\r\nhttps://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413779210, "node_id": "MDU6SXNzdWU0MTM3NzkyMTA=", "number": 13, "title": "Ability to automatically create IDs from content hash of row", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-02-24T04:07:08Z", "updated_at": "2019-02-24T04:36:48Z", "closed_at": "2019-02-24T04:36:48Z", "author_association": "OWNER", "pull_request": null, "body": "Sometimes when you are importing data the underlying source provides records without IDs that can be uniquely identified by their contents.\r\n\r\nA utility mechanism for calculating a sha1 hash of the contents and using that as a unique ID would be useful.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 671130371, "node_id": "MDU6SXNzdWU2NzExMzAzNzE=", "number": 130, "title": "Support tokenize option for FTS", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-01T19:27:22Z", "updated_at": "2020-08-01T20:51:28Z", "closed_at": "2020-08-01T20:51:14Z", "author_association": "OWNER", "pull_request": null, "body": "FTS5 supports things like porter stemming using a `tokenize=` option:\r\n\r\nhttps://www.sqlite.org/fts5.html#tokenizers\r\n\r\nSomething like this in code:\r\n```\r\n CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} (\r\n {columns},\r\n tokenize='porter',\r\n content=[{table}]\r\n );\r\n```\r\nI tried this out just now and it worked exactly as expected.\r\n\r\nSo... `db[table].enable_fts(...) should accept a 'tokenize=` argument, and `sqlite-utils enable-fts ...` should support a `--tokenize` option.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/130/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 675753042, "node_id": "MDU6SXNzdWU2NzU3NTMwNDI=", "number": 131, "title": "sqlite-utils insert: options for column types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-09T18:59:11Z", "updated_at": "2022-03-15T13:21:42Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs.\r\n\r\nIt would be useful if you could do the following:\r\n\r\n- automatically detects the column types based on eg the first 1000 records\r\n- explicitly state the rule for specific columns\r\n\r\n`--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types`\r\n\r\nFor specific columns maybe this:\r\n\r\n sqlite-utils insert db.db images images.tsv \\\r\n --tsv \\\r\n -c id int \\\r\n -c score float", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 675839512, "node_id": "MDU6SXNzdWU2NzU4Mzk1MTI=", "number": 132, "title": "Features for enabling and disabling WAL mode", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-10T03:25:44Z", "updated_at": "2020-08-10T18:59:35Z", "closed_at": "2020-08-10T18:59:35Z", "author_association": "OWNER", "pull_request": null, "body": "I finally figured out how to enable WAL - turns out it's a property of the database file itself: https://github.com/simonw/til/blob/master/sqlite/enabling-wal-mode.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/132/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 677839979, "node_id": "MDU6SXNzdWU2Nzc4Mzk5Nzk=", "number": 133, "title": "Release a sdist to PyPI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-12T16:55:09Z", "updated_at": "2020-08-12T17:05:06Z", "closed_at": "2020-08-12T17:05:06Z", "author_association": "OWNER", "pull_request": null, "body": "https://pypi.org/project/sqlite-utils/#files currently just has a wheel. I need this to package for homebrew: https://github.com/simonw/homebrew-datasette/issues/10", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/133/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683804172, "node_id": "MDU6SXNzdWU2ODM4MDQxNzI=", "number": 134, "title": "--load-extension option for sqlite-utils query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-08-21T20:12:42Z", "updated_at": "2020-08-21T21:06:26Z", "closed_at": "2020-08-21T20:54:19Z", "author_association": "OWNER", "pull_request": null, "body": "I got this error:\r\n```\r\n% sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view'\r\nTraceback (most recent call last):\r\n...\r\n cursor = db.conn.execute(sql, dict(param))\r\nsqlite3.OperationalError: no such function: AsGeoJSON\r\n```\r\nA `--load-extension=/usr/local/lib/mod_spatialite.dylib` option (imitating the same option for Datasette) would help.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/134/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683805434, "node_id": "MDU6SXNzdWU2ODM4MDU0MzQ=", "number": 135, "title": "Code for finding SpatiaLite in the usual locations", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-21T20:15:34Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-08-21T20:30:13Z", "author_association": "OWNER", "pull_request": null, "body": "I built this for `shapefile-to-sqlite` but it would be useful in `sqlite-utils` too:\r\n\r\nhttps://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19\r\n\r\n```python\r\nSPATIALITE_PATHS = (\r\n \"/usr/lib/x86_64-linux-gnu/mod_spatialite.so\",\r\n \"/usr/local/lib/mod_spatialite.dylib\",\r\n)\r\n```\r\n\r\nhttps://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109\r\n\r\n```python\r\ndef find_spatialite():\r\n for path in SPATIALITE_PATHS:\r\n if os.path.exists(path):\r\n return path\r\n return None\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683812642, "node_id": "MDU6SXNzdWU2ODM4MTI2NDI=", "number": 136, "title": "--load-extension=spatialite shortcut option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-21T20:31:25Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-16T19:14:32Z", "author_association": "OWNER", "pull_request": null, "body": "In conjunction with #135 - this would do the same thing as `--load-extension=path-to-spatialite` (see #134)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683830416, "node_id": "MDU6SXNzdWU2ODM4MzA0MTY=", "number": 137, "title": "--load-extension for other sqlite-utils commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-21T21:12:56Z", "updated_at": "2020-10-16T19:14:32Z", "closed_at": "2020-10-16T19:14:32Z", "author_association": "OWNER", "pull_request": null, "body": "e.g. for this:\r\n```\r\ncalands-datasette % sqlite-utils tables calands.db --counts\r\n[{\"table\": \"spatial_ref_sys\", \"count\": 4924},\r\n {\"table\": \"spatialite_history\", \"count\": 14},\r\n {\"table\": \"sqlite_sequence\", \"count\": 1},\r\n {\"table\": \"geometry_columns\", \"count\": 2},\r\n {\"table\": \"spatial_ref_sys_aux\", \"count\": 4873},\r\n {\"table\": \"views_geometry_columns\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns\", \"count\": 0},\r\n {\"table\": \"geometry_columns_statistics\", \"count\": 2},\r\n {\"table\": \"views_geometry_columns_statistics\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_statistics\", \"count\": 0},\r\n {\"table\": \"geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"views_geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"geometry_columns_time\", \"count\": 2},\r\n {\"table\": \"geometry_columns_auth\", \"count\": 2},\r\n {\"table\": \"views_geometry_columns_auth\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_auth\", \"count\": 0},\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 143, in tables\r\n for line in output_rows(_iter(), headers, nl, arrays, json_cols):\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 922, in output_rows\r\n for row, next_row in itertools.zip_longest(current_iter, next_iter):\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 123, in _iter\r\n row.append(db[name].count)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/db.py\", line 458, in count\r\n return self.db.conn.execute(\r\nsqlite3.OperationalError: no such module: VirtualSpatialIndex\r\n```\r\nThe `tables` command could take `--load-extension` too - as could `rows` and other similar commands.\r\n\r\nFollow-on from #134 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/137/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 684118950, "node_id": "MDU6SXNzdWU2ODQxMTg5NTA=", "number": 138, "title": "extracts= doesn't configure foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-23T05:21:15Z", "updated_at": "2020-09-24T22:47:01Z", "closed_at": "2020-09-24T22:46:52Z", "author_association": "OWNER", "pull_request": null, "body": "In using `extracts=` for `shapefiles-to-sqlite` in https://github.com/simonw/shapefile-to-sqlite/issues/9 I've run into a couple of pretty serious flaws:\r\n\r\n- The columns in the original table are still `TEXT` even when the foreign key they are supposed to reference is an `INTEGER` - which means Datasette foreign key features don't actually work\r\n- Those foreign key relationships aren't setup automatically - creating them is left as an exercise for the developer", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/138/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 686978131, "node_id": "MDU6SXNzdWU2ODY5NzgxMzE=", "number": 139, "title": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-08-27T06:25:25Z", "updated_at": "2020-08-28T22:48:51Z", "closed_at": "2020-08-28T22:30:14Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Is there a way to make `.insert_all()` work properly when new columns are introduced outside the first 100 records (with or without the `alter=True` argument)?\r\n\r\nI'm using `.insert_all()` to bulk insert ~3-4k records at a time and it is common for records to need to introduce new columns. However, if new columns are introduced after the first 100 records, `sqlite_utils` doesn't even raise the `OperationalError: table ... has no column named ...` exception; it just silently drops the extra data and moves on.\r\n\r\nIt took me a while to find this little snippet in the [documentation for `.insert_all()`](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts) (it's not mentioned under [Adding columns automatically on insert/update](https://sqlite-utils.readthedocs.io/en/stable/python-api.html#bulk-inserts)):\r\n\r\n> The column types used in the CREATE TABLE statement are automatically derived from the types of data in that first batch of rows. **_Any additional or missing columns in subsequent batches will be ignored._**\r\n\r\nI tried changing the `batch_size` argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.\r\n\r\nIs there a way around this that you would suggest? It seems like it should raise an exception at least.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/139/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413842611, "node_id": "MDU6SXNzdWU0MTM4NDI2MTE=", "number": 14, "title": "Utilities for adding indexes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-02-24T16:57:28Z", "updated_at": "2019-02-24T19:11:28Z", "closed_at": "2019-02-24T19:11:28Z", "author_association": "OWNER", "pull_request": null, "body": "Both in the Python API and the CLI tool. For the CLI tool this should work:\r\n\r\n $ sqlite-utils create-index mydb.db mytable col1 col2\r\n\r\nThis will create a compound index across col1 and col2. The name of the index will be automatically chosen unless you use the `--name=...` option.\r\n\r\nSupport a `--unique` option too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688351054, "node_id": "MDU6SXNzdWU2ODgzNTEwNTQ=", "number": 140, "title": "Idea: insert-files mechanism for adding extra columns with fixed values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T20:57:36Z", "updated_at": "2022-03-20T19:45:45Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Say for example you want to populate a `file_type` column with the value `gif`. That could work like this:\r\n\r\n```\r\nsqlite-utils insert-files gifs.db images *.gif \\\r\n -c path -c md5 -c last_modified:mtime \\\r\n -c file_type:text:gif --pk=path\r\n```\r\nSo a column defined as a `text` column with a value that follows a second colon.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 688352145, "node_id": "MDU6SXNzdWU2ODgzNTIxNDU=", "number": 141, "title": "insert-files support for compressed values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-08-28T20:59:46Z", "updated_at": "2020-09-24T20:36:08Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `sqlar` format supports this, it would be useful if `insert-files` could support this too.\r\n\r\nhttps://www.sqlite.org/sqlar.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/141/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 688386219, "node_id": "MDExOlB1bGxSZXF1ZXN0NDc1NjY1OTg0", "number": 142, "title": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-28T22:22:57Z", "updated_at": "2020-08-30T07:28:23Z", "closed_at": "2020-08-28T22:30:14Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/142", "body": "Closes #139.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/142/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 688389933, "node_id": "MDU6SXNzdWU2ODgzODk5MzM=", "number": 143, "title": "Move to GitHub Actions CI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T22:34:11Z", "updated_at": "2020-08-28T22:41:35Z", "closed_at": "2020-08-28T22:41:35Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/143/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688395275, "node_id": "MDU6SXNzdWU2ODgzOTUyNzU=", "number": 144, "title": "Run some tests against numpy", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-28T22:53:00Z", "updated_at": "2020-08-28T22:57:05Z", "closed_at": "2020-08-28T22:57:04Z", "author_association": "OWNER", "pull_request": null, "body": "Accidentally removed in #143:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/d7d3f962861ef32c5ead8f514c8756f5b6f7c4a0/.travis.yml#L18-L19", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688659182, "node_id": "MDU6SXNzdWU2ODg2NTkxODI=", "number": 145, "title": "Bug when first record contains fewer columns than subsequent records", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-30T05:44:44Z", "updated_at": "2020-09-08T23:21:23Z", "closed_at": "2020-09-08T23:21:23Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "`insert_all()` selects the maximum batch size based on the number of fields in the first record. If the first record has fewer fields than subsequent records (and `alter=True` is passed), this can result in SQL statements with more than the maximum permitted number of host parameters. This situation is perhaps unlikely to occur, but could happen if the first record had, say, 10 columns, such that `batch_size` (based on `SQLITE_MAX_VARIABLE_NUMBER = 999`) would be 99. If the next 98 rows had 11 columns, the resulting SQL statement for the first batch would have `10 * 1 + 11 * 98 = 1088` host parameters (and subsequent batches, if the data were consistent from thereon out, would have `99 * 11 = 1089`).\r\n\r\nI suspect that this bug is masked somewhat by the fact that while:\r\n> [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.\r\n\r\nit is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with `SQLITE_MAX_VARIABLE_NUMBER` set to 250,000, and I believe this is the case for homebrew installations too.\r\n\r\nA test for this issue might look like this:\r\n```python\r\ndef test_columns_not_in_first_record_should_not_cause_batch_to_be_too_large(fresh_db):\r\n # sqlite on homebrew and Debian/Ubuntu etc. is typically compiled with\r\n # SQLITE_MAX_VARIABLE_NUMBER set to 250,000, so we need to exceed this value to\r\n # trigger the error on these systems.\r\n THRESHOLD = 250000\r\n extra_columns = 1 + (THRESHOLD - 1) // 99\r\n records = [\r\n {\"c0\": \"first record\"}, # one column in first record -> batch_size = 100\r\n # fill out the batch with 99 records with enough columns to exceed THRESHOLD\r\n *[\r\n dict([(\"c{}\".format(i), j) for i in range(extra_columns)])\r\n for j in range(99)\r\n ]\r\n ]\r\n try:\r\n fresh_db[\"too_many_columns\"].insert_all(records, alter=True)\r\n except sqlite3.OperationalError:\r\n raise\r\n```\r\n\r\nThe best solution, I think, is simply to process all the records when determining columns, column types, and the batch size. In my tests this doesn't seem to be particularly costly at all, and cuts out a lot of complications (including obviating my implementation of #139 at #142). I'll raise a PR for your consideration.\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/145/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688668680, "node_id": "MDExOlB1bGxSZXF1ZXN0NDc1ODc0NDkz", "number": 146, "title": "Handle case where subsequent records (after first batch) include extra columns", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-30T07:13:58Z", "updated_at": "2020-09-08T23:20:37Z", "closed_at": "2020-09-08T23:20:37Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/146", "body": "Addresses #145.\r\n\r\nI think this should do the job. If it meets with your approval I'll update this PR to include an update to the documentation -- I came across this bug while preparing a PR to update the documentation around `batch_size` in any event.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/146/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 688670158, "node_id": "MDU6SXNzdWU2ODg2NzAxNTg=", "number": 147, "title": "SQLITE_MAX_VARS maybe hard-coded too low", "user": {"value": 96218, "label": "simonwiles"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-08-30T07:26:45Z", "updated_at": "2021-02-15T21:27:55Z", "closed_at": null, "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I came across this while about to open an issue and PR against the documentation for `batch_size`, which is a bit incomplete.\r\n\r\nAs mentioned in #145, while:\r\n\r\n> [`SQLITE_MAX_VARIABLE_NUMBER`](https://www.sqlite.org/limits.html#max_variable_number) ... defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.\r\n\r\nit is common that it is increased at compile time. Debian-based systems, for example, seem to ship with a version of sqlite compiled with SQLITE_MAX_VARIABLE_NUMBER set to 250,000, and I believe this is the case for homebrew installations too.\r\n\r\nIn working to understand what `batch_size` was actually doing and why, I realized that by setting `SQLITE_MAX_VARS` in `db.py` to match the value my sqlite was compiled with (I'm on Debian), I was able to decrease the time to `insert_all()` my test data set (~128k records across 7 tables) from ~26.5s to ~3.5s. Given that this about .05% of my total dataset, this is time I am keen to save...\r\n\r\nUnfortunately, it seems that `sqlite3` in the python standard library doesn't expose the `get_limit()` C API (even though `pysqlite` used to), so it's hard to know what value sqlite has been compiled with (note that this could mean, I suppose, that it's less than 999, and even hardcoding `SQLITE_MAX_VARS` to the conservative default might not be adequate. It can also be lowered -- but not raised -- at runtime). The best I could come up with is `echo \"\" | sqlite3 -cmd \".limits variable_number\"` (only available in `sqlite >= 2015-05-07 (3.8.10)`).\r\n\r\nObviously this couldn't be relied upon in `sqlite_utils`, but I wonder what your opinion would be about exposing `SQLITE_MAX_VARS` as a user-configurable parameter (with suitable \"here be dragons\" warnings)? I'm going to go ahead and monkey-patch it for my purposes in any event, but it seems like it might be worth considering.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/147/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 695276328, "node_id": "MDU6SXNzdWU2OTUyNzYzMjg=", "number": 148, "title": "More attractive indentation of created FTS table schema", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-07T16:49:30Z", "updated_at": "2020-09-07T18:12:50Z", "closed_at": "2020-09-07T18:12:50Z", "author_association": "OWNER", "pull_request": null, "body": "On https://github-to-sqlite.dogsheep.net/github/licenses_fts the create table SQL is displayed as:\r\n```sql\r\nCREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (\r\n [name],\r\n content=[licenses]\r\n );\r\n```\r\nIt would be more aesthetically pleasing if it looked like this:\r\n```sql\r\nCREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (\r\n [name],\r\n content=[licenses]\r\n);\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/148/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695319258, "node_id": "MDU6SXNzdWU2OTUzMTkyNTg=", "number": 149, "title": "FTS table with 7 rows has _fts_docsize table with 9,141 rows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-09-07T18:06:16Z", "updated_at": "2020-09-07T21:16:34Z", "closed_at": "2020-09-07T21:16:34Z", "author_association": "OWNER", "pull_request": null, "body": "I'm seeing a weird issue with some of the SQLite databases that I am using with the FTS5 module.\r\n\r\nI have a database with a `licenses` table that contains 7 rows: \r\n\r\nThe FTS table also has 7 rows: \r\n\r\nSomehow the accompanying `licenses_fts_docsize` shadow table now has 9,141 rows in it! \r\n\r\nAnd `licenses_fts_data` has 41 rows - should I expect that to have 7 rows? \r\n\r\nI have a hunch that it might be a problem with the triggers. These are the triggers that are updating that FTS table: \r\n\r\n| type | name | tbl_name | rootpage | sql |\r\n| --- | --- | --- | --- | --- |\r\n| trigger | licenses_ai | licenses | 0 | `CREATE TRIGGER [licenses_ai] AFTER INSERT ON [licenses] BEGIN INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |\r\n| trigger | licenses_ad | licenses | 0 | `CREATE TRIGGER [licenses_ad] AFTER DELETE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); END` |\r\n| trigger | licenses_au | licenses | 0 | `CREATE TRIGGER [licenses_au] AFTER UPDATE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/149/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413857257, "node_id": "MDU6SXNzdWU0MTM4NTcyNTc=", "number": 15, "title": "Ability to add columns to tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-02-24T19:20:51Z", "updated_at": "2019-02-24T20:04:40Z", "closed_at": "2019-02-24T20:04:40Z", "author_association": "OWNER", "pull_request": null, "body": "Makes sense to do this before foreign keys in #2\r\n\r\nPython:\r\n\r\n db[\"table\"].add_column(\"new_column\", int)\r\n\r\nCLI:\r\n\r\n $ sqlite-utils add-column table new_column INTEGER\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/15/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695359607, "node_id": "MDU6SXNzdWU2OTUzNTk2MDc=", "number": 150, "title": "Feature for tracing SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-07T19:43:08Z", "updated_at": "2020-09-07T21:57:01Z", "closed_at": "2020-09-07T21:57:01Z", "author_association": "OWNER", "pull_request": null, "body": "Debugging `sqlite-utils` when something weird happens (e.g. #149) can be a bit tricky since it runs a bunch of different SQL statements behind the scenes.\r\n\r\nAn optional \"tracing\" mechanism for seeing what SQL is being executed would be useful.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/150/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695360889, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgxNjE2NzA0", "number": 151, "title": "Tracer mechanism for seeing underlying SQL", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-07T19:46:43Z", "updated_at": "2020-09-07T21:57:00Z", "closed_at": "2020-09-07T21:57:00Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/151", "body": "Refs #150. Needs tests and documentation, including for the new `db.execute()` and `db.executescript()` methods.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/151/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 695376054, "node_id": "MDU6SXNzdWU2OTUzNzYwNTQ=", "number": 152, "title": "Turn on recursive_triggers by default", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T20:26:36Z", "updated_at": "2020-09-07T21:17:48Z", "closed_at": "2020-09-07T20:45:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:\r\n\r\n> Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.\r\n\r\nSo I think the fix for the complex issue in #149 is to turn on `recursive_triggers` globally by default for `sqlite-utils`.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/152/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695377804, "node_id": "MDU6SXNzdWU2OTUzNzc4MDQ=", "number": 153, "title": "table.optimize() should delete junk rows from *_fts_docsize", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-09-07T20:31:09Z", "updated_at": "2020-09-24T20:35:46Z", "closed_at": "2020-09-07T21:16:33Z", "author_association": "OWNER", "pull_request": null, "body": "> The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL:\r\n> ```sql\r\n> DELETE FROM [licenses_fts_docsize] WHERE id NOT IN (\r\n> SELECT rowid FROM [licenses_fts]);\r\n> ```\r\n> I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/153/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695441530, "node_id": "MDU6SXNzdWU2OTU0NDE1MzA=", "number": 154, "title": "OperationalError: cannot change into wal mode from within a transaction", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T23:42:44Z", "updated_at": "2020-09-07T23:47:10Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm getting this error when running:\r\n\r\n sqlite-utils enable-wal beta.db\r\n\r\n`OperationalError: cannot change into wal mode from within a transaction`\r\n\r\nI'm worried that maybe that's because of this new code from #152:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 696045581, "node_id": "MDU6SXNzdWU2OTYwNDU1ODE=", "number": 155, "title": "rebuild-fts command and table.rebuild_fts() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-08T17:19:26Z", "updated_at": "2020-09-24T20:35:46Z", "closed_at": "2020-09-08T23:16:10Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite.org/forum/forumpost/fa777fff86\r\n\r\n> Easiest thing would be to run a 'rebuild' to rebuild the FTS index from scratch based on the contents of the content table. i.e.\r\n>\r\n> INSERT INTO licenses_fts(licenses_fts) VALUES('rebuild');\r\n>\r\n> https://www.sqlite.org/fts5.html#the_rebuild_command", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/155/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 697030843, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgzMDI3NTg3", "number": 156, "title": "Typos in tests", "user": {"value": 96218, "label": "simonwiles"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-09T18:00:58Z", "updated_at": "2020-09-09T18:24:50Z", "closed_at": "2020-09-09T18:21:23Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/156", "body": "One of these is my fault, and the other is one I just happened to come across. They're harmless, but might as well be fixed.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/156/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 697179806, "node_id": "MDU6SXNzdWU2OTcxNzk4MDY=", "number": 157, "title": "sqlite-utils add-foreign-keys command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 2, "created_at": "2020-09-09T21:44:30Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-20T20:14:30Z", "author_association": "OWNER", "pull_request": null, "body": "Like `add-foreign-key` but can do multiple foreign keys at once. Inspired by https://github.com/simonw/calands-datasette/blob/99de39dd80a906f5c1f16724467b0cd55ba4ef36/build.sh which does this:\r\n\r\n```\r\nsqlite-utils add-foreign-key calands.db units_with_maps ACCESS_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_NAME\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_LEV\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps LAYER\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AGENCY\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_LEV\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps COUNTY\r\nsqlite-utils add-foreign-key calands.db units_with_maps DES_TP\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/157/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 697203800, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgzMTc1NTA5", "number": 158, "title": "Fix accidental mega long line in docs", "user": {"value": 167319, "label": "tomviner"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-09T22:31:23Z", "updated_at": "2020-09-16T06:21:43Z", "closed_at": "2020-09-16T06:21:43Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/158", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/158/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 702386948, "node_id": "MDU6SXNzdWU3MDIzODY5NDg=", "number": 159, "title": ".delete_where() does not auto-commit (unlike .insert() or .upsert())", "user": {"value": 11712349, "label": "spdkils"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-09-16T01:55:52Z", "updated_at": "2023-04-01T17:21:05Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "When you use the delete_where() function on a table, it never commits....\r\n\r\nIs that intentional?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/159/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 413867537, "node_id": "MDU6SXNzdWU0MTM4Njc1Mzc=", "number": 16, "title": "add_column() should support REFERENCES {other_table}({other_column})", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-02-24T21:00:45Z", "updated_at": "2019-05-29T05:17:59Z", "closed_at": "2019-05-29T04:56:18Z", "author_association": "OWNER", "pull_request": null, "body": "Related to #2 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 705190723, "node_id": "MDU6SXNzdWU3MDUxOTA3MjM=", "number": 160, "title": "table.enable_fts(..., replace=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 1, "created_at": "2020-09-20T21:36:23Z", "updated_at": "2020-09-24T20:35:47Z", "closed_at": "2020-09-20T22:05:51Z", "author_association": "OWNER", "pull_request": null, "body": "I noticed that https://til.simonwillison.net/ search doesn't use porter stemming. I'd like to add that, but since [the build script](https://github.com/simonw/til/blob/9d3f0fca30e94df3970df52b0447907a077e4673/build_database.py) always operates on an existing database (to avoid re-rendering markdown and re-building image thumbnails) I'd like it to only add porter stemming if it's not there already.\r\n\r\nSo I'd like to be able to say \"set up FTS to look like this, and fix it if it doesn't\".\r\n\r\nI think the neatest way to do that is with a `replace=True` argument to `.enable_fts()`, for consistency with `def .create_view(self, name, sql, replace=True)`.\r\n\r\nSo the `replace=True` argument would check and see if the configured FTS exists already with the correct options (columns, stemming, triggers) - and if any of those are incorrect it would call `.disable_fts()` and then create a new FTS configuration with the correct options.\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/160/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 705975133, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkwNjA3OTQ5", "number": 161, "title": "table.transform() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 13, "created_at": "2020-09-21T23:16:59Z", "updated_at": "2020-09-22T07:48:24Z", "closed_at": "2020-09-22T04:20:02Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/161", "body": "Refs #114\r\n\r\n- [x] Ability to change the primary key\r\n- [x] Support for changing default value for columns\r\n- [x] Support for changing `NOT NULL` status of columns\r\n- [x] Support for copying existing foreign keys and removing them\r\n- Support for `conversions=` parameter\r\n- [x] Detailed documentation\r\n- [x] `PRAGMA foreign_keys` stuff", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/161/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 705995722, "node_id": "MDU6SXNzdWU3MDU5OTU3MjI=", "number": 162, "title": "A decorator for registering custom SQL functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-22T00:18:32Z", "updated_at": "2020-09-22T00:40:44Z", "closed_at": "2020-09-22T00:32:17Z", "author_association": "OWNER", "pull_request": null, "body": "Syntactic sugar for `db.conn.create_function` - it would work something like this:\r\n\r\n```python\r\ndb = sqlite_utils.Database(\"mydb.db\")\r\n\r\n@db.register_function\r\ndef scramble(text):\r\n chars = list(text)\r\n random.shuffle(chars)\r\n return \"\".join(chars)\r\n```\r\nThe decorator would inspect the function to find its name and arity (number of arguments). Having run the above you could then do:\r\n```python\r\ndb.execute(\"select scramble('hello')\").fetchall()\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/162/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706001517, "node_id": "MDU6SXNzdWU3MDYwMDE1MTc=", "number": 163, "title": "Idea: conversions= could take Python functions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-22T00:37:12Z", "updated_at": "2021-12-20T00:56:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now you use `conversions=` like this:\r\n\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": \"upper(?)\"})\r\n```\r\nHow about if you could optionally provide a Python function (or a lambda) like this?\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": lambda s: s.upper()})\r\n```\r\nThis would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 706017416, "node_id": "MDU6SXNzdWU3MDYwMTc0MTY=", "number": 164, "title": "sqlite-utils transform sub-command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 4, "created_at": "2020-09-22T01:32:20Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-22T07:48:05Z", "author_association": "OWNER", "pull_request": null, "body": "The `.transform()` method in #114 warrants an equivalent CLI tool.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/164/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706091046, "node_id": "MDU6SXNzdWU3MDYwOTEwNDY=", "number": 165, "title": "Make .transform() a keyword arguments only function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 0, "created_at": "2020-09-22T05:37:29Z", "updated_at": "2020-09-24T20:35:47Z", "closed_at": "2020-09-22T06:39:12Z", "author_association": "OWNER", "pull_request": null, "body": "And rename the first argument from `columns=` to `types=`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/165/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706092617, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkwNzAzMTcz", "number": 166, "title": "Keyword only arguments for transform()", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-22T05:41:44Z", "updated_at": "2020-09-22T06:39:11Z", "closed_at": "2020-09-22T06:39:11Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/166", "body": "Refs #165", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/166/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 706098005, "node_id": "MDU6SXNzdWU3MDYwOTgwMDU=", "number": 167, "title": "Review the foreign key pragma stuff", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 1, "created_at": "2020-09-22T05:55:20Z", "updated_at": "2020-09-23T00:13:02Z", "closed_at": "2020-09-23T00:13:02Z", "author_association": "OWNER", "pull_request": null, "body": "> It is not possible to enable or disable foreign key constraints in the middle of a multi-statement transaction (when SQLite is not in autocommit mode). Attempting to do so does not return an error; it simply has no effect.\r\n\r\nhttps://sqlite.org/foreignkeys.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/167/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706167456, "node_id": "MDU6SXNzdWU3MDYxNjc0NTY=", "number": 168, "title": "Automate (as much as possible) updates published to Homebrew", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-22T08:08:37Z", "updated_at": "2020-11-09T07:43:30Z", "closed_at": "2020-11-09T07:43:30Z", "author_association": "OWNER", "pull_request": null, "body": "I'd like to get new `sqlite-utils` (and Datasette) releases submitted to Homebrew as painlessly as possible.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/168/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706757891, "node_id": "MDU6SXNzdWU3MDY3NTc4OTE=", "number": 169, "title": "Progress bar for \"sqlite-utils extract\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 0, "created_at": "2020-09-22T23:40:21Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-23T00:02:40Z", "author_association": "OWNER", "pull_request": null, "body": "> Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.\r\n>\r\n> The operations are full table scans so calculating progress shouldn't be too difficult.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/169/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413868452, "node_id": "MDU6SXNzdWU0MTM4Njg0NTI=", "number": 17, "title": "Improve and document foreign_keys=... argument to insert/create/etc", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2019-02-24T21:09:11Z", "updated_at": "2019-02-24T23:45:48Z", "closed_at": "2019-02-24T23:45:48Z", "author_association": "OWNER", "pull_request": null, "body": "The `foreign_keys=` argument to `table.insert_all()` and friends can be used to specify foreign key relationships that should be created.\r\n\r\nIt is not yet documented. It also requires you to specify the SQLite type of each column, even though this can be detected by introspecting the referenced table:\r\n\r\n cols = [c for c in self.db[other_table].columns if c.name == other_column]\r\n cols[0].type\r\n\r\nRelates to #2 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706768798, "node_id": "MDU6SXNzdWU3MDY3Njg3OTg=", "number": 170, "title": "Release notes for 2.20", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 1, "created_at": "2020-09-23T00:13:22Z", "updated_at": "2020-09-23T00:31:25Z", "closed_at": "2020-09-23T00:31:25Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/compare/2.19...b8e004", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/170/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707407567, "node_id": "MDU6SXNzdWU3MDc0MDc1Njc=", "number": 171, "title": "Idea: transitive closure tables for tree structures", "user": {"value": 649467, "label": "mhalle"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-23T14:17:33Z", "updated_at": "2020-10-22T04:38:35Z", "closed_at": "2020-10-22T04:07:14Z", "author_association": "NONE", "pull_request": null, "body": "I just read that sqlite has a transitive closure table extension using a virtual table in order to represent trees:\r\n\r\nhttps://charlesleifer.com/blog/querying-tree-structures-in-sqlite-using-python-and-the-transitive-closure-extension/\r\n\r\nEven without this extension, though, a util to build a transitive closure table would allow trees to be queried easily. Since it relies on self-referential foreign keys, the relationships might even be able to be automatically detected. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/171/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707427200, "node_id": "MDU6SXNzdWU3MDc0MjcyMDA=", "number": 172, "title": "Improve performance of extract operations", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-09-23T14:40:50Z", "updated_at": "2020-09-24T15:43:57Z", "closed_at": "2020-09-24T15:43:57Z", "author_association": "OWNER", "pull_request": null, "body": "This command took about 12 minutes (against a 150MB file with 680,000 rows in it):\r\n```\r\nsqlite-utils extract salaries.db salaries \\\r\n 'Organization Group Code' 'Organization Group' \\\r\n --table 'organization_groups' \\\r\n --fk-column 'organization_group_id' \\\r\n --rename 'Organization Group Code' code \\\r\n --rename 'Organization Group' name\r\n```\r\nI'm pretty confident we can do better than that.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/172/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707478649, "node_id": "MDU6SXNzdWU3MDc0Nzg2NDk=", "number": 173, "title": "Progress bar for sqlite-utils insert", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-09-23T15:43:56Z", "updated_at": "2021-11-01T08:42:24Z", "closed_at": "2020-10-27T18:16:04Z", "author_association": "OWNER", "pull_request": null, "body": "It would be nice if `sqlite-utils insert` had a progress bar, for when it's churning through huge CSV files.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/173/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707944044, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkyMjU3NDA1", "number": 174, "title": "Much, much faster extract() implementation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-09-24T07:52:31Z", "updated_at": "2020-09-24T15:44:00Z", "closed_at": "2020-09-24T15:43:56Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/174", "body": "Takes my test down from ten minutes to four seconds. Refs #172.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/174/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 708261775, "node_id": "MDU6SXNzdWU3MDgyNjE3NzU=", "number": 175, "title": "Add docs for .transform(column_order=)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-09-24T15:19:04Z", "updated_at": "2020-09-24T20:35:48Z", "closed_at": "2020-09-24T16:00:56Z", "author_association": "OWNER", "pull_request": null, "body": "> Need to update docs for `.transform()` now that `column_order=` is available.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/174#discussion_r494403327_\r\n\r\nMaybe also add this as an option to `sqlite-utils transform` - since reordering columns is actually a pretty nice capability.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/175/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 708293114, "node_id": "MDU6SXNzdWU3MDgyOTMxMTQ=", "number": 176, "title": "sqlite-utils transform column order option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-24T16:01:21Z", "updated_at": "2020-09-24T20:34:51Z", "closed_at": "2020-09-24T16:11:59Z", "author_association": "OWNER", "pull_request": null, "body": "Split from #175", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/176/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 708301810, "node_id": "MDU6SXNzdWU3MDgzMDE4MTA=", "number": 177, "title": "Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-24T16:13:50Z", "updated_at": "2020-09-24T20:35:03Z", "closed_at": "2020-09-24T16:19:13Z", "author_association": "OWNER", "pull_request": null, "body": "These both currently require you to provide three strings, for `column`, `other_table`, `other_column`.\r\n\r\nJust providing `column` should be enough information.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/177/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709043182, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkzMTYyNzY3", "number": 178, "title": "Update README.md", "user": {"value": 19921, "label": "shakeel"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-25T15:52:11Z", "updated_at": "2020-10-01T14:18:30Z", "closed_at": "2020-09-30T20:29:28Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/178", "body": "The `sqlite-utils insert releases.db releases - --pk` is missing the pk field name, added ` \"id\"` to fix it.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/178/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 709577625, "node_id": "MDU6SXNzdWU3MDk1Nzc2MjU=", "number": 179, "title": "sqlite-utils transform/insert --detect-types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-26T17:28:55Z", "updated_at": "2021-06-19T03:36:16Z", "closed_at": "2021-06-19T03:36:05Z", "author_association": "OWNER", "pull_request": null, "body": "Idea from https://github.com/simonw/datasette-edit-tables/issues/13 - provide Python utility methods and accompanying CLI options for detecting the likely types of TEXT columns.\r\n\r\nSo if you have a text column that actually contained exclusively integer string values, it can let you know and let you run transform against it.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/179/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413871266, "node_id": "MDU6SXNzdWU0MTM4NzEyNjY=", "number": 18, "title": ".insert/.upsert/.insert_all/.upsert_all should add missing columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 2, "created_at": "2019-02-24T21:36:11Z", "updated_at": "2019-05-25T00:42:11Z", "closed_at": "2019-05-25T00:42:11Z", "author_association": "OWNER", "pull_request": null, "body": "This is a larger change, but it would be incredibly useful: if you attempt to insert or update a document with a field that does not currently exist in the underlying table, sqlite-utils should add the appropriate column for you.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709861194, "node_id": "MDU6SXNzdWU3MDk4NjExOTQ=", "number": 180, "title": "Try running some tests using Hypothesis", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-28T01:11:30Z", "updated_at": "2020-10-19T04:51:55Z", "closed_at": "2020-10-19T04:51:55Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by this Twitter conversation: https://twitter.com/simonw/status/1310386009465479168", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/180/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709920027, "node_id": "MDU6SXNzdWU3MDk5MjAwMjc=", "number": 181, "title": "pk=[\"id\"] should have same effect as pk=\"id\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-28T04:28:07Z", "updated_at": "2020-10-14T21:59:47Z", "closed_at": "2020-10-14T21:59:47Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\nIn [11]: db['one'].insert({\"id\": 1, \"name\": \"oentuh\"}, pk=\"id\")\r\nOut[11]:
\r\n\r\nIn [12]: db['two'].insert({\"id\": 1, \"name\": \"oentuh\"}, pk=[\"id\"])\r\nOut[12]:
\r\n\r\nIn [13]: db['one'].schema\r\nOut[13]: 'CREATE TABLE [one] (\\n [id] INTEGER PRIMARY KEY,\\n [name] TEXT\\n)'\r\n\r\nIn [14]: db['two'].schema\r\nOut[14]: 'CREATE TABLE [two] (\\n [id] INTEGER,\\n [name] TEXT\\n)'\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/181/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 711649325, "node_id": "MDU6SXNzdWU3MTE2NDkzMjU=", "number": 182, "title": "Better handling of encodings other than utf-8 for \"sqlite-utils insert\"", "user": {"value": 765871, "label": "kaihendry"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-09-30T05:43:48Z", "updated_at": "2020-10-16T17:20:41Z", "closed_at": "2020-10-16T17:18:52Z", "author_association": "NONE", "pull_request": null, "body": "Makefile:\r\n```\r\ndata.db:\r\n curl -O http://maps.natalian.org/data.txt\r\n go run csv-write.go > data.csv\r\n sqlite-utils insert data.db travels data.csv --csv\r\n\r\nclean:\r\n rm data*\r\n```\r\n[csv-write.go](https://gist.github.com/kaihendry/dff2442de20d73f900026d13bf7a11d9)\r\n\r\n\r\nError message is:\r\n\r\n```\r\nsqlite-utils insert data.db travels data.csv --csv\r\nTraceback (most recent call last):\r\n File \"/home/hendry/.local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 614, in insert\r\n insert_upsert_implementation(\r\n File \"/home/hendry/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 553, in insert_upsert_implementation\r\n headers = next(reader)\r\n File \"/usr/lib/python3.8/codecs.py\", line 322, in decode\r\n (result, consumed) = self._buffer_decode(data, self.errors, final)\r\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 1234: invalid continuation byte\r\nmake: *** [Makefile:4: data.db] Error 1\r\n[hendry@t14s datasette-map]$ sqlite-utils --version\r\nsqlite-utils, version 2.19\r\n```\r\n\r\nLittle bit surprised if Go is spewing out bad Unicode, but I'm not sure how to grok `position 1234`..\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/182/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 712316959, "node_id": "MDU6SXNzdWU3MTIzMTY5NTk=", "number": 183, "title": "Try out GitHub code scanning", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-30T22:16:14Z", "updated_at": "2020-09-30T22:23:44Z", "closed_at": "2020-09-30T22:23:44Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.blog/2020-09-30-code-scanning-is-now-available/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/183/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 716955793, "node_id": "MDExOlB1bGxSZXF1ZXN0NDk5NjAzMzU5", "number": 184, "title": "Test against Python 3.9", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-08T01:37:05Z", "updated_at": "2020-10-08T01:44:06Z", "closed_at": "2020-10-08T01:44:06Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/184", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/184/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 718952107, "node_id": "MDU6SXNzdWU3MTg5NTIxMDc=", "number": 185, "title": "Use db[table] consistently in documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-11T23:39:12Z", "updated_at": "2020-10-12T00:13:41Z", "closed_at": "2020-10-12T00:13:41Z", "author_association": "OWNER", "pull_request": null, "body": "The Python docs have a bunch of examples like this: https://sqlite-utils.readthedocs.io/en/stable/python-api.html\r\n```python\r\ndogs.enable_fts([\"name\", \"twitter\"], create_triggers=True)\r\n```\r\nThis would be easier for people to understand if it looked like this instead:\r\n```python\r\ndb[\"dogs\"].enable_fts([\"name\", \"twitter\"], create_triggers=True)\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/185/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 722816436, "node_id": "MDU6SXNzdWU3MjI4MTY0MzY=", "number": 186, "title": ".extract() shouldn't extract null values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-10-16T02:41:08Z", "updated_at": "2021-08-12T12:32:14Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This almost works, but it creates a rogue `type` record with a value of None.\r\n```\r\nIn [1]: import sqlite_utils\r\nIn [2]: db = sqlite_utils.Database(memory=True)\r\nIn [5]: db[\"creatures\"].insert_all([\r\n {\"id\": 1, \"name\": \"Simon\", \"type\": None},\r\n {\"id\": 2, \"name\": \"Natalie\", \"type\": None},\r\n {\"id\": 3, \"name\": \"Cleo\", \"type\": \"dog\"}], pk=\"id\")\r\nOut[5]:
\r\nIn [7]: db[\"creatures\"].extract(\"type\")\r\nOut[7]:
\r\nIn [8]: list(db[\"creatures\"].rows)\r\nOut[8]: \r\n[{'id': 1, 'name': 'Simon', 'type_id': None},\r\n {'id': 2, 'name': 'Natalie', 'type_id': None},\r\n {'id': 3, 'name': 'Cleo', 'type_id': 2}]\r\nIn [9]: db[\"type\"]\r\nOut[9]:
\r\nIn [10]: list(db[\"type\"].rows)\r\nOut[10]: [{'id': 1, 'type': None}, {'id': 2, 'type': 'dog'}]\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/186/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 723460107, "node_id": "MDU6SXNzdWU3MjM0NjAxMDc=", "number": 187, "title": "Maybe: Utility method / CLI tool for initializing SpatiaLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-16T19:04:03Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-16T19:15:13Z", "author_association": "OWNER", "pull_request": null, "body": "> I think this should initialize SpatiaLite against the current database if it has not been initialized already.\r\n>\r\n> Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/187/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 723708310, "node_id": "MDU6SXNzdWU3MjM3MDgzMTA=", "number": 188, "title": "About loading spatialite", "user": {"value": 30607, "label": "aborruso"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-10-17T08:47:02Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-17T08:52:58Z", "author_association": "NONE", "pull_request": null, "body": "Hi @simonw ,\r\nIf I run\r\n\r\n```\r\nsqlite3\r\n.load /usr/local/lib/mod_spatialite.so\r\nselect spatialite_version();\r\n```\r\n\r\nI have `5.0.0`.\r\n\r\n![image](https://user-images.githubusercontent.com/30607/96332706-d8cd3300-1065-11eb-906b-daf99963198e.png)\r\n\r\n\r\nIf I run\r\n\r\n```\r\nsqlite-utils :memory: \"select spatialite_version()\" --load-extension=spatialite\r\n```\r\n\r\nI have\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/home/aborruso/.local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 936, in query\r\n _load_extensions(db, load_extension)\r\n File \"/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 1326, in _load_extensions\r\n db.conn.load_extension(ext)\r\nTypeError: argument 1 must be str, not None\r\n```\r\n\r\nHow to load properly spatialite extension in sqlite-utils?\r\n\r\nThank you very muc", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/188/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 729818242, "node_id": "MDExOlB1bGxSZXF1ZXN0NTEwMjM1OTA5", "number": 189, "title": "Allow iterables other than Lists in m2m records", "user": {"value": 35681, "label": "adamwolf"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-10-26T18:47:44Z", "updated_at": "2020-10-27T16:28:37Z", "closed_at": "2020-10-27T16:24:21Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/189", "body": "I was playing around with sqlite-utils, creating a Roam Research dogsheep-style importer for Datasette, and ran into a slight snag.\r\n\r\nI wanted to use a generator to add an order column in an importer. It looked something like:\r\n\r\n```\r\ndef order_generator(iterable, attr=None):\r\n if attr is None:\r\n attr = \"order\"\r\n order: int = 0\r\n\r\n for i in iterable:\r\n i[attr] = order\r\n order += 1\r\n yield i\r\n```\r\n\r\nWhen I used this with `insert_all` and other things, it worked fine--but it didn't work as the `records` argument to `m2m`. I dug into it, and sqlite-utils is explicitly checking if the records argument is a list or a tuple. I flipped the check upside down, and now it checks if the argument is a mapping. If it's a mapping, it wraps it in a list, otherwise it leaves it alone.\r\n\r\n(I get that it might not really make sense to put the order column on the second table. I changed my import schema a bit, and no longer have a real example, but maybe this change still makes sense.)\r\n\r\nThe automated tests still pass, but I did not add any new ones.\r\n\r\nLet me know what you think! I'm really loving Datasette and its ecosystem; thanks for everything!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/189/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}