{"id": 347058326, "node_id": "MDExOlB1bGxSZXF1ZXN0MjA1NzcwOTk2", "number": 1, "title": "Make .indexes compatible with older SQLite versions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2018-08-02T15:17:05Z", "updated_at": "2018-08-02T15:17:30Z", "closed_at": "2018-08-02T15:17:30Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/1", "body": "Older SQLite versions return a different set of columns from the PRAGMA we are using.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/1/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 601358649, "node_id": "MDU6SXNzdWU2MDEzNTg2NDk=", "number": 100, "title": "Mechanism for forcing column-type, over-riding auto-detection", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-04-16T19:12:52Z", "updated_at": "2020-04-17T23:53:32Z", "closed_at": "2020-04-17T23:53:32Z", "author_association": "OWNER", "pull_request": null, "body": "As seen in https://github.com/dogsheep/github-to-sqlite/issues/27#issuecomment-614843406 - there's a problem where you insert a record with a `None` value for a column and that column is created as `TEXT` - but actually you intended it to be an `INT` (as later examples will demonstrate).\r\n\r\nSome kind of mechanism for over-riding the detected types of columns would be useful here.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/100/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 601392318, "node_id": "MDU6SXNzdWU2MDEzOTIzMTg=", "number": 101, "title": "README should include an example of CLI data insertion", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-16T19:45:37Z", "updated_at": "2020-04-17T23:59:49Z", "closed_at": "2020-04-17T23:59:49Z", "author_association": "OWNER", "pull_request": null, "body": "Maybe using `curl` from the GitHub API.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/101/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 602569315, "node_id": "MDU6SXNzdWU2MDI1NjkzMTU=", "number": 102, "title": "Can't store an array or dictionary containing a bytes value", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-04-18T22:49:21Z", "updated_at": "2020-05-01T20:45:45Z", "closed_at": "2020-05-01T20:45:45Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\nIn [1]: import sqlite_utils \r\n\r\nIn [2]: db = sqlite_utils.Database(memory=True) \r\n\r\nIn [3]: db[\"t\"].insert({\"id\": 1, \"data\": {\"foo\": b\"bytes\"}}) \r\n---------------------------------------------------------------------------\r\nTypeError Traceback (most recent call last)\r\n in \r\n----> 1 db[\"t\"].insert({\"id\": 1, \"data\": {\"foo\": b\"bytes\"}})\r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert(self, record, pk, foreign_keys, column_order, not_null, defaults, hash_id, alter, ignore, replace, extracts, conversions, columns)\r\n 950 extracts=extracts,\r\n 951 conversions=conversions,\r\n--> 952 columns=columns,\r\n 953 )\r\n 954 \r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, conversions, columns, upsert)\r\n 1052 for key in all_columns:\r\n 1053 value = jsonify_if_needed(\r\n-> 1054 record.get(key, None if key != hash_id else _hash(record))\r\n 1055 )\r\n 1056 if key in extracts:\r\n\r\n~/Dropbox/Development/sqlite-utils/sqlite_utils/db.py in jsonify_if_needed(value)\r\n 1318 def jsonify_if_needed(value):\r\n 1319 if isinstance(value, (dict, list, tuple)):\r\n-> 1320 return json.dumps(value)\r\n 1321 elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):\r\n 1322 return value.isoformat()\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)\r\n 229 cls is None and indent is None and separators is None and\r\n 230 default is None and not sort_keys and not kw):\r\n--> 231 return _default_encoder.encode(obj)\r\n 232 if cls is None:\r\n 233 cls = JSONEncoder\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in encode(self, o)\r\n 197 # exceptions aren't as detailed. The list call should be roughly\r\n 198 # equivalent to the PySequence_Fast that ''.join() would do.\r\n--> 199 chunks = self.iterencode(o, _one_shot=True)\r\n 200 if not isinstance(chunks, (list, tuple)):\r\n 201 chunks = list(chunks)\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in iterencode(self, o, _one_shot)\r\n 255 self.key_separator, self.item_separator, self.sort_keys,\r\n 256 self.skipkeys, _one_shot)\r\n--> 257 return _iterencode(o, 0)\r\n 258 \r\n 259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,\r\n\r\n/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py in default(self, o)\r\n 177 \r\n 178 \"\"\"\r\n--> 179 raise TypeError(f'Object of type {o.__class__.__name__} '\r\n 180 f'is not JSON serializable')\r\n 181 \r\n\r\nTypeError: Object of type bytes is not JSON serializable\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/102/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610853393, "node_id": "MDU6SXNzdWU2MTA4NTMzOTM=", "number": 104, "title": "--schema option to \"sqlite-utils tables\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-05-01T16:55:49Z", "updated_at": "2020-05-01T17:12:37Z", "closed_at": "2020-05-01T17:12:37Z", "author_association": "OWNER", "pull_request": null, "body": "Adds output showing the table schema.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/104/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 610853576, "node_id": "MDU6SXNzdWU2MTA4NTM1NzY=", "number": 105, "title": "\"sqlite-utils views\" command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-01T16:56:11Z", "updated_at": "2020-05-01T20:40:07Z", "closed_at": "2020-05-01T20:38:36Z", "author_association": "OWNER", "pull_request": null, "body": "Similar to `sqlite-utils tables`. See also #104.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/105/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611216862, "node_id": "MDU6SXNzdWU2MTEyMTY4NjI=", "number": 106, "title": "create_view(..., ignore=True, replace=True) parameters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-02T15:45:21Z", "updated_at": "2020-05-02T16:04:51Z", "closed_at": "2020-05-02T16:02:10Z", "author_association": "OWNER", "pull_request": null, "body": "Two new parameters which specify what should happen if the view already exists. I want this for https://github.com/dogsheep/github-to-sqlite/issues/37\r\n\r\nHere's the current `create_view()` implementation:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/b4d953d3ccef28bb81cea40ca165a647b59971fa/sqlite_utils/db.py#L325-L332\r\n\r\n`ignore=True` will not do anything if the view exists already.\r\n\r\n`replace=True` will drop and redefine the view - but only if its SQL definition differs, otherwise it will be left alone.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/106/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611222968, "node_id": "MDU6SXNzdWU2MTEyMjI5Njg=", "number": 107, "title": "sqlite-utils create-view CLI command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-02T16:15:13Z", "updated_at": "2020-05-03T15:36:58Z", "closed_at": "2020-05-03T15:36:37Z", "author_association": "OWNER", "pull_request": null, "body": "Can go with #27 - `sqlite-utils create-table`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/107/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 611326701, "node_id": "MDU6SXNzdWU2MTEzMjY3MDE=", "number": 108, "title": "Documentation unit tests for CLI commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-03T03:58:42Z", "updated_at": "2020-05-03T04:13:57Z", "closed_at": "2020-05-03T04:13:57Z", "author_association": "OWNER", "pull_request": null, "body": "Have a test that ensures all CLI commands are documented.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/108/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 612658444, "node_id": "MDU6SXNzdWU2MTI2NTg0NDQ=", "number": 109, "title": "table.create_index(..., ignore=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-05-05T14:44:21Z", "updated_at": "2020-05-05T14:46:53Z", "closed_at": "2020-05-05T14:46:53Z", "author_association": "OWNER", "pull_request": null, "body": "Option to silently do nothing if the index already exists.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/109/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413740684, "node_id": "MDU6SXNzdWU0MTM3NDA2ODQ=", "number": 11, "title": "Detect numpy types when creating tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2019-02-23T21:09:35Z", "updated_at": "2019-02-24T04:02:20Z", "closed_at": "2019-02-24T04:02:20Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by #8", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/11/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 615477131, "node_id": "MDU6SXNzdWU2MTU0NzcxMzE=", "number": 111, "title": "sqlite-utils drop-table and drop-view commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-10T21:10:42Z", "updated_at": "2020-05-11T01:58:36Z", "closed_at": "2020-05-11T00:44:26Z", "author_association": "OWNER", "pull_request": null, "body": "Would be useful to be able to drop views and tables from the CLI.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/111/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 616271236, "node_id": "MDU6SXNzdWU2MTYyNzEyMzY=", "number": 112, "title": "add_foreign_key(...., ignore=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 4, "created_at": "2020-05-12T00:24:00Z", "updated_at": "2020-09-20T22:17:34Z", "closed_at": "2020-09-20T22:17:34Z", "author_association": "OWNER", "pull_request": null, "body": "When using this library I often find myself wanting to \"add this foreign key, but only if it doesn't exist yet\". The `ignore=True` parameter is increasingly being used for this else where in the library (e.g. in `create_view()`).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/112/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 621286870, "node_id": "MDU6SXNzdWU2MjEyODY4NzA=", "number": 113, "title": "Syntactic sugar for ATTACH DATABASE", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-05-19T21:10:00Z", "updated_at": "2021-02-19T05:09:12Z", "closed_at": "2021-02-19T04:56:36Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/lang_attach.html\r\n\r\nMaybe something like this:\r\n```python\r\ndb.attach(\"other_db\", \"other_db.db\")\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/113/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 621989740, "node_id": "MDU6SXNzdWU2MjE5ODk3NDA=", "number": 114, "title": "table.transform() method for advanced alter table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 26, "created_at": "2020-05-20T18:20:46Z", "updated_at": "2020-09-22T07:51:37Z", "closed_at": "2020-09-22T04:20:02Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite's `ALTER TABLE` can only do the following:\r\n\r\n* Rename a table\r\n* Rename a column\r\n* Add a column\r\n\r\nNotably, it cannot drop columns - so tricks like \"add a float version of this text column, populate it, then drop the old one and rename\" won't work.\r\n\r\nThe docs here https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes describe a way of implementing full alters safely within a transaction, but it's fiddly.\r\n\r\n1. Create new table\r\n2. Copy data\r\n3. Drop old table\r\n4. Rename new into old \r\n\r\nIt would be great if `sqlite-utils` provided an abstraction to help make these kinds of changes safely.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/114/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 637889964, "node_id": "MDU6SXNzdWU2Mzc4ODk5NjQ=", "number": 115, "title": "Ability to execute insert/update statements with the CLI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-06-12T17:01:17Z", "updated_at": "2020-06-12T17:51:11Z", "closed_at": "2020-06-12T17:41:10Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n$ sqlite-utils github.db \"update stars set starred_at = ''\"\r\nTraceback (most recent call last):\r\n File \"/Users/simon/.local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 673, in query\r\n headers = [c[0] for c in cursor.description]\r\nTypeError: 'NoneType' object is not iterable\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/115/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 644122661, "node_id": "MDU6SXNzdWU2NDQxMjI2NjE=", "number": 116, "title": "Documentation for table.pks introspection property", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-06-23T20:27:24Z", "updated_at": "2020-06-23T21:21:33Z", "closed_at": "2020-06-23T21:03:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L535-L540", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/116/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 644161221, "node_id": "MDU6SXNzdWU2NDQxNjEyMjE=", "number": 117, "title": "Support for compound (composite) foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-06-23T21:33:42Z", "updated_at": "2020-06-23T21:40:31Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "It turns out SQLite supports composite foreign keys: https://www.sqlite.org/foreignkeys.html#fk_composite\r\n\r\nTheir example looks like this:\r\n```sql\r\nCREATE TABLE album(\r\n albumartist TEXT,\r\n albumname TEXT,\r\n albumcover BINARY,\r\n PRIMARY KEY(albumartist, albumname)\r\n);\r\n\r\nCREATE TABLE song(\r\n songid INTEGER,\r\n songartist TEXT,\r\n songalbum TEXT,\r\n songname TEXT,\r\n FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname)\r\n);\r\n```\r\n\r\nHere's what that looks like in sqlite-utils:\r\n\r\n```\r\nIn [1]: import sqlite_utils \r\n\r\nIn [2]: import sqlite3 \r\n\r\nIn [3]: conn = sqlite3.connect(\":memory:\") \r\n\r\nIn [4]: conn \r\nOut[4]: \r\n\r\nIn [5]: conn.executescript(\"\"\" \r\n ...: CREATE TABLE album( \r\n ...: albumartist TEXT, \r\n ...: albumname TEXT, \r\n ...: albumcover BINARY, \r\n ...: PRIMARY KEY(albumartist, albumname) \r\n ...: ); \r\n ...: \r\n ...: CREATE TABLE song( \r\n ...: songid INTEGER, \r\n ...: songartist TEXT, \r\n ...: songalbum TEXT, \r\n ...: songname TEXT, \r\n ...: FOREIGN KEY(songartist, songalbum) REFERENCES album(albumartist, albumname) \r\n ...: ); \r\n ...: \"\"\") \r\nOut[5]: \r\n\r\nIn [6]: db = sqlite_utils.Database(conn) \r\n\r\nIn [7]: db.tables \r\nOut[7]: \r\n[,\r\n
]\r\n\r\nIn [8]: db.tables[0].foreign_keys \r\nOut[8]: []\r\n\r\nIn [9]: db.tables[1].foreign_keys \r\nOut[9]: \r\n[ForeignKey(table='song', column='songartist', other_table='album', other_column='albumartist'),\r\n ForeignKey(table='song', column='songalbum', other_table='album', other_column='albumname')]\r\n```\r\nThe table appears to have two separate foreign keys, when actually it has a single compound composite foreign key.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/117/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 652700770, "node_id": "MDU6SXNzdWU2NTI3MDA3NzA=", "number": 119, "title": "Ability to remove a foreign key", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-07T22:31:37Z", "updated_at": "2020-09-24T20:36:59Z", "closed_at": "2020-09-24T20:36:59Z", "author_association": "OWNER", "pull_request": null, "body": "Useful if you add one but make a mistake and need to undo it without recreating the database from scratch.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/119/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413778585, "node_id": "MDExOlB1bGxSZXF1ZXN0MjU1NjU4MTEy", "number": 12, "title": "Support for numpy types, closes #11", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-02-24T03:57:32Z", "updated_at": "2019-02-24T04:02:20Z", "closed_at": "2019-02-24T04:02:20Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/12", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/12/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 652961907, "node_id": "MDU6SXNzdWU2NTI5NjE5MDc=", "number": 121, "title": "Improved (and better documented) support for transactions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-08T04:56:51Z", "updated_at": "2020-09-24T20:36:46Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393_\r\n\r\nWe should put some thought into how this library supports and encourages smart use of transactions.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/121/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 665700495, "node_id": "MDU6SXNzdWU2NjU3MDA0OTU=", "number": 122, "title": "CLI utility for inserting binary files into SQLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-07-26T03:27:39Z", "updated_at": "2020-07-27T07:10:41Z", "closed_at": "2020-07-27T07:09:03Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite BLOB columns can store entire binary files. The challenge is inserting them, since they don't neatly fit into JSON objects.\r\n\r\nIt would be great if the `sqlite-utils` CLI had a trick for helping with this.\r\n\r\nInspired by https://github.com/simonw/datasette-media/issues/14", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/122/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665701216, "node_id": "MDU6SXNzdWU2NjU3MDEyMTY=", "number": 123, "title": "--raw option for outputting binary content", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-07-26T03:35:39Z", "updated_at": "2020-07-26T16:44:11Z", "closed_at": "2020-07-26T16:44:11Z", "author_association": "OWNER", "pull_request": null, "body": "Related to the `insert-files` work in #122 - it should be easy to get binary data back out of the database again.\r\n\r\nOne way to do that could be:\r\n\r\n sqlite-utils files.db \"select content from files where key = 'foo.jpg'\" --raw\r\n\r\nThe `--raw` option would cause just the contents of the first column to be output directly to stdout.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/123/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665802405, "node_id": "MDU6SXNzdWU2NjU4MDI0MDU=", "number": 124, "title": "sqlite-utils query should support named parameters", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-26T15:25:10Z", "updated_at": "2020-07-30T22:57:51Z", "closed_at": "2020-07-27T03:53:58Z", "author_association": "OWNER", "pull_request": null, "body": "To help out with escaping - so you can run this:\r\n\r\n sqlite-utils query \"insert into foo (blah) values (:blah)\" --param blah `something here`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665817570, "node_id": "MDU6SXNzdWU2NjU4MTc1NzA=", "number": 125, "title": "Output binary columns in \"sqlite-utils query\" JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-07-26T16:47:02Z", "updated_at": "2020-07-27T00:49:41Z", "closed_at": "2020-07-27T00:48:45Z", "author_association": "OWNER", "pull_request": null, "body": "You get an error if you try to run a query that returns data from a BLOB.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/125/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 665819048, "node_id": "MDU6SXNzdWU2NjU4MTkwNDg=", "number": 126, "title": "Ability to insert binary data on the CLI using JSON", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-26T16:54:14Z", "updated_at": "2020-07-27T04:00:33Z", "closed_at": "2020-07-27T03:59:45Z", "author_association": "OWNER", "pull_request": null, "body": "> I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says \"these columns are base64 encoded, store the decoded data in a BLOB\".\r\n>\r\n> That would solve inserting binary data using JSON too.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/126/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 666040390, "node_id": "MDU6SXNzdWU2NjYwNDAzOTA=", "number": 127, "title": "Ability to insert files piped to insert-files stdin", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-07-27T07:09:33Z", "updated_at": "2020-07-30T03:08:52Z", "closed_at": "2020-07-30T03:08:18Z", "author_association": "OWNER", "pull_request": null, "body": "> Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.\r\n>\r\n> cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif\r\n>\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/127/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 666639051, "node_id": "MDU6SXNzdWU2NjY2MzkwNTE=", "number": 128, "title": "Support UUID and memoryview types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-07-27T23:08:34Z", "updated_at": "2020-07-30T01:10:43Z", "closed_at": "2020-07-30T01:10:43Z", "author_association": "OWNER", "pull_request": null, "body": "`psycopg2` can return data from PostgreSQL as `uuid.UUID` or `memoryview` objects. These should to be supported by `sqlite-utils` - mainly for https://github.com/simonw/db-to-sqlite", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/128/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 668308777, "node_id": "MDU6SXNzdWU2NjgzMDg3Nzc=", "number": 129, "title": "\"insert-files --sqlar\" for creating SQLite archives", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-07-30T02:28:29Z", "updated_at": "2020-07-30T22:41:01Z", "closed_at": "2020-07-30T22:40:55Z", "author_association": "OWNER", "pull_request": null, "body": "A `--sqlar` option could cause `insert-files` to behave in the same way as SQLite's own sqlar mechanism.\r\n\r\nhttps://www.sqlite.org/sqlar.html and https://sqlite.org/sqlar/doc/trunk/README.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/129/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413779210, "node_id": "MDU6SXNzdWU0MTM3NzkyMTA=", "number": 13, "title": "Ability to automatically create IDs from content hash of row", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2019-02-24T04:07:08Z", "updated_at": "2019-02-24T04:36:48Z", "closed_at": "2019-02-24T04:36:48Z", "author_association": "OWNER", "pull_request": null, "body": "Sometimes when you are importing data the underlying source provides records without IDs that can be uniquely identified by their contents.\r\n\r\nA utility mechanism for calculating a sha1 hash of the contents and using that as a unique ID would be useful.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/13/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 671130371, "node_id": "MDU6SXNzdWU2NzExMzAzNzE=", "number": 130, "title": "Support tokenize option for FTS", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-01T19:27:22Z", "updated_at": "2020-08-01T20:51:28Z", "closed_at": "2020-08-01T20:51:14Z", "author_association": "OWNER", "pull_request": null, "body": "FTS5 supports things like porter stemming using a `tokenize=` option:\r\n\r\nhttps://www.sqlite.org/fts5.html#tokenizers\r\n\r\nSomething like this in code:\r\n```\r\n CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} (\r\n {columns},\r\n tokenize='porter',\r\n content=[{table}]\r\n );\r\n```\r\nI tried this out just now and it worked exactly as expected.\r\n\r\nSo... `db[table].enable_fts(...) should accept a 'tokenize=` argument, and `sqlite-utils enable-fts ...` should support a `--tokenize` option.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/130/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 675753042, "node_id": "MDU6SXNzdWU2NzU3NTMwNDI=", "number": 131, "title": "sqlite-utils insert: options for column types", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-09T18:59:11Z", "updated_at": "2022-03-15T13:21:42Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `insert` command currently results in string types for every column - at least when used against CSV or TSV inputs.\r\n\r\nIt would be useful if you could do the following:\r\n\r\n- automatically detects the column types based on eg the first 1000 records\r\n- explicitly state the rule for specific columns\r\n\r\n`--detect-types` could work for the former - or it could do that by default and allow opt-out using `--no-detect-types`\r\n\r\nFor specific columns maybe this:\r\n\r\n sqlite-utils insert db.db images images.tsv \\\r\n --tsv \\\r\n -c id int \\\r\n -c score float", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/131/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 675839512, "node_id": "MDU6SXNzdWU2NzU4Mzk1MTI=", "number": 132, "title": "Features for enabling and disabling WAL mode", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2020-08-10T03:25:44Z", "updated_at": "2020-08-10T18:59:35Z", "closed_at": "2020-08-10T18:59:35Z", "author_association": "OWNER", "pull_request": null, "body": "I finally figured out how to enable WAL - turns out it's a property of the database file itself: https://github.com/simonw/til/blob/master/sqlite/enabling-wal-mode.md", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/132/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 677839979, "node_id": "MDU6SXNzdWU2Nzc4Mzk5Nzk=", "number": 133, "title": "Release a sdist to PyPI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-12T16:55:09Z", "updated_at": "2020-08-12T17:05:06Z", "closed_at": "2020-08-12T17:05:06Z", "author_association": "OWNER", "pull_request": null, "body": "https://pypi.org/project/sqlite-utils/#files currently just has a wheel. I need this to package for homebrew: https://github.com/simonw/homebrew-datasette/issues/10", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/133/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683804172, "node_id": "MDU6SXNzdWU2ODM4MDQxNzI=", "number": 134, "title": "--load-extension option for sqlite-utils query", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-08-21T20:12:42Z", "updated_at": "2020-08-21T21:06:26Z", "closed_at": "2020-08-21T20:54:19Z", "author_association": "OWNER", "pull_request": null, "body": "I got this error:\r\n```\r\n% sqlite-utils calands.db 'create table superunits_with_maps_view_concrete as select * from superunits_with_maps_view'\r\nTraceback (most recent call last):\r\n...\r\n cursor = db.conn.execute(sql, dict(param))\r\nsqlite3.OperationalError: no such function: AsGeoJSON\r\n```\r\nA `--load-extension=/usr/local/lib/mod_spatialite.dylib` option (imitating the same option for Datasette) would help.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/134/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683805434, "node_id": "MDU6SXNzdWU2ODM4MDU0MzQ=", "number": 135, "title": "Code for finding SpatiaLite in the usual locations", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-21T20:15:34Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-08-21T20:30:13Z", "author_association": "OWNER", "pull_request": null, "body": "I built this for `shapefile-to-sqlite` but it would be useful in `sqlite-utils` too:\r\n\r\nhttps://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L16-L19\r\n\r\n```python\r\nSPATIALITE_PATHS = (\r\n \"/usr/lib/x86_64-linux-gnu/mod_spatialite.so\",\r\n \"/usr/local/lib/mod_spatialite.dylib\",\r\n)\r\n```\r\n\r\nhttps://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L105-L109\r\n\r\n```python\r\ndef find_spatialite():\r\n for path in SPATIALITE_PATHS:\r\n if os.path.exists(path):\r\n return path\r\n return None\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/135/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683812642, "node_id": "MDU6SXNzdWU2ODM4MTI2NDI=", "number": 136, "title": "--load-extension=spatialite shortcut option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-08-21T20:31:25Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-16T19:14:32Z", "author_association": "OWNER", "pull_request": null, "body": "In conjunction with #135 - this would do the same thing as `--load-extension=path-to-spatialite` (see #134)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/136/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 683830416, "node_id": "MDU6SXNzdWU2ODM4MzA0MTY=", "number": 137, "title": "--load-extension for other sqlite-utils commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-21T21:12:56Z", "updated_at": "2020-10-16T19:14:32Z", "closed_at": "2020-10-16T19:14:32Z", "author_association": "OWNER", "pull_request": null, "body": "e.g. for this:\r\n```\r\ncalands-datasette % sqlite-utils tables calands.db --counts\r\n[{\"table\": \"spatial_ref_sys\", \"count\": 4924},\r\n {\"table\": \"spatialite_history\", \"count\": 14},\r\n {\"table\": \"sqlite_sequence\", \"count\": 1},\r\n {\"table\": \"geometry_columns\", \"count\": 2},\r\n {\"table\": \"spatial_ref_sys_aux\", \"count\": 4873},\r\n {\"table\": \"views_geometry_columns\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns\", \"count\": 0},\r\n {\"table\": \"geometry_columns_statistics\", \"count\": 2},\r\n {\"table\": \"views_geometry_columns_statistics\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_statistics\", \"count\": 0},\r\n {\"table\": \"geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"views_geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_field_infos\", \"count\": 0},\r\n {\"table\": \"geometry_columns_time\", \"count\": 2},\r\n {\"table\": \"geometry_columns_auth\", \"count\": 2},\r\n {\"table\": \"views_geometry_columns_auth\", \"count\": 0},\r\n {\"table\": \"virts_geometry_columns_auth\", \"count\": 0},\r\nTraceback (most recent call last):\r\n File \"/usr/local/bin/sqlite-utils\", line 8, in \r\n sys.exit(cli())\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 829, in __call__\r\n return self.main(*args, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 782, in main\r\n rv = self.invoke(ctx)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 1259, in invoke\r\n return _process_result(sub_ctx.command.invoke(sub_ctx))\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 1066, in invoke\r\n return ctx.invoke(self.callback, **ctx.params)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py\", line 610, in invoke\r\n return callback(*args, **kwargs)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 143, in tables\r\n for line in output_rows(_iter(), headers, nl, arrays, json_cols):\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 922, in output_rows\r\n for row, next_row in itertools.zip_longest(current_iter, next_iter):\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py\", line 123, in _iter\r\n row.append(db[name].count)\r\n File \"/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/db.py\", line 458, in count\r\n return self.db.conn.execute(\r\nsqlite3.OperationalError: no such module: VirtualSpatialIndex\r\n```\r\nThe `tables` command could take `--load-extension` too - as could `rows` and other similar commands.\r\n\r\nFollow-on from #134 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/137/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 684118950, "node_id": "MDU6SXNzdWU2ODQxMTg5NTA=", "number": 138, "title": "extracts= doesn't configure foreign keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-23T05:21:15Z", "updated_at": "2020-09-24T22:47:01Z", "closed_at": "2020-09-24T22:46:52Z", "author_association": "OWNER", "pull_request": null, "body": "In using `extracts=` for `shapefiles-to-sqlite` in https://github.com/simonw/shapefile-to-sqlite/issues/9 I've run into a couple of pretty serious flaws:\r\n\r\n- The columns in the original table are still `TEXT` even when the foreign key they are supposed to reference is an `INTEGER` - which means Datasette foreign key features don't actually work\r\n- Those foreign key relationships aren't setup automatically - creating them is left as an exercise for the developer", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/138/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413842611, "node_id": "MDU6SXNzdWU0MTM4NDI2MTE=", "number": 14, "title": "Utilities for adding indexes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2019-02-24T16:57:28Z", "updated_at": "2019-02-24T19:11:28Z", "closed_at": "2019-02-24T19:11:28Z", "author_association": "OWNER", "pull_request": null, "body": "Both in the Python API and the CLI tool. For the CLI tool this should work:\r\n\r\n $ sqlite-utils create-index mydb.db mytable col1 col2\r\n\r\nThis will create a compound index across col1 and col2. The name of the index will be automatically chosen unless you use the `--name=...` option.\r\n\r\nSupport a `--unique` option too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/14/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688351054, "node_id": "MDU6SXNzdWU2ODgzNTEwNTQ=", "number": 140, "title": "Idea: insert-files mechanism for adding extra columns with fixed values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T20:57:36Z", "updated_at": "2022-03-20T19:45:45Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Say for example you want to populate a `file_type` column with the value `gif`. That could work like this:\r\n\r\n```\r\nsqlite-utils insert-files gifs.db images *.gif \\\r\n -c path -c md5 -c last_modified:mtime \\\r\n -c file_type:text:gif --pk=path\r\n```\r\nSo a column defined as a `text` column with a value that follows a second colon.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 688352145, "node_id": "MDU6SXNzdWU2ODgzNTIxNDU=", "number": 141, "title": "insert-files support for compressed values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-08-28T20:59:46Z", "updated_at": "2020-09-24T20:36:08Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "The `sqlar` format supports this, it would be useful if `insert-files` could support this too.\r\n\r\nhttps://www.sqlite.org/sqlar.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/141/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 688389933, "node_id": "MDU6SXNzdWU2ODgzODk5MzM=", "number": 143, "title": "Move to GitHub Actions CI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-08-28T22:34:11Z", "updated_at": "2020-08-28T22:41:35Z", "closed_at": "2020-08-28T22:41:35Z", "author_association": "OWNER", "pull_request": null, "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/143/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 688395275, "node_id": "MDU6SXNzdWU2ODgzOTUyNzU=", "number": 144, "title": "Run some tests against numpy", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-08-28T22:53:00Z", "updated_at": "2020-08-28T22:57:05Z", "closed_at": "2020-08-28T22:57:04Z", "author_association": "OWNER", "pull_request": null, "body": "Accidentally removed in #143:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/d7d3f962861ef32c5ead8f514c8756f5b6f7c4a0/.travis.yml#L18-L19", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/144/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695276328, "node_id": "MDU6SXNzdWU2OTUyNzYzMjg=", "number": 148, "title": "More attractive indentation of created FTS table schema", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-07T16:49:30Z", "updated_at": "2020-09-07T18:12:50Z", "closed_at": "2020-09-07T18:12:50Z", "author_association": "OWNER", "pull_request": null, "body": "On https://github-to-sqlite.dogsheep.net/github/licenses_fts the create table SQL is displayed as:\r\n```sql\r\nCREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (\r\n [name],\r\n content=[licenses]\r\n );\r\n```\r\nIt would be more aesthetically pleasing if it looked like this:\r\n```sql\r\nCREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (\r\n [name],\r\n content=[licenses]\r\n);\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/148/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695319258, "node_id": "MDU6SXNzdWU2OTUzMTkyNTg=", "number": 149, "title": "FTS table with 7 rows has _fts_docsize table with 9,141 rows", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2020-09-07T18:06:16Z", "updated_at": "2020-09-07T21:16:34Z", "closed_at": "2020-09-07T21:16:34Z", "author_association": "OWNER", "pull_request": null, "body": "I'm seeing a weird issue with some of the SQLite databases that I am using with the FTS5 module.\r\n\r\nI have a database with a `licenses` table that contains 7 rows: \r\n\r\nThe FTS table also has 7 rows: \r\n\r\nSomehow the accompanying `licenses_fts_docsize` shadow table now has 9,141 rows in it! \r\n\r\nAnd `licenses_fts_data` has 41 rows - should I expect that to have 7 rows? \r\n\r\nI have a hunch that it might be a problem with the triggers. These are the triggers that are updating that FTS table: \r\n\r\n| type | name | tbl_name | rootpage | sql |\r\n| --- | --- | --- | --- | --- |\r\n| trigger | licenses_ai | licenses | 0 | `CREATE TRIGGER [licenses_ai] AFTER INSERT ON [licenses] BEGIN INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |\r\n| trigger | licenses_ad | licenses | 0 | `CREATE TRIGGER [licenses_ad] AFTER DELETE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); END` |\r\n| trigger | licenses_au | licenses | 0 | `CREATE TRIGGER [licenses_au] AFTER UPDATE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/149/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413857257, "node_id": "MDU6SXNzdWU0MTM4NTcyNTc=", "number": 15, "title": "Ability to add columns to tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2019-02-24T19:20:51Z", "updated_at": "2019-02-24T20:04:40Z", "closed_at": "2019-02-24T20:04:40Z", "author_association": "OWNER", "pull_request": null, "body": "Makes sense to do this before foreign keys in #2\r\n\r\nPython:\r\n\r\n db[\"table\"].add_column(\"new_column\", int)\r\n\r\nCLI:\r\n\r\n $ sqlite-utils add-column table new_column INTEGER\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/15/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695359607, "node_id": "MDU6SXNzdWU2OTUzNTk2MDc=", "number": 150, "title": "Feature for tracing SQL queries", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-07T19:43:08Z", "updated_at": "2020-09-07T21:57:01Z", "closed_at": "2020-09-07T21:57:01Z", "author_association": "OWNER", "pull_request": null, "body": "Debugging `sqlite-utils` when something weird happens (e.g. #149) can be a bit tricky since it runs a bunch of different SQL statements behind the scenes.\r\n\r\nAn optional \"tracing\" mechanism for seeing what SQL is being executed would be useful.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/150/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695360889, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgxNjE2NzA0", "number": 151, "title": "Tracer mechanism for seeing underlying SQL", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-07T19:46:43Z", "updated_at": "2020-09-07T21:57:00Z", "closed_at": "2020-09-07T21:57:00Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/151", "body": "Refs #150. Needs tests and documentation, including for the new `db.execute()` and `db.executescript()` methods.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/151/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 695376054, "node_id": "MDU6SXNzdWU2OTUzNzYwNTQ=", "number": 152, "title": "Turn on recursive_triggers by default", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T20:26:36Z", "updated_at": "2020-09-07T21:17:48Z", "closed_at": "2020-09-07T20:45:14Z", "author_association": "OWNER", "pull_request": null, "body": "https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:\r\n\r\n> Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.\r\n\r\nSo I think the fix for the complex issue in #149 is to turn on `recursive_triggers` globally by default for `sqlite-utils`.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/152/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695377804, "node_id": "MDU6SXNzdWU2OTUzNzc4MDQ=", "number": 153, "title": "table.optimize() should delete junk rows from *_fts_docsize", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-09-07T20:31:09Z", "updated_at": "2020-09-24T20:35:46Z", "closed_at": "2020-09-07T21:16:33Z", "author_association": "OWNER", "pull_request": null, "body": "> The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL:\r\n> ```sql\r\n> DELETE FROM [licenses_fts_docsize] WHERE id NOT IN (\r\n> SELECT rowid FROM [licenses_fts]);\r\n> ```\r\n> I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/153/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 695441530, "node_id": "MDU6SXNzdWU2OTU0NDE1MzA=", "number": 154, "title": "OperationalError: cannot change into wal mode from within a transaction", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-07T23:42:44Z", "updated_at": "2020-09-07T23:47:10Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "I'm getting this error when running:\r\n\r\n sqlite-utils enable-wal beta.db\r\n\r\n`OperationalError: cannot change into wal mode from within a transaction`\r\n\r\nI'm worried that maybe that's because of this new code from #152:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 696045581, "node_id": "MDU6SXNzdWU2OTYwNDU1ODE=", "number": 155, "title": "rebuild-fts command and table.rebuild_fts() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-08T17:19:26Z", "updated_at": "2020-09-24T20:35:46Z", "closed_at": "2020-09-08T23:16:10Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite.org/forum/forumpost/fa777fff86\r\n\r\n> Easiest thing would be to run a 'rebuild' to rebuild the FTS index from scratch based on the contents of the content table. i.e.\r\n>\r\n> INSERT INTO licenses_fts(licenses_fts) VALUES('rebuild');\r\n>\r\n> https://www.sqlite.org/fts5.html#the_rebuild_command", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/155/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 697179806, "node_id": "MDU6SXNzdWU2OTcxNzk4MDY=", "number": 157, "title": "sqlite-utils add-foreign-keys command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 2, "created_at": "2020-09-09T21:44:30Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-20T20:14:30Z", "author_association": "OWNER", "pull_request": null, "body": "Like `add-foreign-key` but can do multiple foreign keys at once. Inspired by https://github.com/simonw/calands-datasette/blob/99de39dd80a906f5c1f16724467b0cd55ba4ef36/build.sh which does this:\r\n\r\n```\r\nsqlite-utils add-foreign-key calands.db units_with_maps ACCESS_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_NAME\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_LEV\r\nsqlite-utils add-foreign-key calands.db units_with_maps AGNCY_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps LAYER\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AGENCY\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_LEV\r\nsqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_TYP\r\nsqlite-utils add-foreign-key calands.db units_with_maps COUNTY\r\nsqlite-utils add-foreign-key calands.db units_with_maps DES_TP\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/157/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413867537, "node_id": "MDU6SXNzdWU0MTM4Njc1Mzc=", "number": 16, "title": "add_column() should support REFERENCES {other_table}({other_column})", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-02-24T21:00:45Z", "updated_at": "2019-05-29T05:17:59Z", "closed_at": "2019-05-29T04:56:18Z", "author_association": "OWNER", "pull_request": null, "body": "Related to #2 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/16/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 705190723, "node_id": "MDU6SXNzdWU3MDUxOTA3MjM=", "number": 160, "title": "table.enable_fts(..., replace=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5896742, "label": "2.19"}, "comments": 1, "created_at": "2020-09-20T21:36:23Z", "updated_at": "2020-09-24T20:35:47Z", "closed_at": "2020-09-20T22:05:51Z", "author_association": "OWNER", "pull_request": null, "body": "I noticed that https://til.simonwillison.net/ search doesn't use porter stemming. I'd like to add that, but since [the build script](https://github.com/simonw/til/blob/9d3f0fca30e94df3970df52b0447907a077e4673/build_database.py) always operates on an existing database (to avoid re-rendering markdown and re-building image thumbnails) I'd like it to only add porter stemming if it's not there already.\r\n\r\nSo I'd like to be able to say \"set up FTS to look like this, and fix it if it doesn't\".\r\n\r\nI think the neatest way to do that is with a `replace=True` argument to `.enable_fts()`, for consistency with `def .create_view(self, name, sql, replace=True)`.\r\n\r\nSo the `replace=True` argument would check and see if the configured FTS exists already with the correct options (columns, stemming, triggers) - and if any of those are incorrect it would call `.disable_fts()` and then create a new FTS configuration with the correct options.\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/160/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 705975133, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkwNjA3OTQ5", "number": 161, "title": "table.transform() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 13, "created_at": "2020-09-21T23:16:59Z", "updated_at": "2020-09-22T07:48:24Z", "closed_at": "2020-09-22T04:20:02Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/161", "body": "Refs #114\r\n\r\n- [x] Ability to change the primary key\r\n- [x] Support for changing default value for columns\r\n- [x] Support for changing `NOT NULL` status of columns\r\n- [x] Support for copying existing foreign keys and removing them\r\n- Support for `conversions=` parameter\r\n- [x] Detailed documentation\r\n- [x] `PRAGMA foreign_keys` stuff", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/161/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 705995722, "node_id": "MDU6SXNzdWU3MDU5OTU3MjI=", "number": 162, "title": "A decorator for registering custom SQL functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-22T00:18:32Z", "updated_at": "2020-09-22T00:40:44Z", "closed_at": "2020-09-22T00:32:17Z", "author_association": "OWNER", "pull_request": null, "body": "Syntactic sugar for `db.conn.create_function` - it would work something like this:\r\n\r\n```python\r\ndb = sqlite_utils.Database(\"mydb.db\")\r\n\r\n@db.register_function\r\ndef scramble(text):\r\n chars = list(text)\r\n random.shuffle(chars)\r\n return \"\".join(chars)\r\n```\r\nThe decorator would inspect the function to find its name and arity (number of arguments). Having run the above you could then do:\r\n```python\r\ndb.execute(\"select scramble('hello')\").fetchall()\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/162/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706001517, "node_id": "MDU6SXNzdWU3MDYwMDE1MTc=", "number": 163, "title": "Idea: conversions= could take Python functions", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-22T00:37:12Z", "updated_at": "2021-12-20T00:56:52Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "Right now you use `conversions=` like this:\r\n\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": \"upper(?)\"})\r\n```\r\nHow about if you could optionally provide a Python function (or a lambda) like this?\r\n```python\r\ndb[\"example\"].insert({\r\n \"name\": \"The Bigfoot Discovery Museum\"\r\n}, conversions={\"name\": lambda s: s.upper()})\r\n```\r\nThis would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 706017416, "node_id": "MDU6SXNzdWU3MDYwMTc0MTY=", "number": 164, "title": "sqlite-utils transform sub-command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 4, "created_at": "2020-09-22T01:32:20Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-22T07:48:05Z", "author_association": "OWNER", "pull_request": null, "body": "The `.transform()` method in #114 warrants an equivalent CLI tool.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/164/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706091046, "node_id": "MDU6SXNzdWU3MDYwOTEwNDY=", "number": 165, "title": "Make .transform() a keyword arguments only function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 0, "created_at": "2020-09-22T05:37:29Z", "updated_at": "2020-09-24T20:35:47Z", "closed_at": "2020-09-22T06:39:12Z", "author_association": "OWNER", "pull_request": null, "body": "And rename the first argument from `columns=` to `types=`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/165/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706092617, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkwNzAzMTcz", "number": 166, "title": "Keyword only arguments for transform()", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-09-22T05:41:44Z", "updated_at": "2020-09-22T06:39:11Z", "closed_at": "2020-09-22T06:39:11Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/166", "body": "Refs #165", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/166/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 706098005, "node_id": "MDU6SXNzdWU3MDYwOTgwMDU=", "number": 167, "title": "Review the foreign key pragma stuff", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 1, "created_at": "2020-09-22T05:55:20Z", "updated_at": "2020-09-23T00:13:02Z", "closed_at": "2020-09-23T00:13:02Z", "author_association": "OWNER", "pull_request": null, "body": "> It is not possible to enable or disable foreign key constraints in the middle of a multi-statement transaction (when SQLite is not in autocommit mode). Attempting to do so does not return an error; it simply has no effect.\r\n\r\nhttps://sqlite.org/foreignkeys.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/167/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706167456, "node_id": "MDU6SXNzdWU3MDYxNjc0NTY=", "number": 168, "title": "Automate (as much as possible) updates published to Homebrew", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-22T08:08:37Z", "updated_at": "2020-11-09T07:43:30Z", "closed_at": "2020-11-09T07:43:30Z", "author_association": "OWNER", "pull_request": null, "body": "I'd like to get new `sqlite-utils` (and Datasette) releases submitted to Homebrew as painlessly as possible.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/168/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706757891, "node_id": "MDU6SXNzdWU3MDY3NTc4OTE=", "number": 169, "title": "Progress bar for \"sqlite-utils extract\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 0, "created_at": "2020-09-22T23:40:21Z", "updated_at": "2020-09-24T20:34:50Z", "closed_at": "2020-09-23T00:02:40Z", "author_association": "OWNER", "pull_request": null, "body": "> Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.\r\n>\r\n> The operations are full table scans so calculating progress shouldn't be too difficult.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/169/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413868452, "node_id": "MDU6SXNzdWU0MTM4Njg0NTI=", "number": 17, "title": "Improve and document foreign_keys=... argument to insert/create/etc", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2019-02-24T21:09:11Z", "updated_at": "2019-02-24T23:45:48Z", "closed_at": "2019-02-24T23:45:48Z", "author_association": "OWNER", "pull_request": null, "body": "The `foreign_keys=` argument to `table.insert_all()` and friends can be used to specify foreign key relationships that should be created.\r\n\r\nIt is not yet documented. It also requires you to specify the SQLite type of each column, even though this can be detected by introspecting the referenced table:\r\n\r\n cols = [c for c in self.db[other_table].columns if c.name == other_column]\r\n cols[0].type\r\n\r\nRelates to #2 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/17/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 706768798, "node_id": "MDU6SXNzdWU3MDY3Njg3OTg=", "number": 170, "title": "Release notes for 2.20", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 5897911, "label": "2.20"}, "comments": 1, "created_at": "2020-09-23T00:13:22Z", "updated_at": "2020-09-23T00:31:25Z", "closed_at": "2020-09-23T00:31:25Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/compare/2.19...b8e004", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/170/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707427200, "node_id": "MDU6SXNzdWU3MDc0MjcyMDA=", "number": 172, "title": "Improve performance of extract operations", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2020-09-23T14:40:50Z", "updated_at": "2020-09-24T15:43:57Z", "closed_at": "2020-09-24T15:43:57Z", "author_association": "OWNER", "pull_request": null, "body": "This command took about 12 minutes (against a 150MB file with 680,000 rows in it):\r\n```\r\nsqlite-utils extract salaries.db salaries \\\r\n 'Organization Group Code' 'Organization Group' \\\r\n --table 'organization_groups' \\\r\n --fk-column 'organization_group_id' \\\r\n --rename 'Organization Group Code' code \\\r\n --rename 'Organization Group' name\r\n```\r\nI'm pretty confident we can do better than that.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/172/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707478649, "node_id": "MDU6SXNzdWU3MDc0Nzg2NDk=", "number": 173, "title": "Progress bar for sqlite-utils insert", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-09-23T15:43:56Z", "updated_at": "2021-11-01T08:42:24Z", "closed_at": "2020-10-27T18:16:04Z", "author_association": "OWNER", "pull_request": null, "body": "It would be nice if `sqlite-utils insert` had a progress bar, for when it's churning through huge CSV files.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/173/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 707944044, "node_id": "MDExOlB1bGxSZXF1ZXN0NDkyMjU3NDA1", "number": 174, "title": "Much, much faster extract() implementation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-09-24T07:52:31Z", "updated_at": "2020-09-24T15:44:00Z", "closed_at": "2020-09-24T15:43:56Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/174", "body": "Takes my test down from ten minutes to four seconds. Refs #172.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/174/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 708261775, "node_id": "MDU6SXNzdWU3MDgyNjE3NzU=", "number": 175, "title": "Add docs for .transform(column_order=)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-09-24T15:19:04Z", "updated_at": "2020-09-24T20:35:48Z", "closed_at": "2020-09-24T16:00:56Z", "author_association": "OWNER", "pull_request": null, "body": "> Need to update docs for `.transform()` now that `column_order=` is available.\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/174#discussion_r494403327_\r\n\r\nMaybe also add this as an option to `sqlite-utils transform` - since reordering columns is actually a pretty nice capability.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/175/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 708293114, "node_id": "MDU6SXNzdWU3MDgyOTMxMTQ=", "number": 176, "title": "sqlite-utils transform column order option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-09-24T16:01:21Z", "updated_at": "2020-09-24T20:34:51Z", "closed_at": "2020-09-24T16:11:59Z", "author_association": "OWNER", "pull_request": null, "body": "Split from #175", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/176/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 708301810, "node_id": "MDU6SXNzdWU3MDgzMDE4MTA=", "number": 177, "title": "Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-24T16:13:50Z", "updated_at": "2020-09-24T20:35:03Z", "closed_at": "2020-09-24T16:19:13Z", "author_association": "OWNER", "pull_request": null, "body": "These both currently require you to provide three strings, for `column`, `other_table`, `other_column`.\r\n\r\nJust providing `column` should be enough information.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/177/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709577625, "node_id": "MDU6SXNzdWU3MDk1Nzc2MjU=", "number": 179, "title": "sqlite-utils transform/insert --detect-types", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-09-26T17:28:55Z", "updated_at": "2021-06-19T03:36:16Z", "closed_at": "2021-06-19T03:36:05Z", "author_association": "OWNER", "pull_request": null, "body": "Idea from https://github.com/simonw/datasette-edit-tables/issues/13 - provide Python utility methods and accompanying CLI options for detecting the likely types of TEXT columns.\r\n\r\nSo if you have a text column that actually contained exclusively integer string values, it can let you know and let you run transform against it.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/179/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 413871266, "node_id": "MDU6SXNzdWU0MTM4NzEyNjY=", "number": 18, "title": ".insert/.upsert/.insert_all/.upsert_all should add missing columns", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 2, "created_at": "2019-02-24T21:36:11Z", "updated_at": "2019-05-25T00:42:11Z", "closed_at": "2019-05-25T00:42:11Z", "author_association": "OWNER", "pull_request": null, "body": "This is a larger change, but it would be incredibly useful: if you attempt to insert or update a document with a field that does not currently exist in the underlying table, sqlite-utils should add the appropriate column for you.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/18/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709861194, "node_id": "MDU6SXNzdWU3MDk4NjExOTQ=", "number": 180, "title": "Try running some tests using Hypothesis", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-28T01:11:30Z", "updated_at": "2020-10-19T04:51:55Z", "closed_at": "2020-10-19T04:51:55Z", "author_association": "OWNER", "pull_request": null, "body": "Inspired by this Twitter conversation: https://twitter.com/simonw/status/1310386009465479168", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/180/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 709920027, "node_id": "MDU6SXNzdWU3MDk5MjAwMjc=", "number": 181, "title": "pk=[\"id\"] should have same effect as pk=\"id\"", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-28T04:28:07Z", "updated_at": "2020-10-14T21:59:47Z", "closed_at": "2020-10-14T21:59:47Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\nIn [11]: db['one'].insert({\"id\": 1, \"name\": \"oentuh\"}, pk=\"id\")\r\nOut[11]:
\r\n\r\nIn [12]: db['two'].insert({\"id\": 1, \"name\": \"oentuh\"}, pk=[\"id\"])\r\nOut[12]:
\r\n\r\nIn [13]: db['one'].schema\r\nOut[13]: 'CREATE TABLE [one] (\\n [id] INTEGER PRIMARY KEY,\\n [name] TEXT\\n)'\r\n\r\nIn [14]: db['two'].schema\r\nOut[14]: 'CREATE TABLE [two] (\\n [id] INTEGER,\\n [name] TEXT\\n)'\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/181/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 712316959, "node_id": "MDU6SXNzdWU3MTIzMTY5NTk=", "number": 183, "title": "Try out GitHub code scanning", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-09-30T22:16:14Z", "updated_at": "2020-09-30T22:23:44Z", "closed_at": "2020-09-30T22:23:44Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.blog/2020-09-30-code-scanning-is-now-available/", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/183/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 716955793, "node_id": "MDExOlB1bGxSZXF1ZXN0NDk5NjAzMzU5", "number": 184, "title": "Test against Python 3.9", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-08T01:37:05Z", "updated_at": "2020-10-08T01:44:06Z", "closed_at": "2020-10-08T01:44:06Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/184", "body": "", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/184/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 718952107, "node_id": "MDU6SXNzdWU3MTg5NTIxMDc=", "number": 185, "title": "Use db[table] consistently in documentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-11T23:39:12Z", "updated_at": "2020-10-12T00:13:41Z", "closed_at": "2020-10-12T00:13:41Z", "author_association": "OWNER", "pull_request": null, "body": "The Python docs have a bunch of examples like this: https://sqlite-utils.readthedocs.io/en/stable/python-api.html\r\n```python\r\ndogs.enable_fts([\"name\", \"twitter\"], create_triggers=True)\r\n```\r\nThis would be easier for people to understand if it looked like this instead:\r\n```python\r\ndb[\"dogs\"].enable_fts([\"name\", \"twitter\"], create_triggers=True)\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/185/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 722816436, "node_id": "MDU6SXNzdWU3MjI4MTY0MzY=", "number": 186, "title": ".extract() shouldn't extract null values", "user": {"value": 9599, "label": "simonw"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2020-10-16T02:41:08Z", "updated_at": "2021-08-12T12:32:14Z", "closed_at": null, "author_association": "OWNER", "pull_request": null, "body": "This almost works, but it creates a rogue `type` record with a value of None.\r\n```\r\nIn [1]: import sqlite_utils\r\nIn [2]: db = sqlite_utils.Database(memory=True)\r\nIn [5]: db[\"creatures\"].insert_all([\r\n {\"id\": 1, \"name\": \"Simon\", \"type\": None},\r\n {\"id\": 2, \"name\": \"Natalie\", \"type\": None},\r\n {\"id\": 3, \"name\": \"Cleo\", \"type\": \"dog\"}], pk=\"id\")\r\nOut[5]:
\r\nIn [7]: db[\"creatures\"].extract(\"type\")\r\nOut[7]:
\r\nIn [8]: list(db[\"creatures\"].rows)\r\nOut[8]: \r\n[{'id': 1, 'name': 'Simon', 'type_id': None},\r\n {'id': 2, 'name': 'Natalie', 'type_id': None},\r\n {'id': 3, 'name': 'Cleo', 'type_id': 2}]\r\nIn [9]: db[\"type\"]\r\nOut[9]:
\r\nIn [10]: list(db[\"type\"].rows)\r\nOut[10]: [{'id': 1, 'type': None}, {'id': 2, 'type': 'dog'}]\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/186/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null} {"id": 723460107, "node_id": "MDU6SXNzdWU3MjM0NjAxMDc=", "number": 187, "title": "Maybe: Utility method / CLI tool for initializing SpatiaLite", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-16T19:04:03Z", "updated_at": "2022-02-05T00:04:26Z", "closed_at": "2020-10-16T19:15:13Z", "author_association": "OWNER", "pull_request": null, "body": "> I think this should initialize SpatiaLite against the current database if it has not been initialized already.\r\n>\r\n> Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/187/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 432217625, "node_id": "MDU6SXNzdWU0MzIyMTc2MjU=", "number": 19, "title": "Incorrect help text for enable-fts command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 4348046, "label": "1.0"}, "comments": 0, "created_at": "2019-04-11T19:46:44Z", "updated_at": "2019-05-25T00:44:31Z", "closed_at": "2019-05-25T00:44:31Z", "author_association": "OWNER", "pull_request": null, "body": "I clearly copied-and-pasted this from the `tables` command without updating it:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/0b1af42ead3b3902347951180b3364ce1942da6e/sqlite_utils/cli.py#L216-L222", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/19/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 730693696, "node_id": "MDExOlB1bGxSZXF1ZXN0NTEwOTU2MTM0", "number": 190, "title": "Progress bar for sqlite-utils insert command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2020-10-27T18:08:53Z", "updated_at": "2020-10-27T18:16:03Z", "closed_at": "2020-10-27T18:16:03Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/190", "body": "Refs #173", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/190/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 731740458, "node_id": "MDU6SXNzdWU3MzE3NDA0NTg=", "number": 191, "title": "Idea: @db.register_function(deterministic=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-10-28T19:45:18Z", "updated_at": "2020-10-28T21:31:06Z", "closed_at": "2020-10-28T21:31:06Z", "author_association": "OWNER", "pull_request": null, "body": "Python 3.8 added a `deterministic` parameter to `db.create_function()`: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function\r\n\r\n`sqlite-utils` could expose this in the decorator, only actually applying it if the Python version supports it (using feature detection) - since nothing will break if it's not applied.\r\n\r\nhttps://sqlite-utils.readthedocs.io/en/stable/python-api.html#registering-custom-sql-functions\r\n\r\n```python\r\ndb = Database(memory=True)\r\n\r\n@db.register_function(deterministic=True)\r\ndef reverse_string(s):\r\n return \"\".join(reversed(list(s)))\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/191/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735532751, "node_id": "MDU6SXNzdWU3MzU1MzI3NTE=", "number": 192, "title": "sqlite-utils search command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 9, "created_at": "2020-11-03T18:07:59Z", "updated_at": "2020-11-08T17:07:01Z", "closed_at": "2020-11-08T17:07:01Z", "author_association": "OWNER", "pull_request": null, "body": "A command that knows how to run a search against a FTS enabled table and return results ranked by relevance.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/192/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735648209, "node_id": "MDU6SXNzdWU3MzU2NDgyMDk=", "number": 193, "title": "--tsv output format option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 0, "created_at": "2020-11-03T21:31:18Z", "updated_at": "2020-11-07T00:09:52Z", "closed_at": "2020-11-07T00:09:52Z", "author_association": "OWNER", "pull_request": null, "body": "We already support `--csv` for output, and the `insert` command accepts `--tsv`. The output format options should accept `--tsv` too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/193/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735650864, "node_id": "MDU6SXNzdWU3MzU2NTA4NjQ=", "number": 194, "title": "3.0 release with some minor breaking changes", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 3, "created_at": "2020-11-03T21:36:31Z", "updated_at": "2020-11-08T17:19:35Z", "closed_at": "2020-11-08T17:19:34Z", "author_association": "OWNER", "pull_request": null, "body": "While working on search (#192) I've spotted a few small changes I would like to make that would break backwards compatibility in minor ways, hence requiring a 3.x release.\r\n\r\n`db[table].search()` - I would like this to default to sorting by rank\r\n\r\nAlso I'd like to free up the `-c` and `-f` options for other purposes from the standard output formats here:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/43eae8b193d362f2b292df73e087ed6f10838144/sqlite_utils/cli.py#L48-L58\r\n\r\nI'd like `-f` to be used to indicate a full-text search column during an insert and `-c` to indicate a column (so you can specify which columns you want to output).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/194/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 735663855, "node_id": "MDExOlB1bGxSZXF1ZXN0NTE1MDE0ODgz", "number": 195, "title": "table.search() improvements plus sqlite-utils search command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2020-11-03T22:02:08Z", "updated_at": "2020-11-06T18:30:49Z", "closed_at": "2020-11-06T18:30:42Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/195", "body": "Refs #192. Still needs tests.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/195/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 736520310, "node_id": "MDU6SXNzdWU3MzY1MjAzMTA=", "number": 196, "title": "Introspect if table is FTS4 or FTS5", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 19, "created_at": "2020-11-05T00:45:50Z", "updated_at": "2020-11-05T03:54:07Z", "closed_at": "2020-11-05T03:54:07Z", "author_association": "OWNER", "pull_request": null, "body": "> I want `.search()` to work against both FTS5 and FTS4 tables - but sort by rank should only work for FTS5.\r\n>\r\n> This means I need to be able to introspect and tell if a table is FTS4 or FTS5.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/192#issuecomment-722054264_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/196/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 737153927, "node_id": "MDU6SXNzdWU3MzcxNTM5Mjc=", "number": 197, "title": "Rethink how table.search() method works", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 5, "created_at": "2020-11-05T18:04:34Z", "updated_at": "2020-11-08T17:07:37Z", "closed_at": "2020-11-08T17:07:37Z", "author_association": "OWNER", "pull_request": null, "body": "I need to improve this method to help build `sqlite-utils search` in #192 (PR is #195).\r\n\r\nThe challenge is deciding how it should handle sorting by relevance - especially since that is easy in FTS5 but not at all easy in FTS4.\r\n\r\n> Latest test failure:\r\n> ```\r\n> 114 ->\t assert [(\"racoons are biting trash pandas\", \"USA\", \"bar\")] == table.search(\r\n> 115 \t \"bite\", order=\"rowid\"\r\n> 116 \t )\r\n> 117 \t\r\n> 118 \t\r\n> 119 \tdef test_optimize_fts(fresh_db):\r\n> (Pdb) table.search(\"bite\")\r\n> [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)]\r\n> ```\r\n> The problem here is that the `table.search()` method now behaves differently for FTS4 v.s. FTS5 tables.\r\n> \r\n> With FTS4 you get back just the table columns.\r\n>\r\n> With FTS5 you also get back the `rowid` as the first column and the `rank` score as the last column.\r\n> \r\n> This is weird. It also makes me question whether having `.search()` return a list of tuples is the right API design.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/197/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 737476423, "node_id": "MDU6SXNzdWU3Mzc0NzY0MjM=", "number": 198, "title": "Support order by relevance against FTS4", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-11-06T05:36:31Z", "updated_at": "2020-11-06T18:30:44Z", "closed_at": "2020-11-06T18:30:44Z", "author_association": "OWNER", "pull_request": null, "body": "For #192 and #197 I've decided I want to be able to order by relevance in FTS4 as well as FTS5.\r\n\r\nThis means I need to port over my work on bm25() from https://github.com/simonw/sqlite-fts4 (since I don't want to add a full dependency).", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/198/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 737855731, "node_id": "MDU6SXNzdWU3Mzc4NTU3MzE=", "number": 199, "title": "@db.register_function(..., replace=False) to avoid double-registering custom functions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2020-11-06T15:39:21Z", "updated_at": "2020-11-06T18:30:44Z", "closed_at": "2020-11-06T18:30:44Z", "author_association": "OWNER", "pull_request": null, "body": "I'd like a mechanism to optionally avoid registering a custom function if it has already been registered.\r\n\r\nSQLite doesn't seem to offer a way to introspect registered custom functions so I'll need to track what has already been registered in `sqlite-utils` instead.\r\n\r\n> Should I register the custom `rank_bm25` SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/199/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 349850687, "node_id": "MDU6SXNzdWUzNDk4NTA2ODc=", "number": 2, "title": "Mechanism for adding foreign keys to an existing table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2018-08-12T22:50:56Z", "updated_at": "2019-02-24T21:34:41Z", "closed_at": "2019-02-24T21:34:41Z", "author_association": "OWNER", "pull_request": null, "body": "SQLite does not have ALTER TABLE support for adding new foreign keys... but it turns out it's possible to make these changes without having to duplicate the entire table by carefully running `UPDATE sqlite_master SET sql=... WHERE type='table' AND name='X';`\r\n\r\nHere's how Django does it: https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/db/backends/sqlite3/schema.py#L103-L125\r\n\r\nAnd here's the official documentation about this: https://sqlite.org/lang_altertable.html#otheralter (scroll to the very bottom of the page)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/2/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 738115165, "node_id": "MDU6SXNzdWU3MzgxMTUxNjU=", "number": 200, "title": "sqlite-utils rows -c option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 1, "created_at": "2020-11-07T00:22:12Z", "updated_at": "2020-11-07T00:28:48Z", "closed_at": "2020-11-07T00:28:47Z", "author_association": "OWNER", "pull_request": null, "body": "To let you specify the exact columns you want. Based on the `-c` option to `sqlite-utils search` in #192.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/200/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 738128913, "node_id": "MDU6SXNzdWU3MzgxMjg5MTM=", "number": 201, "title": ".search(columns=) and sqlite-utils search -c ... bug", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": {"value": 6079500, "label": "3.0"}, "comments": 1, "created_at": "2020-11-07T01:27:26Z", "updated_at": "2020-11-08T16:54:15Z", "closed_at": "2020-11-08T16:54:15Z", "author_association": "OWNER", "pull_request": null, "body": "Both `table.search(columns=)` and the `sqlite-utils search -c` option do not work as expected - they always return both the `rowid` and the `rank` columns even if those have not been requested.\r\n\r\nThis should be fixed before the 3.0 non-alpha release.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/201/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 738514367, "node_id": "MDU6SXNzdWU3Mzg1MTQzNjc=", "number": 202, "title": "sqlite-utils insert -f colname - for configuring full-text search", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2020-11-08T17:30:09Z", "updated_at": "2021-01-03T05:00:36Z", "closed_at": "2021-01-03T05:00:27Z", "author_association": "OWNER", "pull_request": null, "body": "A mechanism for specifying columns that should be configured for full-text search as part of the initial data import:\r\n\r\n sqlite-utils insert mydb.db articles articles.csv --csv -f title -f body", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/202/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 761915790, "node_id": "MDU6SXNzdWU3NjE5MTU3OTA=", "number": 206, "title": "sqlite-utils should suggest --csv if JSON parsing fails", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-12-11T05:17:56Z", "updated_at": "2021-10-30T15:52:17Z", "closed_at": "2021-01-03T18:42:22Z", "author_association": "OWNER", "pull_request": null, "body": "```\r\n~ % gsutil cat gs://ossf-criticality-score/python_top_200.csv | sqlite-utils insert /tmp/crit.db crit -\r\n...\r\n File \"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py\", line 337, in decode\r\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n File \"/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py\", line 355, in raw_decode\r\n raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```\r\nA nicer error message here would be one that says the JSON is invalid but suggests that maybe you could try `--csv`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/206/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 763283616, "node_id": "MDU6SXNzdWU3NjMyODM2MTY=", "number": 207, "title": "sqlite-utils analyze-tables command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2020-12-12T04:33:12Z", "updated_at": "2020-12-13T07:25:23Z", "closed_at": "2020-12-13T07:20:13Z", "author_association": "OWNER", "pull_request": null, "body": "A command which analyzes a table (potentially taking quite a while if the table is large) and outputs information for each column - things like:\r\n\r\n- How many unique values does this column have?\r\n- How many null rows?\r\n- How many blank rows? (defined as empty string)\r\n- What are the 10 most common values?\r\n- What are the 10 least common values?\r\n\r\nThe command can output this information to the terminal, but it should also provide an option for writing the information to a database table so it can be explored later.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/207/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 763320133, "node_id": "MDExOlB1bGxSZXF1ZXN0NTM3NzkxNjc1", "number": 208, "title": "sqlite-utils analyze-tables command and table.analyze_column() method", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2020-12-12T05:27:49Z", "updated_at": "2020-12-13T07:20:16Z", "closed_at": "2020-12-13T07:20:12Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/208", "body": "Refs #207\r\n\r\n- [x] Improve design of CLI output\r\n- [x] Truncate long values in least/most common\r\n- [x] Add a `-c` column selection option\r\n- [x] Tests\r\n- [x] Documentation", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/208/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}