{"id": 1066474200, "node_id": "I_kwDOCGYnMM4_kRrY", "number": 344, "title": "Support STRICT tables", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 14, "created_at": "2021-11-29T20:32:23Z", "updated_at": "2023-12-08T05:22:39Z", "closed_at": "2023-12-08T05:22:39Z", "author_association": "OWNER", "pull_request": null, "body": "New in SQLite 3.37.0, released a few days ago: https://www.sqlite.org/stricttables.html", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/344/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 2001006157, "node_id": "PR_kwDOCGYnMM5f2OZC", "number": 604, "title": "Add more STRICT table support", "user": {"value": 16437338, "label": "tkhattra"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-11-19T19:38:53Z", "updated_at": "2023-12-08T05:17:20Z", "closed_at": "2023-12-08T05:05:27Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/604", "body": "- https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776\r\n\r\nMake `table.transform()` preserve STRICT mode.\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--604.org.readthedocs.build/en/604/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/604/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 2007893839, "node_id": "I_kwDOCGYnMM53rgdP", "number": 605, "title": "Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here?", "user": {"value": 12229877, "label": "Zac-HD"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-11-23T10:19:46Z", "updated_at": "2023-12-08T05:07:54Z", "closed_at": "2023-12-08T05:07:54Z", "author_association": "NONE", "pull_request": null, "body": "I'm currently working on a new feature for Hypothesis, where we can dump a tidy jsonlines table of all the test cases we tried - including arguments, outcomes, timings, coverage, etc. Exploring this seems like a perfect cases for `sqlite-utils` and `datasette`, but I pretty quickly ran into an integer overflow problem and don't want to recommend that experience to my users.\r\n\r\nI originally went to report this as a bug... and then found https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038 almost exactly matched my repro \ud83d\ude05 \r\n\r\nhttps://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063 suggests that using `NUMERIC` would avoid this overflow error, although \"If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL.\" suggests that this would come at the cost of rounding to the nearest float value. Maybe I should just convert large integers to float before writing out my json?\r\n\r\nAfter a bit more hacking, \"manually cast large integers to float\" seems like a decent solution for my particular case, but having written it up I thought I might as well post this issue anyway - I hope it's useful feedback, and won't mind at all if you close as wontfix if it's not.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/605/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 2029161033, "node_id": "I_kwDOCGYnMM548opJ", "number": 606, "title": "str and int as aliases for text and integer", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-12-06T18:35:49Z", "updated_at": "2023-12-06T19:44:04Z", "closed_at": "2023-12-06T18:49:32Z", "author_association": "OWNER", "pull_request": null, "body": "I keep making this mistake:\r\n```bash\r\nsqlite-utils add-column content.db assets _since int\r\n```\r\n```\r\nUsage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME [[integer|float|b\r\n lob|text|INTEGER|FLOAT|BLOB|TEXT]]\r\nTry 'sqlite-utils add-column -h' for help.\r\n\r\nError: Invalid value for '[[integer|float|blob|text|INTEGER|FLOAT|BLOB|TEXT]]':\r\n 'int' is not one of 'integer', 'float', 'blob', 'text', 'INTEGER', 'FLOAT', 'BLOB', 'TEXT'.\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/606/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1976986318, "node_id": "I_kwDOCGYnMM511mrO", "number": 599, "title": "Cannot find spatialite on arm64 linux", "user": {"value": 37802088, "label": "MikeCoats"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-11-03T22:05:51Z", "updated_at": "2023-11-04T01:06:31Z", "closed_at": "2023-11-04T00:33:28Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Initially, I found an issue in `datasette` where it wouldn\u2019t find `spatialite` when running on my Radxa Rock 5B - an RK3588 powered SBC, running the arm64 build of Debian Bullseye. I confirmed the same behaviour on my Raspberry Pi 4 - a BCM2711 powered SBC, running the arm64 build of Debian Bookworm.\r\n\r\n```\r\n$ datasette --load-extension=spatialite example.db\r\nError: Could not find SpatiaLite extension\r\n```\r\n\r\nI did some digging and realised the issue originates in this project. Even with the `libsqlite3-mod-spatialite` package installed, `pytest` skips all of the GIS tests in the project.\r\n\r\n```\r\n$ apt list --installed | grep spatial\r\n[\u2026]\r\nlibsqlite3-mod-spatialite/stable,now 5.0.1-3 arm64 [installed]\r\n\r\n$ ls -l /usr/lib/*/*spatial*\r\nlrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0\r\nlrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0\r\n-rw-r--r-- 1 root root 7348584 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0\r\n```\r\n\r\n```\r\n$ pytest\r\ntests/test_get.py ...... [ 73%]\r\ntests/test_gis.py ssssssssssss [ 75%]\r\ntests/test_hypothesis.py .... [ 75%]\r\n```\r\n\r\nI tracked the issue down to the [`find_sqlite()` function in the `utils.py`](https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/utils.py#L60) file. The [`SPATIALITE_PATHS`](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L34-L39) array doesn\u2019t have an entry for the location of this module on arm64 linux.\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/599/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1884335789, "node_id": "PR_kwDOCGYnMM5Zs0KB", "number": 591, "title": "Test against Python 3.12 preview", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-09-06T16:10:00Z", "updated_at": "2023-11-04T00:58:03Z", "closed_at": "2023-11-04T00:58:02Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/591", "body": "https://dev.to/hugovk/help-test-python-312-beta-1508/\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--591.org.readthedocs.build/en/591/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/591/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 1, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1553425465, "node_id": "I_kwDOCGYnMM5cl2Q5", "number": 522, "title": "Add COLUMN_TYPE_MAPPING for timedelta", "user": {"value": 81377, "label": "maport"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-01-23T16:49:54Z", "updated_at": "2023-11-04T00:49:51Z", "closed_at": "2023-11-04T00:49:51Z", "author_association": "NONE", "pull_request": null, "body": "Currently trying to create a column with Python type `datetime.timedelta` results in an error:\r\n\r\n```\r\n>>> from sqlite_utils import Database\r\n>>> db = Database(\"test.db\")\r\n>>> test_tbl = db['test']\r\n>>> test_tbl.insert({'col1': datetime.timedelta()})\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 2979, in insert\r\n return self.insert_all(\r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 3082, in insert_all\r\n self.create(\r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 1574, in create\r\n self.db.create_table(\r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 961, in create_table\r\n sql = self.create_table_sql(\r\n File \"/usr/local/lib/python3.10/dist-packages/sqlite_utils/db.py\", line 852, in create_table_sql\r\n column_type=COLUMN_TYPE_MAPPING[column_type],\r\nKeyError: \r\n```\r\n\r\nThe reason this would be useful is that `MySQLdb` uses `timedelta` for MySQL `TIME` columns:\r\n\r\n```\r\n>>> import MySQLdb\r\n>>> conn = MySQLdb.connect(host='database', user='user', passwd='pw')\r\n>>> csr = conn.cursor()\r\n>>> csr.execute(\"SELECT CAST('11:20' AS TIME)\")\r\n>>> tuple(csr)\r\n((datetime.timedelta(seconds=40800),),)\r\n```\r\n\r\nSo currently any attempt to convert a MySQL DB with a `TIME` column using `db-to-sqlite` will result in the above error.\r\n\r\nI was rather surprised that `MySQLdb` uses `timedelta` for `TIME` columns but I see that [this column type](https://dev.mysql.com/doc/refman/8.0/en/time.html) is intended for time intervals as well as the time of day so it makes sense. \r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/522/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1919296686, "node_id": "PR_kwDOCGYnMM5bifPC", "number": 596, "title": "Fixes mapping for time fields related to mysql, closes #522", "user": {"value": 4420927, "label": "nezhar"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-09-29T13:41:48Z", "updated_at": "2023-11-04T00:49:50Z", "closed_at": "2023-11-04T00:49:50Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/596", "body": "Adds `COLUMN_TYPE_MAPPING` for `TIME` fields that are mapped as `datetime.timedelta` for MySQL and json represantation for `datetime.timedelta` in order to fix #522\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--596.org.readthedocs.build/en/596/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/596/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1926729132, "node_id": "PR_kwDOCGYnMM5b7Z_y", "number": 598, "title": "Fixed issue #433 - CLI eats cursor", "user": {"value": 62745, "label": "spookylukey"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-10-04T18:06:58Z", "updated_at": "2023-11-04T00:46:55Z", "closed_at": "2023-11-04T00:40:30Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/598", "body": "The issue is that underlying iterator is not fully consumed within the body of the `with file_progress()` block. Instead, that block creates generator expressions like `docs = (dict(zip(headers, row)) for row in reader)`\r\n\r\nThese iterables are consumed later, outside the `with file_progress()` block, which consumes the underlying iterator, and in turn updates the progress bar.\r\n\r\nThis means that the `ProgressBar.__exit__` method gets called before the last time the `ProgressBar.update` method gets called. The result is that the code to make the cursor invisible (inside the `update()` method) is called after the cleanup code to make it visible (in the `__exit__` method).\r\n\r\nThe fix is to move consumption of the `docs` iterators within the progress bar block. (\r\n\r\n(An additional fix, to make ProgressBar more robust against this kind of misuse, would to make it refusing to update after its `__exit__` method had been called, just like files cannot be `read()` after they are closed. That requires a in the click library).\r\n\r\nNote that Github diff obscures the simplicity of this diff, it's just indenting a block of code.\r\n\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--598.org.readthedocs.build/en/598/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/598/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 1, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1239034903, "node_id": "I_kwDOCGYnMM5J2iwX", "number": 433, "title": "CLI eats my cursor", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2022-05-17T18:52:52Z", "updated_at": "2023-11-04T00:46:30Z", "closed_at": "2023-11-04T00:46:30Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I'm not sure why this happens but `sqlite-utils` makes my terminal cursor disappear after running commands like `sqlite-utils insert`. I've only noticed this behavior in `sqlite-utils`, not in any other CLI tools\r\n\r\nI can still type commands after it runs but the text cursor is invisible", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/433/reactions\", \"total_count\": 5, \"+1\": 5, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1977004379, "node_id": "PR_kwDOCGYnMM5elFZf", "number": 600, "title": "Add spatialite arm64 linux path", "user": {"value": 37802088, "label": "MikeCoats"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2023-11-03T22:23:26Z", "updated_at": "2023-11-04T00:34:33Z", "closed_at": "2023-11-04T00:31:49Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/600", "body": "According to both [Debian](https://packages.debian.org/bookworm/arm64/libsqlite3-mod-spatialite/filelist) and [Ubuntu](https://packages.ubuntu.com/mantic/arm64/libsqlite3-mod-spatialite/filelist), the correct \u201ctarget triple\u201d for arm64 is `aarch64-linux-gnu`, so we should be looking in `/usr/lib/aarch64-linux-gnu` for `mod_spatialite.so`.\r\n\r\nI can confirm that on both of my Debian arm64 SBCs, `libsqlite3-mod-spatialite` installs to that path.\r\n\r\n```\r\n$ ls -l /usr/lib/*/*spatial*\r\nlrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0\r\nlrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0\r\n-rw-r--r-- 1 root root 7348584 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0\r\n```\r\n\r\nThis is a set of before and after snippets of pytest\u2019s output for this PR.\r\n\r\n### Before\r\n\r\n```\r\n$ pytest\r\ntests/test_get.py ...... [ 73%]\r\ntests/test_gis.py ssssssssssss [ 75%]\r\ntests/test_hypothesis.py .... [ 75%]\r\n```\r\n\r\n### After\r\n\r\n```\r\n$ pytest\r\ntests/test_get.py ...... [ 73%]\r\ntests/test_gis.py ............ [ 75%]\r\ntests/test_hypothesis.py .... [ 75%]\r\n```\r\n\r\n\r\nIssue: #599\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--600.org.readthedocs.build/en/600/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/600/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1907281675, "node_id": "I_kwDOCGYnMM5xrs8L", "number": 595, "title": "Cascading DELETE not working with Table.delete(pk)", "user": {"value": 123451970, "label": "cycle-data"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-09-21T15:46:41Z", "updated_at": "2023-09-25T09:38:57Z", "closed_at": "2023-09-25T09:38:13Z", "author_association": "NONE", "pull_request": null, "body": "Hi !\r\nI noticed that when I am trying to use the delete method of the Table object,\r\nthe record get properly deleted from the table, but the cascading delete triggers on foreign keys do not activate.\r\n\r\n`self.db[\"contact\"].delete(contact_id)`\r\n\r\nI tried querying the database directly via DB Browser and the triggers work without any issue.\r\nLooked up the source code and behind the scene this method is just querying the database normally so I'm not exactly sure where this behavior comes from.\r\n\r\nThank you in advance for your time ! ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/595/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1886771493, "node_id": "I_kwDOCGYnMM5wddkl", "number": 592, "title": "`table.transform()` should preserve `rowid` values", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-09-08T00:42:38Z", "updated_at": "2023-09-10T17:46:41Z", "closed_at": "2023-09-09T00:45:32Z", "author_association": "OWNER", "pull_request": null, "body": "I just spotted a bug when using https://datasette.io/plugins/datasette-configure-fts and https://datasette.io/plugins/datasette-edit-schema at the same time.\r\n\r\nSteps to reproduce:\r\n\r\n- Configure FTS for a table, then run a test search\r\n- Edit the schema for that table and change the order of columns\r\n- Run the test search again\r\n\r\nI got the wrong search results, which I think is because the `_fts` table pointed to the first table by `rowid` but those `rowid` values were entirely rewritten as a consequence of running `table.transform()` on the table.\r\n\r\nReconfiguring FTS on the table fixed the problem.\r\n\r\nI think `table.transform()` should be able to preserve `rowid` values.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/592/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1886783150, "node_id": "PR_kwDOCGYnMM5Z1H1d", "number": 593, "title": ".transform() now preserves rowid values, refs #592", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-09-08T01:02:28Z", "updated_at": "2023-09-10T17:44:59Z", "closed_at": "2023-09-09T00:45:30Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/593", "body": "Refs:\r\n- #592\r\n\r\n- [x] Tests against weird shaped tables\r\n\r\nI need to test that this works against:\r\n\r\n- `rowid` tables\r\n- Tables that have a column called `rowid` even though they are not rowid tables\r\n\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--593.org.readthedocs.build/en/593/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/593/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1857851384, "node_id": "I_kwDOCGYnMM5uvI_4", "number": 587, "title": "New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-08-19T20:01:26Z", "updated_at": "2023-08-19T20:04:33Z", "closed_at": "2023-08-19T20:04:32Z", "author_association": "OWNER", "pull_request": null, "body": "Extremely detailed story of how I got to this point:\r\n\r\n- https://github.com/simonw/llm/issues/162\r\n\r\nSteps to reproduce (only if that pragma is on though):\r\n```bash\r\npython -c '\r\nimport sqlite_utils\r\ndb = sqlite_utils.Database(memory=True)\r\ndb.execute(\"\"\"\r\nCREATE TABLE \"logs\" (\r\n [id] INTEGER PRIMARY KEY,\r\n [model] TEXT,\r\n [prompt] TEXT,\r\n [system] TEXT,\r\n [prompt_json] TEXT,\r\n [options_json] TEXT,\r\n [response] TEXT,\r\n [response_json] TEXT,\r\n [reply_to_id] INTEGER,\r\n [chat_id] INTEGER REFERENCES [log]([id]),\r\n [duration_ms] INTEGER,\r\n [datetime_utc] TEXT\r\n);\r\n\"\"\")\r\ndb[\"logs\"].add_foreign_key(\"reply_to_id\", \"logs\", \"id\")\r\n'\r\n```\r\nThis succeeds in some environments, fails in others.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/587/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1817289521, "node_id": "I_kwDOCGYnMM5sUaMx", "number": 577, "title": "Get `add_foreign_keys()` to work without modifying `sqlite_master`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-07-23T20:40:18Z", "updated_at": "2023-08-18T17:43:11Z", "closed_at": "2023-08-18T00:48:10Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1165-L1174\r\n\r\nThis is the only place in the code that attempts to modify `sqlite_master` directly, which fails on some Python installations.\r\n\r\nCould this use the `.transform()` trick instead?\r\n\r\nOr automatically switch to that trick if it hits an error?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/577/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1855894222, "node_id": "I_kwDOCGYnMM5unrLO", "number": 585, "title": "CLI equivalents to `transform(add_foreign_keys=)`", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2023-08-18T01:07:15Z", "updated_at": "2023-08-18T01:51:16Z", "closed_at": "2023-08-18T01:51:15Z", "author_association": "OWNER", "pull_request": null, "body": "The new options added in:\r\n- #577\r\nDeserve consideration in the CLI as well.\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/d2bcdc00c6ecc01a6e8135e775ffdb87572b802b/sqlite_utils/db.py#L1706-L1708", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/585/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1855836914, "node_id": "I_kwDOCGYnMM5undLy", "number": 583, "title": "Get rid of test.utils.collapse_whitespace", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-08-17T23:31:09Z", "updated_at": "2023-08-18T00:59:19Z", "closed_at": "2023-08-18T00:59:19Z", "author_association": "OWNER", "pull_request": null, "body": "I have a neater pattern for this now - instead of: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L472-L475\r\n\r\nI now prefer:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L1163-L1171", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/583/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1855838223, "node_id": "PR_kwDOCGYnMM5YM-I3", "number": 584, "title": ".transform() instead of modifying sqlite_master for add_foreign_keys", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 13, "created_at": "2023-08-17T23:32:45Z", "updated_at": "2023-08-18T00:48:13Z", "closed_at": "2023-08-18T00:48:08Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/584", "body": "Refs:\r\n- #577\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--584.org.readthedocs.build/en/584/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/584/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1823160748, "node_id": "I_kwDOCGYnMM5sqzms", "number": 581, "title": "`sqlite-utils convert --pdb` option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-07-26T21:02:50Z", "updated_at": "2023-07-26T21:07:45Z", "closed_at": "2023-07-26T21:06:10Z", "author_association": "OWNER", "pull_request": null, "body": "While using `sqlite-utils convert` I realized it would be handy if you could pass `--pdb` to have it open the debugger at the first instance of a failed conversion.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/581/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816997390, "node_id": "I_kwDOCGYnMM5sTS4O", "number": 576, "title": "Backfill the release notes prior to 0.4", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-07-23T05:41:42Z", "updated_at": "2023-07-23T05:49:51Z", "closed_at": "2023-07-23T05:48:21Z", "author_association": "OWNER", "pull_request": null, "body": "Currently the changelog starts at 0.4:\r\n\r\nhttps://sqlite-utils.datasette.io/en/3.34/changelog.html#id115\r\n\r\nI want the other releases - according to https://pypi.org/project/sqlite-utils/#history there are three missing:\r\n\r\n\"image\"\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/576/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816919568, "node_id": "I_kwDOCGYnMM5sS_4Q", "number": 575, "title": "Python API ability to opt-out of connection plugins", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-07-22T23:01:13Z", "updated_at": "2023-07-22T23:17:22Z", "closed_at": "2023-07-22T23:08:22Z", "author_association": "OWNER", "pull_request": null, "body": "Plugins affecting the CLI by default makes sense to me.\r\n\r\nI'm less confident about them _always_ affecting users of the Python API.\r\n\r\nI'm going to have them apply by default, but I'm going to add a mechanism to opt-out on an individual database basis. Basically this:\r\n\r\n```python\r\nfrom sqlite_utils import Database\r\ndb = Database(memory=True, execute_plugins=False)\r\n# Anything using db from here on will not execute plugins\r\n```\r\ncc @asg017 \r\n\r\nRefs:\r\n- #567 \r\n- #574 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/575/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816918185, "node_id": "I_kwDOCGYnMM5sS_ip", "number": 574, "title": "`prepare_connection()` plugin hook", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-07-22T22:52:47Z", "updated_at": "2023-07-22T23:13:14Z", "closed_at": "2023-07-22T22:59:10Z", "author_association": "OWNER", "pull_request": null, "body": "> Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34!\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424_\r\n ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/574/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1801394744, "node_id": "I_kwDOCGYnMM5rXxo4", "number": 567, "title": "Plugin system", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-07-12T17:02:14Z", "updated_at": "2023-07-22T22:59:37Z", "closed_at": "2023-07-22T22:59:36Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I'd like there to be a plugin system for sqlite-utils, similar to the datasette/llm plugins. I'd like to make plugins that would do things like:\r\n\r\n- Register SQLite extensions for more SQL functions + virtual tables\r\n- Register new subcommands\r\n- Different input file formats for `sqlite-utils memory`\r\n- Different output file formats (in addition to `--csv` `--tsv` `--nl` etc.\r\n\r\nA few real-world use-cases of plugins I'd like to see in sqlite-utils:\r\n\r\n- Register many of my sqlite extensions in sqlite-utils (`sqlite-http`, `sqlite-lines`, `sqlite-regex`, etc.)\r\n- New subcommands to work with `sqlite-vss` vector tables\r\n- Input/ouput Parquet/Avro/Arrow IPC files with `sqlite-arrow`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/567/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816917522, "node_id": "PR_kwDOCGYnMM5WJ6Jm", "number": 573, "title": "feat: Implement a prepare_connection plugin hook", "user": {"value": 15178711, "label": "asg017"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-07-22T22:48:44Z", "updated_at": "2023-07-22T22:59:09Z", "closed_at": "2023-07-22T22:59:09Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/573", "body": "Just like the [Datasette prepare_connection hook](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette), this PR adds a similar hook for the `sqlite-utils` plugin system. \r\n\r\nThe sole argument is `conn`, since I don't believe a `database` or `datasette` argument would be relevant here. \r\n\r\nI want to do this so I can release `sqlite-utils` plugins for my [SQLite extensions](https://github.com/asg017/sqlite-ecosystem), similar to the Datasette plugins I've release for them. \r\n\r\nAn example plugin: https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c\r\n\r\n```bash\r\n$ sqlite-utils install https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c/archive/5f5ad549a40860787629c69ca120a08c32519e99.zip\r\n\r\n$ sqlite-utils memory 'select hello(\"alex\") as response'\r\n[{\"response\": \"Hello, alex!\"}]\r\n```\r\nRefs:\r\n- #574 \r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--573.org.readthedocs.build/en/573/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/573/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1816876211, "node_id": "I_kwDOCGYnMM5sS1Sz", "number": 571, "title": "`.transform(keep_table=...)` option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-07-22T19:49:29Z", "updated_at": "2023-07-22T22:32:18Z", "closed_at": "2023-07-22T22:32:18Z", "author_association": "OWNER", "pull_request": null, "body": ">> Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.\r\n>\r\n> I think `keep_table=\"name_of_table\"` is good for this.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324_\r\n ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/571/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816877910, "node_id": "I_kwDOCGYnMM5sS1tW", "number": 572, "title": "Don't test Python 3.7 against textual", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-07-22T19:57:03Z", "updated_at": "2023-07-22T22:16:50Z", "closed_at": "2023-07-22T22:16:50Z", "author_association": "OWNER", "pull_request": null, "body": "Spotted this in the GitHub Actions logs:\r\n\r\n![IMG_5046](https://github.com/simonw/sqlite-utils/assets/9599/81fb1093-cd8a-4019-a612-2e49b500c933)\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/572/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1786243905, "node_id": "I_kwDOCGYnMM5qd-tB", "number": 564, "title": "Document that running `db.transform()` tidies up the schema indentation", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-07-03T13:59:28Z", "updated_at": "2023-07-22T22:15:34Z", "closed_at": "2023-07-22T22:15:34Z", "author_association": "OWNER", "pull_request": null, "body": "> ... and it turns out running `.transform()` with no arguments still fixes the format of the schema!\r\n\r\n```pycon\r\n>>> db[\"log\"].add_column(\"foo\", str)\r\n\r\n>>> db[\"log\"].add_column(\"bar\", str)\r\n
\r\n>>> db[\"log\"].add_column(\"baz\", str)\r\n
\r\n>>> print(db[\"log\"].schema)\r\nCREATE TABLE \"log\" (\r\n [id] INTEGER PRIMARY KEY,\r\n [name2] TEXT,\r\n [age] INTEGER,\r\n [weight] FLOAT\r\n, [foo] TEXT, [bar] TEXT, [baz] TEXT)\r\n>>> db[\"log\"].transform()\r\n
\r\n>>> print(db[\"log\"].schema)\r\nCREATE TABLE \"log\" (\r\n [id] INTEGER PRIMARY KEY,\r\n [name2] TEXT,\r\n [age] INTEGER,\r\n [weight] FLOAT,\r\n [foo] TEXT,\r\n [bar] TEXT,\r\n [baz] TEXT\r\n)\r\n```\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618347727_\r\n ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/564/reactions\", \"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "draft": null, "state_reason": "completed"} {"id": 1205687423, "node_id": "I_kwDOCGYnMM5H3VR_", "number": 426, "title": "CLI docs should link to Python docs and vice versa", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": {"value": 9599, "label": "simonw"}, "milestone": null, "comments": 1, "created_at": "2022-04-15T16:05:15Z", "updated_at": "2023-07-22T22:13:22Z", "closed_at": "2023-07-22T22:13:22Z", "author_association": "OWNER", "pull_request": null, "body": "For every command/API method there should be a link to the equivalent in the other form factor.\r\n\r\nMaybe also link to the API and CLI reference pages too.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/426/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1786258502, "node_id": "I_kwDOCGYnMM5qeCRG", "number": 565, "title": "Table renaming: db.rename_table() and sqlite-utils rename-table", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-07-03T14:07:42Z", "updated_at": "2023-07-22T22:12:40Z", "closed_at": "2023-07-22T22:12:40Z", "author_association": "OWNER", "pull_request": null, "body": "> I find myself wanting two new features in `sqlite-utils`:\r\n> - The ability to have the new transformed table set to a specific name, while keeping the old table around\r\n> - The ability to rename a table (`sqlite-utils` doesn't have a table rename function at all right now)\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618375042_\r\n ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/565/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816851056, "node_id": "I_kwDOCGYnMM5sSvJw", "number": 568, "title": "table.create(..., replace=True)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2023-07-22T18:12:22Z", "updated_at": "2023-07-22T19:25:35Z", "closed_at": "2023-07-22T19:15:44Z", "author_association": "OWNER", "pull_request": null, "body": "Found myself using this pattern to quickly prototype a schema:\r\n\r\n```python\r\nimport sqlite_utils\r\ndb = sqlite_utils.Database(memory=True)\r\n\r\nprint(db[\"answers_chunks\"].create({\r\n \"id\": int,\r\n \"content\": str,\r\n \"embedding_type_id\": int,\r\n \"embedding\": bytes,\r\n \"embedding_content_md5\": str,\r\n \"source\": str,\r\n}, pk=\"id\", transform=True).schema)\r\n```\r\n\r\n\"image\"\r\n\r\nUsing `replace=True` to drop and then recreate the table would be neat here, and would be consistent with other places that use `replace=True`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/568/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816852402, "node_id": "I_kwDOCGYnMM5sSvey", "number": 569, "title": "register_command plugin hook", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-07-22T18:17:27Z", "updated_at": "2023-07-22T19:19:35Z", "closed_at": "2023-07-22T19:19:35Z", "author_association": "OWNER", "pull_request": null, "body": "> I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450_\r\n ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/569/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1816857105, "node_id": "I_kwDOCGYnMM5sSwoR", "number": 570, "title": "`sqlite-utils install -e` option", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-07-22T18:32:23Z", "updated_at": "2023-07-22T18:55:59Z", "closed_at": "2023-07-22T18:32:56Z", "author_association": "OWNER", "pull_request": null, "body": "As seen in LLM.\r\n\r\nNeeded while working on:\r\n- #567", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/570/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1785360409, "node_id": "I_kwDOCGYnMM5qanAZ", "number": 563, "title": "`--empty-null` option when importing CSV", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-07-03T05:23:36Z", "updated_at": "2023-07-03T05:44:43Z", "closed_at": "2023-07-03T05:42:30Z", "author_association": "OWNER", "pull_request": null, "body": "CSV files with empty cells in (which come through as the empty string) are common and a bit gross.\r\n\r\nHaving an option that means \"and if it's an empty string store `null` instead) would be cool.\r\n\r\nI brainstormed name options here https://chat.openai.com/share/c947b738-ee7d-419c-af90-bc84e90987da", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/563/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1777548699, "node_id": "I_kwDOCGYnMM5p8z2b", "number": 561, "title": "`--stop-after` option for `insert` and `upsert` commands", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-06-27T18:44:15Z", "updated_at": "2023-06-27T18:50:09Z", "closed_at": "2023-06-27T18:50:08Z", "author_association": "OWNER", "pull_request": null, "body": "I found myself wanting to insert rows from a 849MB CSV file without processing the whole thing: https://huggingface.co/datasets/jerpint-org/HackAPrompt-Playground-Submissions/tree/main", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/561/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1773458985, "node_id": "PR_kwDOCGYnMM5T2mMb", "number": 560, "title": "Use sqlean if available in environment", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2023-06-25T19:48:48Z", "updated_at": "2023-06-26T08:21:00Z", "closed_at": "2023-06-25T23:25:51Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/560", "body": "Refs:\r\n- #559 \r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--560.org.readthedocs.build/en/560/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/560/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 810618495, "node_id": "MDU6SXNzdWU4MTA2MTg0OTU=", "number": 235, "title": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified", "user": {"value": 6913891, "label": "kristomi"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 18, "created_at": "2021-02-17T23:33:23Z", "updated_at": "2023-06-26T01:47:01Z", "closed_at": "2023-06-25T23:25:53Z", "author_association": "NONE", "pull_request": null, "body": "Thanks for what seems like a truly great suite of libraries. I wanted to try out Datasette, but never got more than half way through your YouTube video with the SF tree dataset. Whenever I try to extract a column, I get a `sqlite3.OperationalError: table sqlite_master may not be modified` error from Python. This snippet reproduces the error on my system, Python 3.9.1 and sqlite-utils 3.5 on an M1 Macbook Pro running in rosetta mode:\r\n```\r\ncurl \"https://data.nasa.gov/resource/y77d-th95.json\" | \\\r\n sqlite-utils insert meteorites.db meteorites - --pk=id\r\nsqlite-utils extract meteorites.db meteorites recclass\r\n```\r\n\r\nI have tried googling the problem, but all I've found is that this *might* be a problem with the sqlite3 database running in defensive mode, but I definitely can't know for sure. Does the problem seem familiar to you?\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/235/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1773450152, "node_id": "I_kwDOCGYnMM5ptLOo", "number": 559, "title": "sqlean support", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-06-25T19:27:26Z", "updated_at": "2023-06-25T23:25:53Z", "closed_at": "2023-06-25T23:25:53Z", "author_association": "OWNER", "pull_request": null, "body": "If sqlean is available, use that.\r\n\r\nRefs:\r\n- https://github.com/nalgeon/sqlean.py/issues/1#issuecomment-1605707788\r\n\r\nThis will provide a good workaround for:\r\n- #235 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/559/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1655860104, "node_id": "I_kwDOCGYnMM5ismuI", "number": 535, "title": "rows: --transpose or psql extended view-like functionality", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-04-05T15:37:33Z", "updated_at": "2023-06-15T08:39:49Z", "closed_at": "2023-06-14T22:05:28Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "It would be nice if the rows subcommand had a flag, perhaps called `--transpose` which would print in long form instead of wide. Similar to extended display mode in psql (`\\x`)\r\n\r\nIn other words instead of this:\r\n\r\n```\r\nsqlite-utils rows --limit 5 --fmt github track_metadata.db songs\r\n```\r\n\r\n| track_id | title | song_id | release | artist_id | artist_mbid | artist_name | duration | artist_familiarity | artist_hotttnesss | year | track_7digitalid | shs_perf | shs_work |\r\n|--------------------|-------------------|--------------------|--------------------------------------|--------------------|--------------------------------------|------------------|------------|----------------------|---------------------|--------|--------------------|------------|------------|\r\n| TRMMMYQ128F932D901 | Silent Night | SOQMMHC12AB0180CB8 | Monster Ballads X-Mas | ARYZTJS1187B98C555 | 357ff05d-848a-44cf-b608-cb34b5701ae5 | Faster Pussy cat | 252.055 | 0.649822 | 0.394032 | 2003 | 7032331 | -1 | 0 |\r\n| TRMMMKD128F425225D | Tanssi vaan | SOVFVAK12A8C1350D9 | Karkuteill\u00e4 | ARMVN3U1187FB3A1EB | 8d7ef530-a6fd-4f8f-b2e2-74aec765e0f9 | Karkkiautomaatti | 156.551 | 0.439604 | 0.356992 | 1995 | 1514808 | -1 | 0 |\r\n| TRMMMRX128F93187D9 | No One Could Ever | SOGTUKN12AB017F4F1 | Butter | ARGEKB01187FB50750 | 3d403d44-36ce-465c-ad43-ae877e65adc4 | Hudson Mohawke | 138.971 | 0.643681 | 0.437504 | 2006 | 6945353 | -1 | 0 |\r\n| TRMMMCH128F425532C | Si Vos Quer\u00e9s | SOBNYVR12A8C13558C | De Culo | ARNWYLR1187B9B2F9C | 12be7648-7094-495f-90e6-df4189d68615 | Yerba Brava | 145.058 | 0.448501 | 0.372349 | 2003 | 2168257 | -1 | 0 |\r\n| TRMMMWA128F426B589 | Tangle Of Aspens | SOHSBXH12A8C13B0DF | Rene Ablaze Presents Winter Sessions | AREQDTE1269FB37231 | | Der Mystic | 514.298 | 0 | 0 | 0 | 2264873 | -1 | 0 |\r\n\r\n\r\nThe output would look something like this:\r\n\r\n```\r\n$ for col in (sqlite-columns track_metadata.db songs)\r\n sqlite-utils --fmt github track_metadata.db \"select $col from songs order by rowid desc limit 5\"\r\nend\r\n```\r\n\r\n| track_id |\r\n|--------------------|\r\n| TRYYYVU12903CD01E3 |\r\n| TRYYYDJ128F9310A21 |\r\n| TRYYYMG128F4260ECA |\r\n| TRYYYJO128F426DA37 |\r\n| TRYYYUS12903CD2DF0 |\r\n| title |\r\n|-------------------------------------|\r\n| Fernweh feat. Sektion Kuchik\u00e4schtli |\r\n| Faraday |\r\n| Novemba |\r\n| Jago Chhadeo |\r\n| O Samba Da Vida |\r\n| song_id |\r\n|--------------------|\r\n| SOWXJXQ12AB0189F43 |\r\n| SOLXGOR12A81C21EB7 |\r\n| SOHODZI12A8C137BB3 |\r\n| SOXQYIQ12A8C137FBB |\r\n| SOTXAME12AB018F136 |\r\n| release |\r\n|---------------------------------|\r\n| So Oder So |\r\n| The Trance Collection Vol. 2 |\r\n| Dub_Connected: electronic music |\r\n| Naale Baba Lassi Pee Gya |\r\n| Pacha V.I.P. |\r\n| artist_id |\r\n|--------------------|\r\n| AR7PLM21187B990D08 |\r\n| ARCMCOK1187B9B1073 |\r\n| ARZ3R6M1187B9AF750 |\r\n| ART5FZD1187B9A7FCF |\r\n| AR7Z4J81187FB3FC59 |\r\n| artist_mbid |\r\n|--------------------------------------|\r\n| 3af2b07e-c91c-4160-9bda-f0b9e3144ed3 |\r\n| 4ac5f3de-c5ad-475e-ad50-41f1ef9dba20 |\r\n| 8b97e9c8-61f5-4615-9a96-276f24204e34 |\r\n| 2357c400-9109-42b6-b3fe-9e2d9f8e3872 |\r\n| 9d50cb20-7e42-45cc-b0dd-154c3e92a577 |\r\n| artist_name |\r\n|----------------|\r\n| Texta |\r\n| Elude |\r\n| Gabriel Le Mar |\r\n| Kuldeep Manak |\r\n| Kiko Navarro |\r\n| duration |\r\n|------------|\r\n| 295.079 |\r\n| 484.519 |\r\n| 553.038 |\r\n| 244.166 |\r\n| 217.443 |\r\n| artist_familiarity |\r\n|----------------------|\r\n| 0.552977 |\r\n| 0.403668 |\r\n| 0.556918 |\r\n| 0.4015 |\r\n| 0.528617 |\r\n| artist_hotttnesss |\r\n|---------------------|\r\n| 0.454869 |\r\n| 0.256935 |\r\n| 0.336914 |\r\n| 0.374866 |\r\n| 0.411595 |\r\n| year |\r\n|--------|\r\n| 2004 |\r\n| 0 |\r\n| 0 |\r\n| 0 |\r\n| 0 |\r\n| track_7digitalid |\r\n|--------------------|\r\n| 8486723 |\r\n| 5472456 |\r\n| 2219291 |\r\n| 1632096 |\r\n| 7522478 |\r\n| shs_perf |\r\n|------------|\r\n| -1 |\r\n| -1 |\r\n| -1 |\r\n| -1 |\r\n| -1 |\r\n| shs_work |\r\n|------------|\r\n| 0 |\r\n| 0 |\r\n| 0 |\r\n| 0 |\r\n| 0 |\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/535/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1581090327, "node_id": "I_kwDOCGYnMM5ePYYX", "number": 529, "title": "Microsoft line endings", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-02-12T02:20:48Z", "updated_at": "2023-06-14T23:12:12Z", "closed_at": "2023-06-14T23:11:47Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "sqlite-utils prints `\\r\\n` but [it should probably](https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/) print `\\n` (unless the platform is detected as Windows?)\r\n\r\nIt has tripped me up a few times when piping the output of sqlite-utils to other programs:\r\n\r\n```\r\n$ sqlite-utils --no-headers --csv ~/lb/fs/d.db 'select path from media limit 1' | cat -A\r\n/mnt/d7/file^M$\r\n$ sqlite-utils --no-headers --csv ~/lb/fs/d.db 'select path from media limit 1' | tr -d '\\r' | cat -A\r\n/mnt/d7/file$\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/529/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1740150327, "node_id": "I_kwDOCGYnMM5nuJY3", "number": 557, "title": "Aliased ROWID option for tables created from alter=True commands", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-06-04T05:29:28Z", "updated_at": "2023-06-14T06:09:21Z", "closed_at": "2023-06-05T19:26:26Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "> If you use INTEGER PRIMARY KEY column, the VACUUM does not change the values of that column. However, if you use unaliased rowid, the VACUUM command will reset the rowid values.\r\n\r\nROWID should never be used with foreign keys but the simple act of aliasing rowid to id (which is what happens when one does `id integer primary key` DDL) makes it OK.\r\n\r\nIt would be convenient if there were more options to use a string column (eg. filepath) as the PK, and be able to use it during upserts, but when creating a foreign key, to create an integer column which aliases rowid\r\n\r\nI made an attempt to switch to integer primary keys here but it is not going well... In my usecase the path column is a business key. Yes, it should be as simple as including the `id` column in any select statement where I plan on using `upsert` but it would be nice if this could be abstracted away somehow https://github.com/chapmanjacobd/library/commit/788cd125be01d76f0fe2153335d9f6b21db1343c\r\n\r\nhttps://github.com/chapmanjacobd/library/actions/runs/5173602136/jobs/9319024777", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/557/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1718612569, "node_id": "I_kwDOCGYnMM5mb_JZ", "number": 552, "title": "Document how to setup shell auto-completion", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-05-21T19:20:41Z", "updated_at": "2023-05-21T21:05:16Z", "closed_at": "2023-05-21T21:03:40Z", "author_association": "OWNER", "pull_request": null, "body": "https://click.palletsprojects.com/en/8.1.x/shell-completion/\r\n\r\nThis works for `zsh`:\r\n\r\n eval \"$(_SQLITE_UTILS_COMPLETE=zsh_source sqlite-utils)\"\r\n\r\nThis will probably work for `bash`:\r\n\r\n eval \"$(_SQLITE_UTILS_COMPLETE=bash_source sqlite-utils)\"\r\n\r\nNeed to add this to the installation docs here: https://sqlite-utils.datasette.io/en/stable/installation.html - along with the pattern for adding that to `.zshrc` or whatever.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/552/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1718607907, "node_id": "I_kwDOCGYnMM5mb-Aj", "number": 551, "title": "Make as many examples in the CLI docs as possible copy-and-pastable", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-05-21T19:04:10Z", "updated_at": "2023-05-21T21:04:04Z", "closed_at": "2023-05-21T20:57:24Z", "author_association": "OWNER", "pull_request": null, "body": "e.g. in this section:\r\n\r\nhttps://sqlite-utils.datasette.io/en/stable/cli.html#running-queries-directly-against-csv-or-json\r\n\r\n\"image\"\r\n\r\nThe little copy button will also copy the `$ ` which breaks the examples when copied.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/551/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1718635018, "node_id": "PR_kwDOCGYnMM5Q9lY4", "number": 553, "title": "Reformatted CLI examples in docs", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-05-21T20:44:34Z", "updated_at": "2023-05-21T20:57:27Z", "closed_at": "2023-05-21T20:57:23Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/553", "body": "Refs:\r\n- #551\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--553.org.readthedocs.build/en/553/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/553/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1718517882, "node_id": "I_kwDOCGYnMM5mboB6", "number": 545, "title": "Try out Trogon for a tui interface", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 6, "created_at": "2023-05-21T14:08:25Z", "updated_at": "2023-05-21T19:33:13Z", "closed_at": "2023-05-21T18:41:58Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/Textualize/trogon", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/545/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1718595700, "node_id": "I_kwDOCGYnMM5mb7B0", "number": 550, "title": "AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-05-21T18:24:39Z", "updated_at": "2023-05-21T18:42:25Z", "closed_at": "2023-05-21T18:41:58Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/actions/runs/5039064797/jobs/9036965488\r\n\r\n```\r\nTraceback (most recent call last):\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/bin/flake8\", line 8, in \r\n sys.exit(main())\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/cli.py\", line 22, in main\r\n app.run(argv)\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py\", line 363, in run\r\n self._run(argv)\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py\", line 350, in _run\r\n self.initialize(argv)\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py\", line 330, in initialize\r\n self.find_plugins(config_finder)\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py\", line 153, in find_plugins\r\n self.check_plugins = plugin_manager.Checkers(local_plugins.extension)\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py\", line 357, in __init__\r\n self.namespace, local_plugins=local_plugins\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py\", line 238, in __init__\r\n self._load_entrypoint_plugins()\r\n File \"/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py\", line 254, in _load_entrypoint_plugins\r\n eps = importlib_metadata.entry_points().get(self.namespace, ())\r\nAttributeError: 'EntryPoints' object has no attribute 'get'\r\nError: Process completed with exit code 1.\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/550/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1718586377, "node_id": "PR_kwDOCGYnMM5Q9cAv", "number": 549, "title": "TUI powered by Trogon", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-05-21T17:55:42Z", "updated_at": "2023-05-21T18:42:00Z", "closed_at": "2023-05-21T18:41:56Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/549", "body": "Refs:\r\n- #545\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--549.org.readthedocs.build/en/549/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/549/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1718576761, "node_id": "I_kwDOCGYnMM5mb2Z5", "number": 548, "title": "analyze-tables should validate provide --column names", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-05-21T17:20:24Z", "updated_at": "2023-05-21T17:35:52Z", "closed_at": "2023-05-21T17:35:52Z", "author_association": "OWNER", "pull_request": null, "body": "Noticed this while testing:\r\n- #547 \r\n\r\nIf you pass a non-existent column to `-c/--column` you don't get an error message.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/548/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1718572201, "node_id": "I_kwDOCGYnMM5mb1Sp", "number": 547, "title": "No need to show common values if everything is null", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-05-21T17:05:07Z", "updated_at": "2023-05-21T17:19:21Z", "closed_at": "2023-05-21T17:19:21Z", "author_association": "OWNER", "pull_request": null, "body": "Noticed this:\r\n\r\n```\r\n% sqlite-utils analyze-tables content.db repos -c delete_branch_on_merge --common-limit 20 --no-least\r\nrepos.delete_branch_on_merge: (1/1)\r\n\r\n Total rows: 158\r\n Null rows: 158\r\n Blank rows: 0\r\n\r\n Distinct values: 0\r\n\r\n Most common:\r\n 158: None\r\n```\r\n\r\nThe `158: None` there is duplicate information considering we already know there are 158/158 null rows.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/547/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1718515590, "node_id": "I_kwDOCGYnMM5mbneG", "number": 544, "title": "New options for analyze-tables --common-limit --no-most and --no-least", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-05-21T14:03:19Z", "updated_at": "2023-05-21T17:03:06Z", "closed_at": "2023-05-21T16:19:31Z", "author_association": "OWNER", "pull_request": null, "body": "The \"least common\" section is frequently uninteresting, especially for huge tables with a large number of repeated-once values.\r\n\r\n sqlite-utils analyze-tables content.db repos --common-limit 20 --no-least", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/544/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1718550688, "node_id": "PR_kwDOCGYnMM5Q9VH0", "number": 546, "title": "Analyze tables options: --common-limit, --no-most, --no-least", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-05-21T15:54:39Z", "updated_at": "2023-05-21T16:19:30Z", "closed_at": "2023-05-21T16:19:30Z", "author_association": "OWNER", "pull_request": "simonw/sqlite-utils/pulls/546", "body": "Refs #544\r\n\r\n- [x] Documentation for CLI options\r\n- [x] Documentation for new Python API parameters: `most_common: bool` and `least_common: bool`\r\n- [x] Tests for CLI\r\n- [x] Tests for Python API", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/546/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1578790070, "node_id": "I_kwDOCGYnMM5eGmy2", "number": 527, "title": "`Table.convert()` skips falsey values", "user": {"value": 167893, "label": "mcarpenter"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2023-02-10T00:00:52Z", "updated_at": "2023-05-09T21:15:05Z", "closed_at": "2023-05-08T21:03:24Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "# Summary\r\n\r\nBy design, `Table.convert()` does [not attempt](https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L2663) conversion of falsey values (`None`, `\"\"`, `0`, ...). This is surprising (directly contradicts the docstring) and `convert()` may quietly skip cells where the user assumed a conversion would take place. \r\n\r\n# Example\r\nIncrement a column of integers by one\r\n\r\n``` python\r\nfrom sqlite_utils import Database\r\n\r\ndb = Database(memory=True)\r\ntable = db['table']\r\ncol = 'x'\r\ntable.insert_all([{col: 0}, {col:1}])\r\nprint(table.get(1)) # 0\r\nprint(table.get(2)) # 1\r\nprint()\r\n\r\ntable.convert(col, lambda x: x+1)\r\nprint(table.get(1)) # got 0, expected 1 \u26a0\u26a0\u26a0\r\nprint(table.get(2)) # got 2, expected 2\r\n```\r\n\r\nAnother example might be, say, transforming cells containing empty string to `NULL`.\r\n\r\n# Discussion\r\n\r\nThis was, I think, a pragmatic choice so that consumers can skip writing guard clauses for these falsey values (particularly from the CLI). But this surprising undocumented behavior can lead to incorrect data. I don't think this is a good trade-off between convenience and correctness.\r\n\r\nIn the absence of this convenience users will either have to write guard clauses into their conversion expressions (or adapt the called function to do the same), so: \r\n``` python\r\n fn(value) if value else value\r\n```\r\ninstead of:\r\n``` python\r\n fn(value)\r\n```\r\nThis is more typing and sometimes I will forget, and there will be errors. (But they will be noisy errors, which is a good thing).\r\n\r\nSuch a change will certainly inconvenience some existing consumers; there will be some breakage. But I think this is worth it to avoid quietly not converting some values by default, which can lead to quietly bad data.\r\n\r\nI have a PR that I will attach, please take a look and see what you think.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/527/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1665200812, "node_id": "PR_kwDOCGYnMM5OKveS", "number": 537, "title": "Support self-referencing FKs in `Table.create`", "user": {"value": 544011, "label": "numist"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-04-12T20:26:59Z", "updated_at": "2023-05-08T22:45:33Z", "closed_at": "2023-05-08T21:10:01Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/537", "body": "\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--537.org.readthedocs.build/en/537/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/537/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1701018909, "node_id": "I_kwDOCGYnMM5lY30d", "number": 543, "title": "Tests broken on Windows due to new convert() lambda names", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-05-08T22:11:29Z", "updated_at": "2023-05-08T22:19:04Z", "closed_at": "2023-05-08T22:19:04Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/actions/runs/4920084038/jobs/8788501314\r\n```python\r\nsql = 'update [example] set [dt] = lambda_-9223371942137158589([dt]);'\r\n```\r\nFrom:\r\n- #526", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/543/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1516644980, "node_id": "I_kwDOCGYnMM5aZip0", "number": 520, "title": "rows_from_file() raises confusing error if file-like object is not in binary mode", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2023-01-02T19:00:14Z", "updated_at": "2023-05-08T22:08:07Z", "closed_at": "2023-05-08T22:08:07Z", "author_association": "OWNER", "pull_request": null, "body": "I got this error:\r\n\r\n```\r\n File \"/Users/simon/Dropbox/Development/openai-to-sqlite/openai_to_sqlite/cli.py\", line 27, in embeddings\r\n rows, _ = rows_from_file(input)\r\n ^^^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/simon/.local/share/virtualenvs/openai-to-sqlite-jt4obeb2/lib/python3.11/site-packages/sqlite_utils/utils.py\", line 305, in rows_from_file\r\n first_bytes = buffered.peek(2048).strip()\r\n ^^^^^^^^^^^^^^^^^^^\r\n```\r\nFrom this code:\r\n```python\r\n\r\n@cli.command()\r\n@click.argument(\r\n \"db_path\",\r\n type=click.Path(file_okay=True, dir_okay=False, allow_dash=False),\r\n)\r\n@click.option(\r\n \"-i\",\r\n \"--input\",\r\n type=click.File(\"r\"),\r\n default=\"-\",\r\n)\r\ndef embeddings(db_path, input):\r\n \"Store embeddings for one or more text documents\"\r\n click.echo(\"Here is some output\")\r\n db = sqlite_utils.Database(db_path)\r\n rows, _ = rows_from_file(input)\r\n print(list(rows))\r\n```\r\nThe error went away when I changed it to `type=click.File(\"rb\")`.\r\n\r\nThis should either be called out in the documentation or `rows_from_file()` should be fixed to handle text-mode files in addition to binary files.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/520/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1279144769, "node_id": "I_kwDOCGYnMM5MPjNB", "number": 448, "title": "Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto'", "user": {"value": 236907, "label": "mungewell"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-06-21T21:48:27Z", "updated_at": "2023-05-08T22:01:00Z", "closed_at": "2023-05-08T22:01:00Z", "author_association": "NONE", "pull_request": null, "body": "Attempting to run the example given here (without extra bracket ;-):\r\nhttps://sqlite-utils.datasette.io/en/stable/python-api.html#reading-rows-from-a-file\r\n```\r\nfrom sqlite_utils.utils import rows_from_file\r\nimport io\r\n\r\nrows, format = rows_from_file(io.StringIO(\"id,name\\n1,Cleo\"))\r\nprint(list(rows), format)\r\n# Outputs [{'id': '1', 'name': 'Cleo'}] Format.CSV\r\n```\r\n\r\nGives error\r\n```\r\n>\"c:\\Program Files\\Python37\\python.exe\" test2.py\r\nTraceback (most recent call last):\r\n File \"test2.py\", line 4, in \r\n rows, format = rows_from_file(io.StringIO(\"id,name\\n1,Cleo\"))\r\n File \"C:\\Users\\swood\\Downloads\\sqlite-utils-main-20220621\\sqlite-utils-main\\sqlite_utils\\utils.py\", line 300, in rows_from_file\r\n first_bytes = buffered.peek(2048).strip()\r\nAttributeError: '_io.StringIO' object has no attribute 'readinto'\r\n```\r\n\r\nI am running Python on Windows.\r\n```\r\n>\"c:\\Program Files\\Python37\\python.exe\"\r\nPython 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/448/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1575131737, "node_id": "I_kwDOCGYnMM5d4ppZ", "number": 525, "title": "Repeated calls to `Table.convert()` fail", "user": {"value": 167893, "label": "mcarpenter"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-02-07T22:40:47Z", "updated_at": "2023-05-08T21:59:41Z", "closed_at": "2023-05-08T21:54:02Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "## Summary\r\nWhen using the API, repeated calls to `Table.convert()` do not work correctly since all conversions quietly use the callable (function, lambda) from the first call to `convert()` only. Subsequent invocations with different callables use the callable from the first invocation only.\r\n\r\n## Example\r\n```python\r\nfrom sqlite_utils import Database\r\n\r\ndb = Database(memory=True)\r\ntable = db['table']\r\ncol = 'x'\r\ntable.insert_all([{col: 1}])\r\nprint(table.get(1))\r\n\r\ntable.convert(col, lambda x: x*2)\r\nprint(table.get(1))\r\n\r\ndef zeroize(x):\r\n return 0\r\n#zeroize = lambda x: 0\r\n#zeroize.__name__ = 'zeroize'\r\ntable.convert(col, zeroize)\r\nprint(table.get(1))\r\n```\r\n\r\nOutput:\r\n```\r\n{'x': 1}\r\n{'x': 2}\r\n{'x': 4}\r\n```\r\nExpected:\r\n```\r\n{'x': 1}\r\n{'x': 2}\r\n{'x': 0}\r\n```\r\n\r\n## Explanation\r\nThis is some relevant [documentation](https://github.com/simonw/sqlite-utils/blob/1491b66dd7439dd87cd5cd4c4684f46eb3c5751b/docs/python-api.rst#registering-custom-sql-functions:~:text=By%20default%20registering%20a%20function%20with%20the%20same%20name%20and%20number%20of%20arguments%20will%20have%20no%20effect).\r\n\r\n * `Table.convert()` takes a `Callable` to perform data conversion on a column\r\n * The `Callable` is passed to `Database.register_function()`\r\n * `Database.register_function()` uses the callable's `__name__` attribute for registration\r\n * (Aside: all lambdas have a `__name__` of ``: I thought this was the problem, and it was close, but not quite)\r\n * However `convert()` first wraps the callable by local function [`convert_value()`](https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L2661)\r\n * Consequently `register_function()` sees name `convert_value` for all invocations from `convert()`\r\n * `register_function()` silently ignores registrations using the same name, retaining only the first such registration\r\n\r\nThere's a mismatch between the comments and the code: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L404\r\n\r\nbut actually the existing function is returned/used instead (as the \"registering custom sql functions\" doc I linked above says too). Seems like this can be rectified to match the comment?\r\n\r\n## Suggested fix\r\nI think there are four things:\r\n1. The call to `register_function()` from `convert()`should have an explicit `name=` parameter (to continue using `convert_value()` and the progress bar).\r\n2. For functions, this name can be the real function name. (I understand the sqlite api needs a name, and it's nice if those are recognizable names where possible). For lambdas would `'lambda-{uuid}'` or similar be acceptable? \r\n3. `register_function()` really should throw an error on repeated attempts to register a duplicate (function, arity)-pair.\r\n4. A test? I haven't looked at the test framework here but seems this should be testable.\r\n\r\n## See also \r\n- #458 ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/525/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1576990618, "node_id": "PR_kwDOCGYnMM5JkkED", "number": 526, "title": "Fix repeated calls to `Table.convert()`", "user": {"value": 167893, "label": "mcarpenter"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2023-02-09T00:14:49Z", "updated_at": "2023-05-08T21:56:05Z", "closed_at": "2023-05-08T21:53:58Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/526", "body": "Fixes #525. All tests pass.\r\n\r\nThere's perhaps a better way to name lambdas? There could be a collision if a caller passes a function with name like `lambda_123456`.\r\n\r\nSQLite [documentation](https://www.sqlite.org/appfunc.html) is a little, ah, lite on function name specs. If there is a character that can be used in place of underscore in a SQLite function name that is not permitted in a Python function identifier then that could be a good way to prevent accidental collisions. (I tried dash, colon, dot, no joy).\r\n\r\nOtherwise, there is little chance of this happening and if it should happen the risk is mitigated by now throwing an exception in the case of a (name, arity) collision without `replace=True`.\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--526.org.readthedocs.build/en/526/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/526/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1465194249, "node_id": "I_kwDOCGYnMM5XVRcJ", "number": 514, "title": "upsert of new row with check constraints fails", "user": {"value": 193185, "label": "cldellow"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-11-26T16:12:23Z", "updated_at": "2023-05-08T21:50:52Z", "closed_at": "2023-05-08T21:50:51Z", "author_association": "NONE", "pull_request": null, "body": "(I originally opened this in https://github.com/simonw/datasette-insert/issues/20, but I see that that library depends on sqlite-utils)\r\n\r\nIn the case of a new row, upsert first adds the row, specifying only its pkeys: https://github.com/simonw/sqlite-utils/blob/965ca0d5f5bffe06cc02cd7741344d1ddddf9d56/sqlite_utils/db.py#L2783-L2787\r\n\r\nThis means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/514/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1465194930, "node_id": "PR_kwDOCGYnMM5DvZxa", "number": 515, "title": "upsert new rows with constraints, fixes #514", "user": {"value": 193185, "label": "cldellow"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-11-26T16:15:21Z", "updated_at": "2023-05-08T21:27:11Z", "closed_at": "2023-05-08T21:27:10Z", "author_association": "NONE", "pull_request": "simonw/sqlite-utils/pulls/515", "body": "This fixes #514 by making the initial insert for upserts include all columns, so that new rows can be added to tables with non-pkey columns that have constraints.\r\n\r\n(aside: I'm not a python programmer. `pip`? `pipenv`? `venv`? These are mystical incantations to me. The process to set up this repo for local development and testing was _so easy_. Thank you for the excellent contributing documentation!)\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--515.org.readthedocs.build/en/515/\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/515/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1044267332, "node_id": "I_kwDOCGYnMM4-PkFE", "number": 336, "title": "sqlite-util tranform --column-order mangles columns of type \"timestamp\"", "user": {"value": 536941, "label": "fgregg"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2021-11-04T01:15:38Z", "updated_at": "2023-05-08T21:13:38Z", "closed_at": "2023-05-08T21:13:38Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Reproducible code below:\r\n\r\n```bash\r\n> echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db\r\n> sqlite3 foo.db\r\nSQLite version 3.36.0 2021-06-18 18:36:39\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP);\r\nsqlite> .exit\r\n> sqlite-utils transform foo.db bar --column-order baz\r\nsqlite3 foo.db\r\nSQLite version 3.36.0 2021-06-18 18:36:39\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE IF NOT EXISTS \"bar\" (\r\n [baz] TEXT,\r\n [created_at] FLOAT DEFAULT 'CURRENT_TIMESTAMP'\r\n);\r\nsqlite> .exit\r\n> sqlite-utils transform foo.db bar --column-order baz\r\n> sqlite3 foo.db\r\nSQLite version 3.36.0 2021-06-18 18:36:39\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE IF NOT EXISTS \"bar\" (\r\n [baz] TEXT,\r\n [created_at] FLOAT DEFAULT '''CURRENT_TIMESTAMP'''\r\n);\r\n```\r\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/336/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1432377191, "node_id": "I_kwDOCGYnMM5VYFdn", "number": 509, "title": "`sqlite-utils transform` breaks DEFAULT string values and STRFTIME()", "user": {"value": 2199875, "label": "kennysong"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-11-02T02:32:23Z", "updated_at": "2023-05-08T21:13:38Z", "closed_at": "2023-05-08T21:13:38Z", "author_association": "NONE", "pull_request": null, "body": "Very nice library! Our team found sqlite-utils through @simonw's [comment on the \"Simple declarative schema migration for SQLite\" article](https://news.ycombinator.com/item?id=31249823), and we were excited to use it, but unfortunately `sqlite-utils transform` seems to break our DB. \r\n\r\nRunning `sqlite-utils transform` to modify a column mangles their DEFAULT values:\r\n\r\n- Default string values are wrapped in extra single quotes\r\n- Function expressions such as [`STRFTIME()`](https://www.sqlite.org/lang_datefunc.html) are turned into strings!\r\n\r\n------\r\n\r\nHere are steps to reproduce:\r\n\r\n**Original database**\r\n\r\n```\r\n$ sqlite3 test.db << EOF\r\nCREATE TABLE mytable (\r\n col1 TEXT DEFAULT 'foo',\r\n col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'))\r\n)\r\nEOF\r\n\r\n$ sqlite3 test.db \"SELECT sql FROM sqlite_master WHERE name = 'mytable';\"\r\nCREATE TABLE mytable (\r\n col1 TEXT DEFAULT 'foo',\r\n col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'))\r\n)\r\n```\r\n\r\n**Modified database after sqlite-utils**\r\n\r\n```\r\n$ sqlite3 test.db \"INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;\"\r\nfoo|2022-11-02 02:26:58.038\r\n\r\n$ sqlite-utils transform test.db mytable --rename col1 renamedcol1\r\n\r\n$ sqlite3 test.db \"SELECT sql FROM sqlite_master WHERE name = 'mytable';\"\r\nCREATE TABLE \"mytable\" (\r\n [renamedcol1] TEXT DEFAULT '''foo''',\r\n [col2] TEXT DEFAULT 'STRFTIME(''%Y-%m-%d %H:%M:%f'', ''NOW'')'\r\n)\r\n\r\n$ sqlite3 test.db \"INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;\"\r\nfoo|2022-11-02 02:26:58.038\r\n'foo'|STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')\r\n```\r\n\r\n(Related: #336)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/509/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1505568103, "node_id": "PR_kwDOCGYnMM5F609a", "number": 519, "title": "Fixes breaking DEFAULT values", "user": {"value": 13819005, "label": "rhoboro"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-12-21T01:27:52Z", "updated_at": "2023-05-08T21:13:37Z", "closed_at": "2023-05-08T21:13:37Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/519", "body": "Fixes #509, Fixes #336\r\n\r\nThanks for the great library!\r\nI fixed a bug that `sqlite-utils transform` breaks DEFAULT values.\r\nAll tests already present passed with no changes, and I added some tests for this PR.\r\n\r\nIn #509 case, fixed here.\r\n\r\n```shell\r\n$ sqlite3 test.db << EOF\r\nCREATE TABLE mytable (\r\n col1 TEXT DEFAULT 'foo',\r\n col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'))\r\n)\r\nEOF\r\n\r\n$ sqlite3 test.db \"SELECT sql FROM sqlite_master WHERE name = 'mytable';\"\r\nCREATE TABLE mytable (\r\n col1 TEXT DEFAULT 'foo',\r\n col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'))\r\n)\r\n\r\n$ sqlite3 test.db \"INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;\"\r\nfoo|2022-12-21 01:15:39.669\r\n\r\n$ sqlite-utils transform test.db mytable --rename col1 renamedcol1\r\n$ sqlite3 test.db \"SELECT sql FROM sqlite_master WHERE name = 'mytable';\"\r\nCREATE TABLE \"mytable\" (\r\n [renamedcol1] TEXT DEFAULT 'foo',\r\n [col2] TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) # \u2190 Non-String Value\r\n)\r\n\r\n$ sqlite3 test.db \"INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;\"\r\nfoo|2022-12-21 01:15:39.669\r\nfoo|2022-12-21 01:15:56.432\r\n```\r\n\r\nAnd #336 case also fixed.\r\nSpecial values are described [here](https://www.sqlite.org/lang_createtable.html).\r\n\r\n> 3.2. The DEFAULT clause\r\n> ... A default value may also be one of the special case-independent keywords CURRENT_TIME, CURRENT_DATE or CURRENT_TIMESTAMP.\r\n\r\n```shell\r\n$ echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db\r\n$ sqlite3 foo.db\r\nSQLite version 3.39.5 2022-10-14 20:58:05\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP);\r\nsqlite> .exit\r\n\r\n$ sqlite-utils transform foo.db bar --column-order baz\r\n$ sqlite3 foo.db\r\nSQLite version 3.39.5 2022-10-14 20:58:05\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE IF NOT EXISTS \"bar\" (\r\n [baz] TEXT,\r\n [created_at] FLOAT DEFAULT CURRENT_TIMESTAMP\r\n);\r\nsqlite> .exit\r\n\r\n$ sqlite-utils transform foo.db bar --column-order baz\r\n$ sqlite3 foo.db\r\nSQLite version 3.39.5 2022-10-14 20:58:05\r\nEnter \".help\" for usage hints.\r\nsqlite> .schema bar\r\nCREATE TABLE IF NOT EXISTS \"bar\" (\r\n [baz] TEXT,\r\n [created_at] FLOAT DEFAULT CURRENT_TIMESTAMP # \u2190 Non-String Value\r\n);\r\n```\r\n\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--519.org.readthedocs.build/en/519/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/519/reactions\", \"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1578793661, "node_id": "PR_kwDOCGYnMM5Jqn1u", "number": 528, "title": "Enable `Table.convert()` on falsey values", "user": {"value": 167893, "label": "mcarpenter"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-02-10T00:04:09Z", "updated_at": "2023-05-08T21:08:23Z", "closed_at": "2023-05-08T21:08:23Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/528", "body": "Fixes #527\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--528.org.readthedocs.build/en/528/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/528/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1620254998, "node_id": "I_kwDOCGYnMM5gkyEW", "number": 532, "title": "Show more information when JSON can't be imported with sqlite-utils insert", "user": {"value": 83080728, "label": "voltagex"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-12T06:41:44Z", "updated_at": "2023-05-08T20:32:16Z", "closed_at": "2023-05-08T20:32:02Z", "author_association": "NONE", "pull_request": null, "body": "I am currently trying to import the [JSON export of my data from Discord](https://support.discord.com/hc/en-us/articles/360004027692-Requesting-a-Copy-of-your-Data), specifically `activity/reporting/events-*.json`\r\n\r\n```\r\nsqlite-utils.exe insert test.db reporting events-2023-00000-of-00001.json\r\n [###################################-] 99% 00:00:00\r\nError: Invalid JSON - use --csv for CSV or --tsv for TSV files\r\n```\r\n\r\nPlease show more information as to *why* this is invalid, if possible.\r\n\r\nI am using version 3.30 with Python 3.10 on Windows 11.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/532/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1695428235, "node_id": "I_kwDOCGYnMM5lDi6L", "number": 538, "title": "`table.upsert_all` fails to write rows when `not_null` is present", "user": {"value": 1231935, "label": "xavdid"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-05-04T07:30:38Z", "updated_at": "2023-05-08T20:06:35Z", "closed_at": "2023-05-08T19:27:02Z", "author_association": "NONE", "pull_request": null, "body": "I found an odd bug today, where calls to `table.upsert_all` don't write rows if you include the `not_null` kwarg.\r\n\r\n## Repro Example\r\n\r\n```py\r\nfrom sqlite_utils import Database\r\n\r\ndb = Database(\"upsert-test.db\")\r\n\r\ndb[\"comments\"].upsert_all(\r\n [{\"id\": 1, \"name\": \"david\"}],\r\n pk=\"id\",\r\n not_null=[\"name\"],\r\n)\r\n\r\nassert list(db[\"comments\"].rows) # err!\r\n```\r\n\r\nThe schema is correctly created:\r\n\r\n```sql\r\nCREATE TABLE [comments] (\r\n [id] INTEGER PRIMARY KEY,\r\n [name] TEXT NOT NULL\r\n)\r\n```\r\n\r\nBut no rows are created. Removing either the `not_null` kwargs works as expected, as does an `insert_all` call.\r\n\r\n## Version Info\r\n\r\n- Python: `3.11.0`\r\n- sqlite-utils: `3.30`\r\n- sqlite: `3.39.5 2022-10-14`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/538/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1622640374, "node_id": "I_kwDOCGYnMM5gt4b2", "number": 534, "title": " ResourceWarning: unclosed file", "user": {"value": 1244826, "label": "djhenderson"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-03-14T03:02:18Z", "updated_at": "2023-05-08T19:56:29Z", "closed_at": "2023-05-08T19:56:29Z", "author_association": "NONE", "pull_request": null, "body": "Issuing either\r\n\r\n```\r\npy -Wdefault -m sqlite_utils insert dogs.db dogs dogs0.csv --csv\r\n [#############-----------------------] 36%\r\n [####################################] 100%C:\\Users\\Doug\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\sqlite_utils\\cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs0.csv' encoding='utf-8-sig'>\r\n insert_upsert_implementation(\r\nResourceWarning: Enable tracemalloc to get the object allocation traceback\r\n```\r\nor\r\n```\r\nset pythonwarnings=default\r\nsqlite-utils insert dogs.db dogs dogs0.csv --csv\r\n [#############-----------------------] 36%\r\n [####################################] 100%C:\\Users\\Doug\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\sqlite_utils\\cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs0.csv' encoding='utf-8-sig'>\r\n insert_upsert_implementation(\r\nResourceWarning: Enable tracemalloc to get the object allocation traceback\r\n```\r\n\r\nexhibits a ResourceWarning indicating that the CSV file being loaded is not closed.\r\n\r\nsqlite-utils --version\r\nsqlite-utils, version 3.30\r\npy --version\r\nPython 3.11.2\r\nWindows Version 10.0.19045 Build 19045\r\nSQLite version 3.41.0 2023-02-21 18:09:37\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/534/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1699184583, "node_id": "I_kwDOCGYnMM5lR3_H", "number": 540, "title": "sphinx.builders.linkcheck build error", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-05-07T18:37:09Z", "updated_at": "2023-05-08T04:56:13Z", "closed_at": "2023-05-07T18:42:36Z", "author_association": "OWNER", "pull_request": null, "body": "https://readthedocs.org/projects/sqlite-utils/builds/20512693/\r\n```\r\n Running Sphinx v6.2.1\r\n\r\nTraceback (most recent call last):\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py\", line 442, in load_extension\r\n mod = import_module(extname)\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/importlib/__init__.py\", line 127, in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\n File \"\", line 1014, in _gcd_import\r\n File \"\", line 991, in _find_and_load\r\n File \"\", line 975, in _find_and_load_unlocked\r\n File \"\", line 671, in _load_unlocked\r\n File \"\", line 783, in exec_module\r\n File \"\", line 219, in _call_with_frames_removed\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/builders/linkcheck.py\", line 20, in \r\n from requests import Response\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/requests/__init__.py\", line 43, in \r\n import urllib3\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/urllib3/__init__.py\", line 38, in \r\n raise ImportError(\r\nImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py\", line 280, in build_main\r\n app = Sphinx(args.sourcedir, args.confdir, args.outputdir,\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py\", line 225, in __init__\r\n self.setup_extension(extension)\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py\", line 404, in setup_extension\r\n self.registry.load_extension(self, extname)\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py\", line 445, in load_extension\r\n raise ExtensionError(__('Could not import extension %s') % extname,\r\nsphinx.errors.ExtensionError: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)\r\n\r\nExtension error:\r\nCould not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/540/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1699174055, "node_id": "I_kwDOCGYnMM5lR1an", "number": 539, "title": "`--raw-lines` option, like `--raw` for multiple lines", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-05-07T18:07:46Z", "updated_at": "2023-05-07T18:43:24Z", "closed_at": "2023-05-07T18:26:18Z", "author_association": "OWNER", "pull_request": null, "body": "I wanted to output newline-separated output of the first column of every row in the results - like `--row` but for more than one line.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/539/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1203842656, "node_id": "I_kwDOCGYnMM5HwS5g", "number": 425, "title": "`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 5, "created_at": "2022-04-13T22:16:53Z", "updated_at": "2023-04-15T20:14:58Z", "closed_at": "2022-04-13T22:48:57Z", "author_association": "OWNER", "pull_request": null, "body": "Got this error while investigating:\r\n- #421\r\n\r\nEven though I was using the `LD_PRELOAD` trick from https://til.simonwillison.net/sqlite/ld-preload to use a newer version of SQLite.\r\n\r\n_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354_", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/425/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1659525418, "node_id": "PR_kwDOCGYnMM5N35VZ", "number": 536, "title": "Add paths for homebrew on Apple silicon", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-04-08T13:34:21Z", "updated_at": "2023-04-13T01:44:43Z", "closed_at": "2023-04-13T01:44:43Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/536", "body": "Does what it says and nothing else. This is the same set of paths as Datasette uses.\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--536.org.readthedocs.build/en/536/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/536/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1620164673, "node_id": "PR_kwDOCGYnMM5L08O8", "number": 531, "title": "Add paths for homebrew on Apple silicon", "user": {"value": 25778, "label": "eyeseast"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2023-03-11T22:27:52Z", "updated_at": "2023-04-09T01:49:44Z", "closed_at": "2023-04-09T01:49:43Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/531", "body": "This also passes in the extension path when specified in GIS methods. Wherever we know an extension path, we use `db.init_spatialite(find_spatialite() or load_extension)`.\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--531.org.readthedocs.build/en/531/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/531/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1114543475, "node_id": "I_kwDOCGYnMM5CbpVz", "number": 388, "title": "Link to stable docs from older versions", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-01-26T01:55:46Z", "updated_at": "2023-03-26T23:43:12Z", "closed_at": "2022-01-26T02:00:22Z", "author_association": "OWNER", "pull_request": null, "body": "https://sqlite-utils.datasette.io/en/2.14.1/ isn't showing a link to the stable release right now.\r\n\r\nI should also apply the same fix I used for Datasette in:\r\n- https://github.com/simonw/datasette/issues/1608\r\n\r\nTIL: https://til.simonwillison.net/readthedocs/link-from-latest-to-stable", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/388/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1620516340, "node_id": "I_kwDOCGYnMM5glx30", "number": 533, "title": "ReadTheDocs error: not all arguments converted during string formatting", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2023-03-12T21:21:05Z", "updated_at": "2023-03-12T21:25:33Z", "closed_at": "2023-03-12T21:25:33Z", "author_association": "OWNER", "pull_request": null, "body": "This came up as a failure running tests for:\r\n- #531\r\n\r\nTraceback on https://readthedocs.org/projects/sqlite-utils/builds/19749348/\r\n\r\n```\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/docutils/parsers/rst/states.py\", line 889, in interpreted\r\n nodes, messages2 = role_fn(role, rawsource, text, lineno, self)\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py\", line 103, in role\r\n title = caption % part\r\nTypeError: not all arguments converted during string formatting\r\n\r\nException occurred:\r\n File \"/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py\", line 103, in role\r\n title = caption % part\r\nTypeError: not all arguments converted during string formatting\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/533/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1572766460, "node_id": "I_kwDOCGYnMM5dvoL8", "number": 524, "title": "Transformation type `--type DATETIME`", "user": {"value": 21095447, "label": "4l1fe"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 15, "created_at": "2023-02-06T15:18:42Z", "updated_at": "2023-02-15T12:10:54Z", "closed_at": "2023-02-15T12:10:54Z", "author_association": "NONE", "pull_request": null, "body": "Hey. Currently i do transformation with the type `--type TEXT`, but i noticed using the sqlalchemy based library [dataset](https://github.com/pudo/dataset) that is reading and writing differ depending on the column types `TEXT`, `DATETIME`.\r\n\r\nIs it possible to alter a column type to `DATETIME` somehow using Sqlite-Utils?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/524/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1000275035, "node_id": "PR_kwDOCGYnMM4r7n-9", "number": 327, "title": "Extract expand: Support JSON Arrays", "user": {"value": 101753, "label": "phaer"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2021-09-19T10:34:30Z", "updated_at": "2022-12-29T09:05:36Z", "closed_at": "2022-12-29T09:05:36Z", "author_association": "NONE", "pull_request": "simonw/sqlite-utils/pulls/327", "body": "Hi,\r\n\r\nI needed to extract data in JSON Arrays to normalize data imports. I've quickly hacked the following together based on #241 which refers to #239 where you, @simonw, wrote:\r\n\r\n> Could this handle lists of objects too? That would be pretty amazing - if the column has a [{...}, {...}] list in it could turn that into a many-to-many.\r\n\r\nThey way this works in my work is that many-to-many relationships are created for anything that maps to an dictionary in a list, and many-to-one relations for everything else (assumed to be scalar values). Not sure what the best approach here would be? Are many-to-one relationships are at all useful here?\r\n\r\nWhat do you think about this approach? I could try to add it to the cli interface and documentation if wanted.\r\n\r\nThanks for this awesome piece of software in any case! :sun_with_face: ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/327/reactions\", \"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1487764628, "node_id": "I_kwDOCGYnMM5YrXyU", "number": 518, "title": "flake8 ValueError: Error code '#' supplied to 'extend-ignore' option...", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 0, "created_at": "2022-12-10T01:30:24Z", "updated_at": "2022-12-10T01:36:46Z", "closed_at": "2022-12-10T01:36:46Z", "author_association": "OWNER", "pull_request": null, "body": "> `Error code '#' supplied to 'extend-ignore' option does not match '^[A-Z]{1,3}[0-9]{0,3}$'`\r\n\r\nhttps://github.com/simonw/sqlite-utils/actions/runs/3662011265/jobs/6190770361\r\n\r\nI think from this:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/e660635cea6c32f4022818380b1e1ee88e7c93a6/setup.cfg#L1-L3\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/518/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1487757143, "node_id": "I_kwDOCGYnMM5YrV9X", "number": 517, "title": "Drop support for Python 3.6", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-12-10T01:23:31Z", "updated_at": "2022-12-10T01:36:36Z", "closed_at": "2022-12-10T01:36:36Z", "author_association": "OWNER", "pull_request": null, "body": "CI has started failing for Python 3.6: https://github.com/simonw/sqlite-utils/actions/runs/3576322798\r\n\r\n\"image\"\r\n\r\nIt's fixable by swiching away from `ubuntu-latest` according to:\r\n\r\n- https://github.com/actions/setup-python/issues/355#issuecomment-1335042510\r\n\r\nBut https://endoflife.date/python says that 3.6 end of life was almost 6 years ago, and end of security support nearly 1 year ago.\r\n\r\nSo I'm OK dropping support entirely - Python 3.6 users will still be able to install version 3.30, just not any releases that come next.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/517/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1434911255, "node_id": "I_kwDOCGYnMM5VhwIX", "number": 510, "title": "Cannot enable FTS5 despite it being available", "user": {"value": 1176293, "label": "ar-jan"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-11-03T16:03:49Z", "updated_at": "2022-11-18T18:37:52Z", "closed_at": "2022-11-17T10:36:28Z", "author_association": "NONE", "pull_request": null, "body": "When I do `sqlite-utils enable-fts my.db table_name column_name` (with or without `--fts5`), I get an FTS4 virtual table instead of the expected FTS5.\r\n\r\nFTS5 is however available and Python/SQLite versions do not seem to be the issue. I can manually create the FTS5 virtual table, and then Datasette also works with it from this same Python environment.\r\n\r\n`>>> sqlite3.version`\r\n`2.6.0`\r\n`>>> sqlite3.sqlite_version`\r\n`3.39.4`\r\n\r\n`PRAGMA compile_options;` includes `ENABLE_FTS5`.\r\n\r\n`sqlite-utils, version 3.30`.\r\n\r\nAny ideas what's happening and how to fix?", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/510/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1430563092, "node_id": "PR_kwDOCGYnMM5B6_6K", "number": 508, "title": "Allow surrogates in parameters", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-10-31T22:11:49Z", "updated_at": "2022-11-17T15:11:16Z", "closed_at": "2022-10-31T22:55:36Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/508", "body": "closes #507\r\n\r\nhttps://dwheeler.com/essays/fixing-unix-linux-filenames.html\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--508.org.readthedocs.build/en/508/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/508/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1450952393, "node_id": "I_kwDOCGYnMM5We8bJ", "number": 512, "title": "mypy failures in CI", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-11-16T06:22:48Z", "updated_at": "2022-11-16T07:49:51Z", "closed_at": "2022-11-16T07:49:50Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/actions/runs/3472012235 failed on Python 3.11:\r\n\r\nTruncated output:\r\n```\r\nsqlite_utils/db.py:2467: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True\r\nsqlite_utils/db.py:2467: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase\r\nsqlite_utils/db.py:2530: error: Incompatible default for argument \"where\" (default has type \"None\", argument has type \"str\") [assignment]\r\nsqlite_utils/db.py:2530: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True\r\nsqlite_utils/db.py:2530: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase\r\nsqlite_utils/db.py:2658: error: Argument 1 to \"count_where\" of \"Queryable\" has incompatible type \"Optional[str]\"; expected \"str\" [arg-type]\r\nFound 23 errors in 1 file (checked 51 source files)\r\n```\r\nBest look at https://github.com/hauntsaninja/no_implicit_optional", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/512/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1436539554, "node_id": "I_kwDOCGYnMM5Vn9qi", "number": 511, "title": "[insert_all, upsert_all] IntegrityError: constraint failed", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-11-04T19:21:48Z", "updated_at": "2022-11-04T22:59:54Z", "closed_at": "2022-11-04T22:54:09Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "My understand is that `INSERT OR IGNORE` will ignore when inserts would cause duplicate keys so I'm not sure exactly why the error is raised from `sqlite3`.\r\n\r\n```\r\nimport argparse\r\nfrom pathlib import Path\r\n\r\nfrom xklb import db, utils\r\nfrom xklb.utils import log\r\n\r\n\r\ndef parse_args() -> argparse.Namespace:\r\n parser = argparse.ArgumentParser()\r\n parser.add_argument(\"database\")\r\n parser.add_argument(\"dbs\", nargs=\"*\")\r\n parser.add_argument(\"--upsert\")\r\n parser.add_argument(\"--db\", \"-db\", help=argparse.SUPPRESS)\r\n parser.add_argument(\"--verbose\", \"-v\", action=\"count\", default=0)\r\n args = parser.parse_args()\r\n\r\n if args.db:\r\n args.database = args.db\r\n Path(args.database).touch()\r\n args.db = db.connect(args)\r\n log.info(utils.dict_filter_bool(args.__dict__))\r\n\r\n return args\r\n\r\n\r\ndef merge_db(args, source_db):\r\n source_db = str(Path(source_db).resolve())\r\n\r\n s_db = db.connect(argparse.Namespace(database=source_db, verbose=args.verbose))\r\n for table in [s for s in s_db.table_names() if not \"_fts\" in s and not s.startswith(\"sqlite_\")]:\r\n log.info(\"[%s]: %s\", source_db, table)\r\n with s_db.conn:\r\n data = s_db[table].rows\r\n\r\n with args.db.conn:\r\n if args.upsert:\r\n args.db[table].upsert_all(data, pk=args.upsert.split(\",\"), alter=True)\r\n else:\r\n args.db[table].insert_all(data, alter=True, replace=True)\r\n\r\n\r\ndef merge_dbs():\r\n args = parse_args()\r\n for s_db in args.dbs:\r\n merge_db(args, s_db)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n merge_dbs()\r\n\r\n```\r\n\r\n```\r\n$ lb-dev merge video.db tube_71.db --upsert path -vv\r\nSQL: INSERT OR IGNORE INTO [media]([path]) VALUES(?); - params: ['https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz']\r\n...\r\nFile ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:3122, in Table.insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, hash_id_columns, alter, ignore, replace, truncate, extracts, conversions, columns, upsert, analyze)\r\n 3116 all_columns += [\r\n 3117 column for column in record if column not in all_columns\r\n 3118 ]\r\n 3120 first = False\r\n-> 3122 self.insert_chunk(\r\n 3123 alter,\r\n 3124 extracts,\r\n 3125 chunk,\r\n 3126 all_columns,\r\n 3127 hash_id,\r\n 3128 hash_id_columns,\r\n 3129 upsert,\r\n 3130 pk,\r\n 3131 conversions,\r\n 3132 num_records_processed,\r\n 3133 replace,\r\n 3134 ignore,\r\n 3135 )\r\n 3137 if analyze:\r\n 3138 self.analyze()\r\n\r\nFile ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:2887, in Table.insert_chunk(self, alter, extracts, chunk, all_columns, hash_id, hash_id_columns, upsert, pk, conversions, num_records_processed, replace, ignore)\r\n 2885 for query, params in queries_and_params:\r\n 2886 try:\r\n-> 2887 result = self.db.execute(query, params)\r\n 2888 except OperationalError as e:\r\n 2889 if alter and (\" column\" in e.args[0]):\r\n 2890 # Attempt to add any missing columns, then try again\r\n\r\nFile ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:484, in Database.execute(self, sql, parameters)\r\n 482 self._tracer(sql, parameters)\r\n 483 if parameters is not None:\r\n--> 484 return self.conn.execute(sql, parameters)\r\n 485 else:\r\n 486 return self.conn.execute(sql)\r\n\r\nIntegrityError: constraint failed\r\n> /home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py(484)execute()\r\n 482 self._tracer(sql, parameters)\r\n 483 if parameters is not None:\r\n--> 484 return self.conn.execute(sql, parameters)\r\n 485 else:\r\n 486 return self.conn.execute(sql)\r\n```\r\n\r\n```\r\nsqlite3 --version\r\n3.36.0 2021-06-18 18:36:39\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/511/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 473083260, "node_id": "MDU6SXNzdWU0NzMwODMyNjA=", "number": 50, "title": "\"Too many SQL variables\" on large inserts", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2019-07-25T21:43:31Z", "updated_at": "2022-11-04T14:38:36Z", "closed_at": "2019-07-28T11:59:33Z", "author_association": "OWNER", "pull_request": null, "body": "Reported here: https://github.com/dogsheep/healthkit-to-sqlite/issues/9\r\n\r\nIt looks like there's a default limit of 999 variables - we need to be smart about that, maybe dynamically lower the batch size based on the number of columns.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/50/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1429029604, "node_id": "I_kwDOCGYnMM5VLULk", "number": 506, "title": "Make `cursor.rowcount` accessible (wontfix)", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-10-30T21:51:55Z", "updated_at": "2022-11-01T17:37:47Z", "closed_at": "2022-11-01T17:37:13Z", "author_association": "OWNER", "pull_request": null, "body": "In building this Datasette feature on top of `sqlite-utils` I thought it might be useful to expose the number of rows that had been affected by a bulk insert or update - the `cursor.rowcount`:\r\n\r\n- https://github.com/simonw/datasette/issues/1866\r\n\r\nThis isn't currently exposed by `sqlite-utils`.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/506/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1430325103, "node_id": "I_kwDOCGYnMM5VQQdv", "number": 507, "title": "conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-10-31T18:49:51Z", "updated_at": "2022-11-01T00:40:17Z", "closed_at": "2022-11-01T00:40:16Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "I'm not really sure what caused this and it happened in the middle of my program (after running for 35775 seconds).\r\n\r\n```\r\nExtracting metadata 49.9% (chunk 9893 of 19831)\r\n...\r\n File \"/home/xk/.local/lib/python3.10/site-packages/xklb/fs_extract.py\", line 90, in extract_chunk\r\n args.db[\"media\"].insert_all(utils.list_dict_filter_bool(media), pk=\"path\", alter=True, replace=True)\r\n File \"/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py\", line 3107, in insert_all\r\n self.insert_chunk(\r\n File \"/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py\", line 2872, in insert_chunk\r\n result = self.db.execute(query, params)\r\n File \"/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py\", line 483, in execute\r\n return self.conn.execute(sql, parameters)\r\nUnicodeEncodeError: 'utf-8' codec can't encode character '\\udcc3' in position 62: surrogates not allowed\r\n```\r\n\r\nThis might be relevant: https://stackoverflow.com/questions/31898353/python-cant-encode-with-surrogateescape\r\n\r\nI'm going to try re-running with \r\n\r\n```py\r\n def execute(\r\n self, sql: str, parameters: Optional[Union[Iterable, dict]] = None\r\n ) -> sqlite3.Cursor:\r\n \"\"\"\r\n Execute SQL query and return a ``sqlite3.Cursor``.\r\n\r\n :param sql: SQL query to execute\r\n :param parameters: Parameters to use in that query - an iterable for ``where id = ?``\r\n parameters, or a dictionary for ``where id = :id``\r\n \"\"\"\r\n try:\r\n if self._tracer:\r\n self._tracer(sql, parameters)\r\n if parameters is not None:\r\n return self.conn.execute(sql, parameters)\r\n else:\r\n return self.conn.execute(sql)\r\n except UnicodeEncodeError:\r\n sql = sql.encode('utf-8', 'surrogatepass').decode('utf-8')\r\n if parameters is not None:\r\n parameters = parameters.encode('utf-8', 'surrogatepass').decode('utf-8')\r\n return self.execute(sql, parameters)\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/507/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1423182778, "node_id": "I_kwDOCGYnMM5U1Au6", "number": 505, "title": "Release sqlite-utils 3.30", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-10-25T22:20:05Z", "updated_at": "2022-10-25T22:41:26Z", "closed_at": "2022-10-25T22:41:16Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/compare/3.29...defa2974c6d3abc19be28d6b319649b8028dc966", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/505/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1392690202, "node_id": "I_kwDOCGYnMM5TAsQa", "number": 495, "title": "Support JSON values returned from .convert() functions", "user": {"value": 649467, "label": "mhalle"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 3, "created_at": "2022-09-30T16:33:49Z", "updated_at": "2022-10-25T21:23:37Z", "closed_at": "2022-10-25T21:23:28Z", "author_association": "NONE", "pull_request": null, "body": "When using the convert function on a JSON column, the result of the conversion function must be a string. If the return value is either a dict (object) or a list (array), the convert call will error out with an unhelpful user defined function exception. \r\n\r\nIt makes sense that since the original column value was a string and required conversion to data structures, the result should be converted back into a JSON string as well. However, other functions auto-convert to JSON string representation, so the fact that convert doesn't could be surprising.\r\n\r\nAt least the documentation should note this requirement, because the sqlite error messages won't readily reveal the issue.\r\n\r\nJf only sqlite's JSON column type meant something :)", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/495/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1393212964, "node_id": "I_kwDOCGYnMM5TCr4k", "number": 497, "title": "column_names", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-10-01T03:34:21Z", "updated_at": "2022-10-25T21:09:28Z", "closed_at": "2022-10-25T21:09:28Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "It would be nice to have a `column_names`. Similar to `table_names`.\r\n\r\nOr if you could get one or all of the following syntax to work for both Database and Table that might be even better: \r\n\r\nStyle 1\r\n- `if 'table1' in db`\r\n- `if 'col1' in db['table1']`\r\n\r\nStyle 2\r\n- `if 'table1' in db.tables`\r\n- `if 'col1' in db['table1'].columns`\r\n\r\nmaybe the table ones actually work but I'm too lazy to check. I just know that I have to do:\r\n\r\n `[c.name for c in db['table1'].columns]`\r\n\r\nEdit: This is possible with `columns_dict`. I have actually used that before but I forgot about it. Feel free to close, but I do think accessing this data could be more consistent and intuitive.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/497/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1423069384, "node_id": "I_kwDOCGYnMM5U0lDI", "number": 504, "title": "db.close() method, calling db.conn.close()", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-10-25T20:50:50Z", "updated_at": "2022-10-25T21:00:29Z", "closed_at": "2022-10-25T20:57:47Z", "author_association": "OWNER", "pull_request": null, "body": "I ended up needing to use `db.conn.close()` to fix this issue:\r\n- #503\r\n\r\nI think `.close()` should be a method on `Database` itself.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/504/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1423000702, "node_id": "I_kwDOCGYnMM5U0UR-", "number": 503, "title": "test_recreate failing on Windows Python 3.11", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 10, "created_at": "2022-10-25T20:01:41Z", "updated_at": "2022-10-25T20:47:34Z", "closed_at": "2022-10-25T20:45:43Z", "author_association": "OWNER", "pull_request": null, "body": "https://github.com/simonw/sqlite-utils/actions/runs/3323672128/jobs/5494726927\r\n\r\nRelated:\r\n- #502\r\n\r\n```\r\nFAILED tests/test_recreate.py::test_recreate[True-True] - \r\n PermissionError: [WinError 32] The process cannot access the file because it is being used by another process:\r\n 'C:\\\\Users\\\\runneradmin\\\\AppData\\\\Local\\\\Temp\\\\pytest-of-runneradmin\\\\pytest-0\\\\test_recreate_True_True_0\\\\data.db'\r\nFAILED tests/test_recreate.py::test_recreate[False-True] - \r\n PermissionError: [WinError 32] The process cannot access the file because it is being used by another process:\r\n 'C:\\\\Users\\\\runneradmin\\\\AppData\\\\Local\\\\Temp\\\\pytest-of-runneradmin\\\\pytest-0\\\\test_recreate_False_True_0\\\\data.db'\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/503/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1422954582, "node_id": "I_kwDOCGYnMM5U0JBW", "number": 502, "title": "Fix tests for Python 3.11", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-10-25T19:20:31Z", "updated_at": "2022-10-25T19:23:47Z", "closed_at": "2022-10-25T19:23:47Z", "author_association": "OWNER", "pull_request": null, "body": "The way errors are represented has changed: https://github.com/simonw/sqlite-utils/actions/runs/3323588047/jobs/5494127154\r\n```\r\n_________________________ test_query_invalid_function __________________________\r\n\r\ndb_path = '/tmp/pytest-of-runner/pytest-0/test_query_invalid_function0/test.db'\r\n\r\n def test_query_invalid_function(db_path):\r\n result = CliRunner().invoke(\r\n cli.cli, [db_path, \"select bad()\", \"--functions\", \"def invalid_python\"]\r\n )\r\n assert result.exit_code == 1\r\n> assert (\r\n result.output.strip()\r\n == \"Error: Error in functions definition: invalid syntax (, line 1)\"\r\n )\r\nE AssertionError: assert 'Error: Error...ing>, line 1)' == 'Error: Error...ing>, line 1)'\r\nE - Error: Error in functions definition: invalid syntax (, line 1)\r\nE ? ^^^^^^ ^^^^^^\r\nE + Error: Error in functions definition: expected '(' (, line 1)\r\nE ? ^^^^^^^ ^^^\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/502/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1413641049, "node_id": "I_kwDOCGYnMM5UQnNZ", "number": 501, "title": "Tests failing due to updated tabulate library", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-10-18T18:07:52Z", "updated_at": "2022-10-18T18:23:40Z", "closed_at": "2022-10-18T18:23:40Z", "author_association": "OWNER", "pull_request": null, "body": "Failure here: https://github.com/simonw/sqlite-utils/actions/runs/3275786702/jobs/5391063221\r\n\r\nI figured out the problem:\r\n\r\n```diff\r\ndiff --git a/docs/cli-reference.rst b/docs/cli-reference.rst\r\nindex b88e38a..82b4b6c 100644\r\n--- a/docs/cli-reference.rst\r\n+++ b/docs/cli-reference.rst\r\n@@ -112,11 +112,15 @@ See :ref:`cli_query`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst,\r\n- simple, textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira,\r\n+ latex, latex_booktabs, latex_longtable, latex_raw,\r\n+ mediawiki, mixed_grid, mixed_outline, moinmoin,\r\n+ orgtbl, outline, pipe, plain, presto, pretty,\r\n+ psql, rounded_grid, rounded_outline, rst, simple,\r\n+ simple_grid, simple_outline, textile, tsv,\r\n+ unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not\r\n escaped strings\r\n -r, --raw Raw output, first column of first row\r\n@@ -176,11 +180,15 @@ See :ref:`cli_memory`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst,\r\n- simple, textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira,\r\n+ latex, latex_booktabs, latex_longtable, latex_raw,\r\n+ mediawiki, mixed_grid, mixed_outline, moinmoin,\r\n+ orgtbl, outline, pipe, plain, presto, pretty,\r\n+ psql, rounded_grid, rounded_outline, rst, simple,\r\n+ simple_grid, simple_outline, textile, tsv,\r\n+ unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not\r\n escaped strings\r\n -r, --raw Raw output, first column of first row\r\n@@ -401,11 +409,14 @@ See :ref:`cli_search`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --load-extension TEXT Path to SQLite extension, with optional :entrypoint\r\n@@ -651,11 +662,14 @@ See :ref:`cli_tables`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --columns Include list of columns for each table\r\n@@ -689,11 +703,14 @@ See :ref:`cli_views`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --columns Include list of columns for each view\r\n@@ -732,11 +749,15 @@ See :ref:`cli_rows`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst,\r\n- simple, textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira,\r\n+ latex, latex_booktabs, latex_longtable, latex_raw,\r\n+ mediawiki, mixed_grid, mixed_outline, moinmoin,\r\n+ orgtbl, outline, pipe, plain, presto, pretty,\r\n+ psql, rounded_grid, rounded_outline, rst, simple,\r\n+ simple_grid, simple_outline, textile, tsv,\r\n+ unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not\r\n escaped strings\r\n --load-extension TEXT Path to SQLite extension, with optional\r\n@@ -768,11 +789,14 @@ See :ref:`cli_triggers`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --load-extension TEXT Path to SQLite extension, with optional :entrypoint\r\n@@ -804,11 +828,14 @@ See :ref:`cli_indexes`.\r\n --tsv Output TSV\r\n --no-headers Omit CSV headers\r\n -t, --table Output as a formatted table\r\n- --fmt TEXT Table format - one of fancy_grid, fancy_outline,\r\n- github, grid, html, jira, latex, latex_booktabs,\r\n- latex_longtable, latex_raw, mediawiki, moinmoin,\r\n- orgtbl, pipe, plain, presto, pretty, psql, rst, simple,\r\n- textile, tsv, unsafehtml, youtrack\r\n+ --fmt TEXT Table format - one of asciidoc, double_grid,\r\n+ double_outline, fancy_grid, fancy_outline, github,\r\n+ grid, heavy_grid, heavy_outline, html, jira, latex,\r\n+ latex_booktabs, latex_longtable, latex_raw, mediawiki,\r\n+ mixed_grid, mixed_outline, moinmoin, orgtbl, outline,\r\n+ pipe, plain, presto, pretty, psql, rounded_grid,\r\n+ rounded_outline, rst, simple, simple_grid,\r\n+ simple_outline, textile, tsv, unsafehtml, youtrack\r\n --json-cols Detect JSON cols and output them as JSON, not escaped\r\n strings\r\n --load-extension TEXT Path to SQLite extension, with optional :entrypoint\r\ndiff --git a/docs/cli.rst b/docs/cli.rst\r\nindex 8bc4176..1d67e88 100644\r\n--- a/docs/cli.rst\r\n+++ b/docs/cli.rst\r\n@@ -187,10 +187,15 @@ Available ``--fmt`` options are:\r\n cog.out(\"\\n\" + \"\\n\".join('- ``{}``'.format(t) for t in tabulate.tabulate_formats) + \"\\n\\n\")\r\n .. ]]]\r\n \r\n+- ``asciidoc``\r\n+- ``double_grid``\r\n+- ``double_outline``\r\n - ``fancy_grid``\r\n - ``fancy_outline``\r\n - ``github``\r\n - ``grid``\r\n+- ``heavy_grid``\r\n+- ``heavy_outline``\r\n - ``html``\r\n - ``jira``\r\n - ``latex``\r\n@@ -198,15 +203,22 @@ Available ``--fmt`` options are:\r\n - ``latex_longtable``\r\n - ``latex_raw``\r\n - ``mediawiki``\r\n+- ``mixed_grid``\r\n+- ``mixed_outline``\r\n - ``moinmoin``\r\n - ``orgtbl``\r\n+- ``outline``\r\n - ``pipe``\r\n - ``plain``\r\n - ``presto``\r\n - ``pretty``\r\n - ``psql``\r\n+- ``rounded_grid``\r\n+- ``rounded_outline``\r\n - ``rst``\r\n - ``simple``\r\n+- ``simple_grid``\r\n+- ``simple_outline``\r\n - ``textile``\r\n - ``tsv``\r\n - ``unsafehtml``\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/501/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1413610718, "node_id": "I_kwDOCGYnMM5UQfze", "number": 500, "title": "Turn --flatten into a documented utility function", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-10-18T17:43:36Z", "updated_at": "2022-10-18T18:02:10Z", "closed_at": "2022-10-18T18:00:40Z", "author_association": "OWNER", "pull_request": null, "body": "The `--flatten` implementation isn't currently available to Python code - people have to roll their own implementation. Feedback from a conversation at DjangoCon.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/500/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1404013495, "node_id": "PR_kwDOCGYnMM5AicIh", "number": 498, "title": "fix: enable-fts permanently save triggers", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-10-11T05:10:51Z", "updated_at": "2022-10-15T04:33:08Z", "closed_at": "2022-10-11T06:34:31Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/498", "body": "I was wondering why my all my databases were giving wild search results. Turns out create_trigger was not sticking!\r\n\r\nRunning `sqlite-utils triggers x.db` shows `[]` after running `enable-fts` using the python api. Looking at the counts trigger it seems that is the right way to save triggers. triggers show up now\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--498.org.readthedocs.build/en/498/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/498/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1361355564, "node_id": "I_kwDOCGYnMM5RJKMs", "number": 482, "title": "balanced table default column_order", "user": {"value": 7908073, "label": "chapmanjacobd"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-09-05T03:00:18Z", "updated_at": "2022-10-10T17:43:02Z", "closed_at": "2022-09-06T20:17:27Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "Is there any performance or size difference with column order in SQLITE ? similar to this https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/\r\n\r\nIt might be interesting to have an option to create with an optimized column order. I'm assuming this would look something like INTEGER columns, REAL columns, BLOB columns, TEXT columns, NULL columns. NULL columns at the end because they are more likely to be TEXT and it is impossible to know if they will become INTEGER\r\n\r\n(Of course, any schema evolution would reduce optimization but maybe column order could also be re-evaluated when schema changes)\r\n\r\nedit:\r\n\r\nthis is easy to accomplish with the existing `transform` method:\r\n\r\n```\r\nint_columns = [k for k, v in table_columns.items() if v == int]\r\ndb[table].transform(column_order=[*int_columns])\r\n```", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/482/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1386593843, "node_id": "I_kwDOCGYnMM5Spb4z", "number": 494, "title": "Document how to use Just", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-09-26T19:25:12Z", "updated_at": "2022-09-26T19:32:36Z", "closed_at": "2022-09-26T19:26:39Z", "author_association": "OWNER", "pull_request": null, "body": "I'm using `just` a lot know, based on this file - I should add that to https://sqlite-utils.datasette.io/en/latest/contributing.html\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/afbd2b2cba45cccb305c3d4638d18db4dd3d4bbd/Justfile#L1-L24", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/494/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1363765916, "node_id": "I_kwDOCGYnMM5RSWqc", "number": 483, "title": "`sqlite-utils install` command", "user": {"value": 9599, "label": "simonw"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 2, "created_at": "2022-09-06T20:13:55Z", "updated_at": "2022-09-26T19:04:43Z", "closed_at": "2022-09-26T18:57:15Z", "author_association": "OWNER", "pull_request": null, "body": "With the addition of `--functions` in:\r\n- #471\r\n\r\nIn addition to the existing `convert` command, there are now very good reasons to want to install additional packages into the same virtual environment as `sqlite-utils` itself, to allow them to be used with those features.\r\n\r\nThis isn't easy if you installed the tool with `pipx` or `brew install sqlite-utils`.\r\n\r\nDatasette solved this problem with the `datasette install` command:\r\n\r\n- https://github.com/simonw/datasette/issues/925\r\n\r\n`sqlite-utils` could benefit from the same idea.", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/483/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1382457780, "node_id": "I_kwDOCGYnMM5SZqG0", "number": 490, "title": "Ability to insert multi-line files", "user": {"value": 6180701, "label": "jeqo"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 4, "created_at": "2022-09-22T13:29:22Z", "updated_at": "2022-09-26T18:24:44Z", "closed_at": "2022-09-23T16:37:58Z", "author_association": "NONE", "pull_request": null, "body": "I was looking into how to parse application log files that contain multiline text (e.g. Java stack traces) into sqlite. \r\nI can see that at the moment `--lines` helps, but falls short when processing multi-line texts.\r\n\r\nI wonder if this functionality would be useful for sqlite-utils. A similar approach to Elastic logstash/filebeat can be adopted: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html \r\n\r\nPotential changes:\r\n\r\n- add a `--multiline` option\r\n- additional properties for\r\n - multiline-pattern (regex expression)\r\n - multiline-negate: true/false\r\n - multiline-what: previous or next\r\n\r\nOr if this is achievable in a different way, please share. Thanks!", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/490/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"} {"id": 1366512990, "node_id": "PR_kwDOCGYnMM4-nBs9", "number": 486, "title": "progressbar for inserts/upserts of all fileformats, closes #485", "user": {"value": 99098079, "label": "MischaU8"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 7, "created_at": "2022-09-08T14:58:02Z", "updated_at": "2022-09-15T20:40:03Z", "closed_at": "2022-09-15T20:37:51Z", "author_association": "CONTRIBUTOR", "pull_request": "simonw/sqlite-utils/pulls/486", "body": "\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--486.org.readthedocs.build/en/486/\n\r\n", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/486/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null} {"id": 1366423176, "node_id": "I_kwDOCGYnMM5RcfaI", "number": 485, "title": "Progressbar not shown when inserting/upserting jsonlines file", "user": {"value": 99098079, "label": "MischaU8"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2022-09-08T14:13:18Z", "updated_at": "2022-09-15T20:39:52Z", "closed_at": "2022-09-15T20:37:52Z", "author_association": "CONTRIBUTOR", "pull_request": null, "body": "When inserting or upserting a jsonlines file, no progressbar is shown. Expected behavior is that, just like with .csv/.tsv files, also for a jsonlines file (--nl), unless --silent is provided, a progressbar is shown.\r\n\r\n```bash\r\nsql-utils upsert mydb.db posts posts.jl --nl --pk post_id\r\n(silence)\r\n```\r\n\r\nCurrently `file_progress` is only called within the tsv/csv logic, however I think it can be safely wrapped around all the all the input formats that use `decoded`: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py#L963", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/485/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}