\r\n>>> db[\"http://example.com\"].enable_fts([\"name\"])\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n db[\"http://example.com\"].enable_fts([\"name\"])\r\n File \"/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py\", l\r\nine 705, in enable_fts\r\n self.populate_fts(columns)\r\n File \"/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py\", l\r\nine 715, in populate_fts\r\n self.db.conn.executescript(sql)\r\nsqlite3.OperationalError: unrecognized token: \":\"\r\n>>> \r\n```", "created_at": "2019-09-01T06:29:05Z", "updated_at": "2019-09-02T17:23:21Z", "closed_at": "2019-09-02T17:23:21Z", "merged_at": null, "merge_commit_sha": "79852e97ecb69b88da87da0cba2a55887cf83bda", "assignee": null, "milestone": null, "draft": 0, "head": "83ca4c802f5d5102e73ff366e61514ded81dc7a1", "base": "cb70f7d10996b844154bf3da88779dd1f65590bc", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/56", "merged_by": null, "auto_merge": null}
{"id": 313105634, "node_id": "MDExOlB1bGxSZXF1ZXN0MzEzMTA1NjM0", "number": 57, "state": "closed", "locked": 0, "title": "Add triggers while enabling FTS", "user": {"value": 49260, "label": "amjith"}, "body": "This adds the option for a user to set up triggers in the database to keep their FTS table in sync with the parent table. \r\n\r\nRef: https://sqlite.org/fts5.html#external_content_and_contentless_tables\r\n\r\nI would prefer to make the creation of triggers the default behavior, but that will break existing usage where people have been calling `populate_fts` after inserting new rows.\r\n\r\nI am happy to make changes to the PR as you see fit. ", "created_at": "2019-09-02T04:23:40Z", "updated_at": "2019-09-03T01:03:59Z", "closed_at": "2019-09-02T23:42:29Z", "merged_at": "2019-09-02T23:42:29Z", "merge_commit_sha": "405e092d5916e70df10f82d15e9c052aa9ee8d80", "assignee": null, "milestone": null, "draft": 0, "head": "e01943271b17115fbe0e81d523126d2fb1c7c24b", "base": "cb70f7d10996b844154bf3da88779dd1f65590bc", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/57", "merged_by": null, "auto_merge": null}
{"id": 249680944, "node_id": "MDExOlB1bGxSZXF1ZXN0MjQ5NjgwOTQ0", "number": 9, "state": "closed", "locked": 0, "title": ":pencil: Updates my_database.py to my_database.db", "user": {"value": 50527, "label": "jefftriplett"}, "body": "I noticed that both `.py` and `.db` were used in the docs and assumed you'd prefer `.db`. ", "created_at": "2019-02-01T17:35:43Z", "updated_at": "2019-02-24T03:55:04Z", "closed_at": "2019-02-24T03:55:04Z", "merged_at": "2019-02-24T03:55:04Z", "merge_commit_sha": "c5068a0972651b3e359ebc2d6c1486b8b7d2c242", "assignee": null, "milestone": null, "draft": 0, "head": "1ad604fbbd3311f041357190796a3613c0c729d1", "base": "441c131db5cc68e197db19f0623ff8a96c90c3ff", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/9", "merged_by": null, "auto_merge": null}
{"id": 1542299634, "node_id": "PR_kwDOCGYnMM5b7Z_y", "number": 598, "state": "closed", "locked": 0, "title": "Fixed issue #433 - CLI eats cursor", "user": {"value": 62745, "label": "spookylukey"}, "body": "The issue is that underlying iterator is not fully consumed within the body of the `with file_progress()` block. Instead, that block creates generator expressions like `docs = (dict(zip(headers, row)) for row in reader)`\r\n\r\nThese iterables are consumed later, outside the `with file_progress()` block, which consumes the underlying iterator, and in turn updates the progress bar.\r\n\r\nThis means that the `ProgressBar.__exit__` method gets called before the last time the `ProgressBar.update` method gets called. The result is that the code to make the cursor invisible (inside the `update()` method) is called after the cleanup code to make it visible (in the `__exit__` method).\r\n\r\nThe fix is to move consumption of the `docs` iterators within the progress bar block. (\r\n\r\n(An additional fix, to make ProgressBar more robust against this kind of misuse, would to make it refusing to update after its `__exit__` method had been called, just like files cannot be `read()` after they are closed. That requires a in the click library).\r\n\r\nNote that Github diff obscures the simplicity of this diff, it's just indenting a block of code.\r\n\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--598.org.readthedocs.build/en/598/\n\r\n", "created_at": "2023-10-04T18:06:58Z", "updated_at": "2023-11-04T00:46:55Z", "closed_at": "2023-11-04T00:40:30Z", "merged_at": "2023-11-04T00:40:30Z", "merge_commit_sha": "37273d7f63f08872aa1c90c4233a0580e384ac19", "assignee": null, "milestone": null, "draft": 0, "head": "76113d1cb1387811d556fb7a32e0628492c34aea", "base": "622c3a5a7dd53a09c029e2af40c2643fe7579340", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/598", "merged_by": null, "auto_merge": null}
{"id": 529090560, "node_id": "MDExOlB1bGxSZXF1ZXN0NTI5MDkwNTYw", "number": 204, "state": "closed", "locked": 0, "title": "use jsonify_if_need for sql updates", "user": {"value": 78035, "label": "mfa"}, "body": "", "created_at": "2020-11-29T10:49:00Z", "updated_at": "2020-12-08T17:49:42Z", "closed_at": "2020-12-08T17:49:42Z", "merged_at": "2020-12-08T17:49:42Z", "merge_commit_sha": "c5f4f0f70ce394dfec6054c3c5aaedf330887093", "assignee": null, "milestone": null, "draft": 0, "head": "8ea9ad1913aa63395391818ebb4e43bfce936dda", "base": "60d3c4821be4cf25c41097c1e8b79b2e60c5ead5", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/204", "merged_by": null, "auto_merge": null}
{"id": 445023326, "node_id": "MDExOlB1bGxSZXF1ZXN0NDQ1MDIzMzI2", "number": 118, "state": "closed", "locked": 0, "title": "Add insert --truncate option", "user": {"value": 79913, "label": "tsibley"}, "body": "\r\n\r\nDeletes all rows in the table (if it exists) before inserting new rows.\r\nSQLite doesn't implement a TRUNCATE TABLE statement but does optimize an\r\nunqualified DELETE FROM.\r\n\r\nThis can be handy if you want to refresh the entire contents of a table\r\nbut a) don't have a PK (so can't use --replace), b) don't want the table\r\nto disappear (even briefly) for other connections, and c) have to handle\r\nrecords that used to exist being deleted.\r\n\r\nIdeally the replacement of rows would appear instantaneous to other\r\nconnections by putting the DELETE + INSERT in a transaction, but this is\r\nvery difficult without breaking other code as the current transaction\r\nhandling is inconsistent and non-systematic. There exists the\r\npossibility for the DELETE to succeed but the INSERT to fail, leaving an\r\nempty table. This is not much worse, however, than the current\r\npossibility of one chunked INSERT succeeding and being committed while\r\nthe next chunked INSERT fails, leaving a partially complete operation.", "created_at": "2020-07-06T21:58:40Z", "updated_at": "2020-07-08T17:26:21Z", "closed_at": "2020-07-08T17:26:21Z", "merged_at": "2020-07-08T17:26:21Z", "merge_commit_sha": "ae4593316ccf5e42ad26f27033193834a7e696c8", "assignee": null, "milestone": null, "draft": 0, "head": "332f7d770b84734dbed4842ab3ed24ee5b687889", "base": "f8277d0fb9c05a88a9ff01d996e31d55f0f0a645", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/118", "merged_by": null, "auto_merge": null}
{"id": 445833908, "node_id": "MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4", "number": 120, "state": "closed", "locked": 0, "title": "Fix query command's support for DML", "user": {"value": 79913, "label": "tsibley"}, "body": "See commit messages for details. I ran into this while investigating another feature/issue.", "created_at": "2020-07-08T01:36:34Z", "updated_at": "2020-07-08T05:14:04Z", "closed_at": "2020-07-08T05:14:04Z", "merged_at": "2020-07-08T05:14:04Z", "merge_commit_sha": "f8277d0fb9c05a88a9ff01d996e31d55f0f0a645", "assignee": null, "milestone": null, "draft": 0, "head": "6a660d12a27864d6ab552e11eef9fd13bc281198", "base": "d0cdaaaf00249230e847be3a3b393ee2689fbfe4", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/120", "merged_by": null, "auto_merge": null}
{"id": 475665984, "node_id": "MDExOlB1bGxSZXF1ZXN0NDc1NjY1OTg0", "number": 142, "state": "closed", "locked": 0, "title": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records", "user": {"value": 96218, "label": "simonwiles"}, "body": "Closes #139.", "created_at": "2020-08-28T22:22:57Z", "updated_at": "2020-08-30T07:28:23Z", "closed_at": "2020-08-28T22:30:14Z", "merged_at": "2020-08-28T22:30:14Z", "merge_commit_sha": "947bb7626fd1763608a470adf9cf5f156ef003e9", "assignee": null, "milestone": null, "draft": 0, "head": "1b992c7c1e5855caa87c7939efad287052f446f3", "base": "ea87c2b943fdd162c42a900ac0aea5ecc2f4b9d9", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/142", "merged_by": null, "auto_merge": null}
{"id": 475874493, "node_id": "MDExOlB1bGxSZXF1ZXN0NDc1ODc0NDkz", "number": 146, "state": "closed", "locked": 0, "title": "Handle case where subsequent records (after first batch) include extra columns", "user": {"value": 96218, "label": "simonwiles"}, "body": "Addresses #145.\r\n\r\nI think this should do the job. If it meets with your approval I'll update this PR to include an update to the documentation -- I came across this bug while preparing a PR to update the documentation around `batch_size` in any event.", "created_at": "2020-08-30T07:13:58Z", "updated_at": "2020-09-08T23:20:37Z", "closed_at": "2020-09-08T23:20:37Z", "merged_at": "2020-09-08T23:20:36Z", "merge_commit_sha": "e6d202b742a7b531fffa593703d34f8337632d68", "assignee": null, "milestone": null, "draft": 0, "head": "40fcea41525682fdc2f601da787b29d5a8517dcd", "base": "deb2eb013ff85bbc828ebc244a9654f0d9c3139e", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/146", "merged_by": null, "auto_merge": null}
{"id": 483027587, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgzMDI3NTg3", "number": 156, "state": "closed", "locked": 0, "title": "Typos in tests", "user": {"value": 96218, "label": "simonwiles"}, "body": "One of these is my fault, and the other is one I just happened to come across. They're harmless, but might as well be fixed.", "created_at": "2020-09-09T18:00:58Z", "updated_at": "2020-09-09T18:24:50Z", "closed_at": "2020-09-09T18:21:23Z", "merged_at": "2020-09-09T18:21:23Z", "merge_commit_sha": "367082e787101fb90901ef3214804ab23a92ce46", "assignee": null, "milestone": null, "draft": 0, "head": "b777776753ef08f35dfaa7d86b2d5082e8904f27", "base": "6be61263642d8e46ec54cf5f51af74e0df2f2393", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/156", "merged_by": null, "auto_merge": null}
{"id": 737050557, "node_id": "PR_kwDOCGYnMM4r7n-9", "number": 327, "state": "closed", "locked": 0, "title": "Extract expand: Support JSON Arrays", "user": {"value": 101753, "label": "phaer"}, "body": "Hi,\r\n\r\nI needed to extract data in JSON Arrays to normalize data imports. I've quickly hacked the following together based on #241 which refers to #239 where you, @simonw, wrote:\r\n\r\n> Could this handle lists of objects too? That would be pretty amazing - if the column has a [{...}, {...}] list in it could turn that into a many-to-many.\r\n\r\nThey way this works in my work is that many-to-many relationships are created for anything that maps to an dictionary in a list, and many-to-one relations for everything else (assumed to be scalar values). Not sure what the best approach here would be? Are many-to-one relationships are at all useful here?\r\n\r\nWhat do you think about this approach? I could try to add it to the cli interface and documentation if wanted.\r\n\r\nThanks for this awesome piece of software in any case! :sun_with_face: ", "created_at": "2021-09-19T10:34:30Z", "updated_at": "2022-12-29T09:05:36Z", "closed_at": "2022-12-29T09:05:36Z", "merged_at": null, "merge_commit_sha": "f0105cde23452cb4c8a15fc6096154b15d9b7c5a", "assignee": null, "milestone": null, "draft": 0, "head": "2840c697aa9817462d864ed5f8a7696d749fe039", "base": "8d641ab08ac449081e96f3e25bd6c0226870948a", "author_association": "NONE", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/327", "merged_by": null, "auto_merge": null}
{"id": 943686022, "node_id": "PR_kwDOCGYnMM44P4GG", "number": 437, "state": "closed", "locked": 0, "title": "docs to dogs", "user": {"value": 114388, "label": "yurivish"}, "body": "Fixes a typo.", "created_at": "2022-05-22T15:50:33Z", "updated_at": "2022-05-30T21:32:41Z", "closed_at": "2022-05-30T21:32:41Z", "merged_at": "2022-05-30T21:32:41Z", "merge_commit_sha": "9fedfc69d7239ac49900051e1c48ee9cdd470d9e", "assignee": null, "milestone": null, "draft": 0, "head": "9e540417821385939db0fdd2d0a859f85be377e6", "base": "59be60c471fd7a2c4be7f75e8911163e618ff5ca", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/437", "merged_by": null, "auto_merge": null}
{"id": 483175509, "node_id": "MDExOlB1bGxSZXF1ZXN0NDgzMTc1NTA5", "number": 158, "state": "closed", "locked": 0, "title": "Fix accidental mega long line in docs", "user": {"value": 167319, "label": "tomviner"}, "body": "", "created_at": "2020-09-09T22:31:23Z", "updated_at": "2020-09-16T06:21:43Z", "closed_at": "2020-09-16T06:21:43Z", "merged_at": "2020-09-16T06:21:43Z", "merge_commit_sha": "7805d53bcf11199bd1f2b07e05ae90151f9d0eb0", "assignee": null, "milestone": null, "draft": 0, "head": "5c1d8a5f3ac8684657eefe4f57ff5ff605f6b633", "base": "367082e787101fb90901ef3214804ab23a92ce46", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/158", "merged_by": null, "auto_merge": null}
{"id": 1234321667, "node_id": "PR_kwDOCGYnMM5JkkED", "number": 526, "state": "closed", "locked": 0, "title": "Fix repeated calls to `Table.convert()`", "user": {"value": 167893, "label": "mcarpenter"}, "body": "Fixes #525. All tests pass.\r\n\r\nThere's perhaps a better way to name lambdas? There could be a collision if a caller passes a function with name like `lambda_123456`.\r\n\r\nSQLite [documentation](https://www.sqlite.org/appfunc.html) is a little, ah, lite on function name specs. If there is a character that can be used in place of underscore in a SQLite function name that is not permitted in a Python function identifier then that could be a good way to prevent accidental collisions. (I tried dash, colon, dot, no joy).\r\n\r\nOtherwise, there is little chance of this happening and if it should happen the risk is mitigated by now throwing an exception in the case of a (name, arity) collision without `replace=True`.\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--526.org.readthedocs.build/en/526/\r\n\r\n", "created_at": "2023-02-09T00:14:49Z", "updated_at": "2023-05-08T21:56:05Z", "closed_at": "2023-05-08T21:53:58Z", "merged_at": "2023-05-08T21:53:58Z", "merge_commit_sha": "02f5c4d69d7b4baebde015c56e5bc62923f33314", "assignee": null, "milestone": null, "draft": 0, "head": "486e0cc1cd67e98c22d1f125403144cdedfa3d58", "base": "fc221f9b62ed8624b1d2098e564f525c84497969", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/526", "merged_by": null, "auto_merge": null}
{"id": 1235909998, "node_id": "PR_kwDOCGYnMM5Jqn1u", "number": 528, "state": "closed", "locked": 0, "title": "Enable `Table.convert()` on falsey values", "user": {"value": 167893, "label": "mcarpenter"}, "body": "Fixes #527\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--528.org.readthedocs.build/en/528/\n\r\n", "created_at": "2023-02-10T00:04:09Z", "updated_at": "2023-05-08T21:08:23Z", "closed_at": "2023-05-08T21:08:23Z", "merged_at": null, "merge_commit_sha": "279fb2c85c6b3fc7b593aaf4245bd3a038bff5f0", "assignee": null, "milestone": null, "draft": 0, "head": "a09df3844ab91b70cebb7b888fc07cb21dffd01c", "base": "fc221f9b62ed8624b1d2098e564f525c84497969", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/528", "merged_by": null, "auto_merge": null}
{"id": 727012996, "node_id": "MDExOlB1bGxSZXF1ZXN0NzI3MDEyOTk2", "number": 324, "state": "closed", "locked": 0, "title": "Use python-dateutil package instead of dateutils", "user": {"value": 191622, "label": "meatcar"}, "body": "While working on updating `sqlite-utils` for NixOS/Nixpkgs, I came a cross the following:\r\n\r\nIn 5ec6686153e29ae10d4921a1ad4c841f192f20e2, a new dependency was added on `dateutils` (https://pypi.org/project/dateutils/). \r\n\r\nI believe this is unintentional, and instead `python-dateutil` (https://pypi.org/project/python-dateutil/) was intended.\r\n\r\nMy reasoning is:\r\n- `python-dateutil` is imported here in [recipes.py](https://github.com/simonw/sqlite-utils/blob/5ec6686153e29ae10d4921a1ad4c841f192f20e2/sqlite_utils/recipes.py#L1) \r\n- The `mypy` `type-python-dateutil` dependency in [setup.py](https://github.com/simonw/sqlite-utils/blob/5ec6686153e29ae10d4921a1ad4c841f192f20e2/setup.py#L36)\r\n- `python-dateutil` is a dependency of `dateutils` as seen in the output in [docs/tutorial.ipynb](https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/docs/tutorial.ipynb#L43)\r\n\r\nSeems like the trailing \"s\" seems to be the source of confusion \ud83d\ude05\r\n\r\nI've swapped the dependencies out, hope this helps.", "created_at": "2021-09-03T18:31:19Z", "updated_at": "2021-11-14T23:25:40Z", "closed_at": "2021-11-14T23:25:40Z", "merged_at": "2021-11-14T23:25:40Z", "merge_commit_sha": "bc4c42d68879c710c851dba3c98deda96ca6caa8", "assignee": null, "milestone": null, "draft": 0, "head": "275ac3b282947fa5df5d18de835d5d958a778e72", "base": "77c240df56068341561e95e4a412cbfa24dc5bc7", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/324", "merged_by": null, "auto_merge": null}
{"id": 1136499802, "node_id": "PR_kwDOCGYnMM5DvZxa", "number": 515, "state": "closed", "locked": 0, "title": "upsert new rows with constraints, fixes #514", "user": {"value": 193185, "label": "cldellow"}, "body": "This fixes #514 by making the initial insert for upserts include all columns, so that new rows can be added to tables with non-pkey columns that have constraints.\r\n\r\n(aside: I'm not a python programmer. `pip`? `pipenv`? `venv`? These are mystical incantations to me. The process to set up this repo for local development and testing was _so easy_. Thank you for the excellent contributing documentation!)\r\n\r\n\r\n----\r\n:books: Documentation preview :books:: https://sqlite-utils--515.org.readthedocs.build/en/515/\r\n\r\n", "created_at": "2022-11-26T16:15:21Z", "updated_at": "2023-05-08T21:27:11Z", "closed_at": "2023-05-08T21:27:10Z", "merged_at": null, "merge_commit_sha": "c3713ef6944cbeacf36e462712cecac2176db692", "assignee": null, "milestone": null, "draft": 0, "head": "32f8173a8fe830c224e39a0a514cd12e78de7028", "base": "965ca0d5f5bffe06cc02cd7741344d1ddddf9d56", "author_association": "NONE", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/515", "merged_by": null, "auto_merge": null}
{"id": 1022525059, "node_id": "PR_kwDOCGYnMM488n6D", "number": 463, "state": "closed", "locked": 0, "title": "Use Read the Docs action v1", "user": {"value": 244656, "label": "humitos"}, "body": "Read the Docs repository was renamed from `readthedocs/readthedocs-preview` to `readthedocs/actions/`. Now, the `preview` action is under `readthedocs/actions/preview` and is tagged as `v1`\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--463.org.readthedocs.build/en/463/\n\r\n", "created_at": "2022-08-10T10:31:47Z", "updated_at": "2022-08-18T08:30:14Z", "closed_at": "2022-08-17T23:11:16Z", "merged_at": "2022-08-17T23:11:16Z", "merge_commit_sha": "83e7339255e811c62e6db8498c483c44a84d0f28", "assignee": null, "milestone": null, "draft": 0, "head": "364110b97bd1c6e0d2bc841730e5dc1a595e537d", "base": "271433fdd18e436b0a527ab899cb6f6fa67f23d0", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/463", "merged_by": null, "auto_merge": null}
{"id": 564215011, "node_id": "MDExOlB1bGxSZXF1ZXN0NTY0MjE1MDEx", "number": 225, "state": "closed", "locked": 0, "title": "fix for problem in Table.insert_all on search for columns per chunk of rows", "user": {"value": 261237, "label": "nieuwenhoven"}, "body": "Hi,\r\n\r\nI ran into a problem when trying to create a database from my Apple Healthkit data using [healthkit-to-sqlite](https://github.com/dogsheep/healthkit-to-sqlite). The program crashed because of an invalid insert statement that was generated for table `rDistanceCycling`. \r\n\r\nThe actual problem turned out to be in [sqlite-utils](https://github.com/simonw/sqlite-utils). `Table.insert_all` processes the data to be inserted in chunks of rows and checks for every chunk which columns are used, and it will collect all column names in the variable `all_columns`. The collection of columns is done using a nested list comprehension that is not completely correct. \r\n\r\nI'm using a Windows machine and had to make a few adjustments to the tests in order to be able to run them because they had a posix dependency.\r\n\r\nThanks, kind regards,\r\n\r\nFrans\r\n\r\n```\r\n# this is a (condensed) chunk of data from my Apple healthkit export that caused the problem.\r\n# the 3 last items in the chunk have additional keys: metadata_HKMetadataKeySyncVersion and metadata_HKMetadataKeySyncIdentifier\r\n\r\nchunk = [{'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '7.0.1',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>',\r\n 'unit': 'km', 'creationDate': '2020-10-10 12:29:09 +0100', 'startDate': '2020-10-10 12:29:06 +0100',\r\n 'endDate': '2020-10-10 12:29:07 +0100', 'value': '0.00518016'},\r\n {'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '7.0.1',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>',\r\n 'unit': 'km', 'creationDate': '2020-10-10 12:29:10 +0100', 'startDate': '2020-10-10 12:29:07 +0100',\r\n 'endDate': '2020-10-10 12:29:08 +0100', 'value': '0.00544049'},\r\n {'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '6.2.6',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',\r\n 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:40:50 +0100',\r\n 'endDate': '2020-07-15 16:42:49 +0100', 'value': '0.952092', 'metadata_HKMetadataKeySyncVersion': '1',\r\n 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520450.99823:616520569.99360:119'},\r\n {'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '6.2.6',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',\r\n 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:42:49 +0100',\r\n 'endDate': '2020-07-15 16:44:51 +0100', 'value': '0.848983', 'metadata_HKMetadataKeySyncVersion': '1',\r\n 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520569.99360:616520691.98826:119'},\r\n {'sourceName': 'Apple\u00c2\\xa0Watch van Frans', 'sourceVersion': '6.2.6',\r\n 'device': '<, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',\r\n 'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:44:51 +0100',\r\n 'endDate': '2020-07-15 16:46:50 +0100', 'value': '0.834403', 'metadata_HKMetadataKeySyncVersion': '1',\r\n 'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520691.98826:616520810.98305:119'}]\r\n\r\n\r\n\r\ndef all_columns_old():\r\n all_columns = [col for col in chunk[0]]\r\n all_columns += [column for record in chunk\r\n for column in record if column not in all_columns]\r\n return all_columns\r\n\r\n\r\ndef all_columns_new():\r\n all_columns = [col for col in chunk[0]]\r\n for record in chunk:\r\n all_columns += [column for column in record if column not in all_columns]\r\n return all_columns\r\n\r\n\r\n\r\nif __name__ == '__main__':\r\n from pprint import pprint\r\n\r\n print('problem: ')\r\n pprint(all_columns_old())\r\n print('\\nfix: ')\r\n pprint(all_columns_new())\r\n\r\n```\r\n", "created_at": "2021-01-29T20:16:07Z", "updated_at": "2021-02-14T21:04:13Z", "closed_at": "2021-02-14T21:04:13Z", "merged_at": null, "merge_commit_sha": "1cba965a1ddc2bd77db3bc3912aa7e8467e2fa2f", "assignee": null, "milestone": null, "draft": 0, "head": "929ea7551135df0cc2ac9d67f4fbbecf701a11f6", "base": "36dc7e3909a44878681c266b90f9be76ac749f2d", "author_association": "NONE", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/225", "merged_by": null, "auto_merge": null}
{"id": 583407989, "node_id": "MDExOlB1bGxSZXF1ZXN0NTgzNDA3OTg5", "number": 244, "state": "closed", "locked": 0, "title": "Typo in upsert example", "user": {"value": 387669, "label": "j-e-d"}, "body": "Remove extra `[`", "created_at": "2021-03-02T23:14:14Z", "updated_at": "2021-05-19T02:58:21Z", "closed_at": "2021-05-19T02:58:21Z", "merged_at": "2021-05-19T02:58:21Z", "merge_commit_sha": "328211eaca1247cd6b33a2c0a54642f87866d85b", "assignee": null, "milestone": null, "draft": 0, "head": "1899200fa5a43f638573ea2c483df62819ba53b1", "base": "09c3386f55f766b135b6a1c00295646c4ae29bec", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/244", "merged_by": null, "auto_merge": null}
{"id": 1000800283, "node_id": "PR_kwDOCGYnMM47pwAb", "number": 455, "state": "closed", "locked": 0, "title": "in extract code, check equality with IS instead of = for nulls", "user": {"value": 536941, "label": "fgregg"}, "body": "sqlite \"IS\" is equivalent to SQL \"IS NOT DISTINCT FROM\"\r\n\r\ncloses #423", "created_at": "2022-07-19T13:40:25Z", "updated_at": "2022-08-27T14:45:03Z", "closed_at": "2022-08-27T14:45:03Z", "merged_at": "2022-08-27T14:45:03Z", "merge_commit_sha": "c5f8a2eb1a81a18b52825cc649112f71fe419b12", "assignee": null, "milestone": null, "draft": 0, "head": "1b35a92e3ede76f0f29f6f8dcd899f44b2abbb02", "base": "855bce8c3823718def13e0b8928c58bf857e41b2", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/455", "merged_by": null, "auto_merge": null}
{"id": 1311438738, "node_id": "PR_kwDOCGYnMM5OKveS", "number": 537, "state": "closed", "locked": 0, "title": "Support self-referencing FKs in `Table.create`", "user": {"value": 544011, "label": "numist"}, "body": "\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--537.org.readthedocs.build/en/537/\n\r\n", "created_at": "2023-04-12T20:26:58Z", "updated_at": "2023-05-08T22:45:33Z", "closed_at": "2023-05-08T21:10:01Z", "merged_at": "2023-05-08T21:10:01Z", "merge_commit_sha": "39ef137e6760d385dc48d03eccf9b89943636fc7", "assignee": null, "milestone": null, "draft": 0, "head": "a75abeb61b91a28650d3b9933e7ec80ad0d92529", "base": "c0251cc9271260de73b4227859a51fab9b4cb745", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/537", "merged_by": null, "auto_merge": null}
{"id": 672359609, "node_id": "MDExOlB1bGxSZXF1ZXN0NjcyMzU5NjA5", "number": 277, "state": "closed", "locked": 0, "title": "add -h support closes #276", "user": {"value": 601708, "label": "mcint"}, "body": "This appears to be the [canonical solution](https://click.palletsprojects.com/en/7.x/documentation/#help-parameter-customization).", "created_at": "2021-06-17T08:08:26Z", "updated_at": "2021-06-18T14:56:59Z", "closed_at": "2021-06-18T14:56:59Z", "merged_at": "2021-06-18T14:56:59Z", "merge_commit_sha": "aa652b6afe43d2b40fabc7a513c3e68866e030a5", "assignee": null, "milestone": null, "draft": 0, "head": "abbd324a2178f09b19966ec36ab2f584d1824c59", "base": "a19ce1a4d0048d389411cfe11a5dbe4c503720e1", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/277", "merged_by": null, "auto_merge": null}
{"id": 774610166, "node_id": "PR_kwDOCGYnMM4uK5z2", "number": 337, "state": "closed", "locked": 0, "title": "Default values for `--attach` and `--param` options", "user": {"value": 771193, "label": "urbas"}, "body": "It seems that `click` 8.x uses `None` as the default value for `multiple=True` options.\r\n\r\nThis change makes the code forward-compatible with `click` 8.x.\r\n\r\nSee this build failure for more info: https://hydra.nixos.org/build/156926608", "created_at": "2021-11-05T21:57:53Z", "updated_at": "2021-11-05T22:33:03Z", "closed_at": "2021-11-05T22:33:02Z", "merged_at": null, "merge_commit_sha": "eb8bf28da1794638a5693043cd5268f506a674d3", "assignee": null, "milestone": null, "draft": 0, "head": "095fc64c5399d75d44d304571a21293d06d817f0", "base": "fda4dad23a0494890267fbe8baf179e2b56ee914", "author_association": "NONE", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/337", "merged_by": null, "auto_merge": null}
{"id": 521287994, "node_id": "MDExOlB1bGxSZXF1ZXN0NTIxMjg3OTk0", "number": 203, "state": "open", "locked": 0, "title": "changes to allow for compound foreign keys", "user": {"value": 1049910, "label": "drkane"}, "body": "Add support for compound foreign keys, as per issue #117 \r\n\r\nNot sure if this is the right approach. In particular I'm unsure about:\r\n\r\n - the new `ForeignKey` class, which replaces the namedtuple in order to ensure that `column` and `other_column` are forced into tuples. The class does the job, but doesn't feel very elegant.\r\n - I haven't rewritten `guess_foreign_table` to take account of multiple columns, so it just checks for the first column in the foreign key definition. This isn't ideal.\r\n - I haven't added any ability to the CLI to add compound foreign keys, it's only in the python API at the moment.\r\n\r\nThe PR also contains a minor related change that columns and tables are always quoted in foreign key definitions.", "created_at": "2020-11-16T00:30:10Z", "updated_at": "2023-01-25T18:47:18Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "0507a9464314f84e9e58b1931c583df51d757d7c", "assignee": null, "milestone": null, "draft": 0, "head": "5e43e31c2b9bcf6b5d1460b0f848fed019ed42a6", "base": "f1277f638f3a54a821db6e03cb980adad2f2fa35", "author_association": "FIRST_TIME_CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/203", "merged_by": null, "auto_merge": null}
{"id": 592289324, "node_id": "MDExOlB1bGxSZXF1ZXN0NTkyMjg5MzI0", "number": 245, "state": "closed", "locked": 0, "title": "Correct some typos", "user": {"value": 1076745, "label": "dbready"}, "body": "Noticed a typo in the docs and followed that up with a spellcheck. Had to bite my tongue at some of the British spellings.", "created_at": "2021-03-13T04:26:56Z", "updated_at": "2021-05-19T02:58:04Z", "closed_at": "2021-05-19T02:58:04Z", "merged_at": "2021-05-19T02:58:04Z", "merge_commit_sha": "3e62ab62a88992d4bbb0fe83debec3bacd93ebf3", "assignee": null, "milestone": null, "draft": 0, "head": "79de6b639f9d14818a5006af4f63e4e20c961a10", "base": "22f1d9e1999f70af4c5b0f880a820cd9eead3942", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/245", "merged_by": null, "auto_merge": null}
{"id": 992299943, "node_id": "PR_kwDOCGYnMM47JUun", "number": 452, "state": "closed", "locked": 0, "title": "Add duplicate table feature", "user": {"value": 1690072, "label": "davidleejy"}, "body": "This PR addresses a feature request raised in issue #449. Specifically this PR adds a functionality that lets users duplicate a table via:\r\n\r\n```python\r\ntable_new = db[\"my_table\"].duplicate(\"new_table\")\r\n```\r\n\r\nTest added in file `tests/test_duplicate.py`.\r\n\r\nHappy to make changes to meet maintainers' feedback, if any. ", "created_at": "2022-07-09T20:24:31Z", "updated_at": "2022-07-15T21:21:37Z", "closed_at": "2022-07-15T21:21:36Z", "merged_at": "2022-07-15T21:21:36Z", "merge_commit_sha": "b366e68deb0780048a23610c279552f8529d4726", "assignee": null, "milestone": null, "draft": 0, "head": "eef350fe543c6301c61b257c5f708e0e16ed5a34", "base": "42440d6345c242ee39778045e29143fb550bd2c2", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/452", "merged_by": null, "auto_merge": null}
{"id": 614727036, "node_id": "MDExOlB1bGxSZXF1ZXN0NjE0NzI3MDM2", "number": 254, "state": "closed", "locked": 0, "title": "Fix incorrect create-table cli description", "user": {"value": 1935268, "label": "robjwells"}, "body": "The description for `create-table` was duplicated from `create-index`.", "created_at": "2021-04-13T20:03:15Z", "updated_at": "2021-05-19T04:43:46Z", "closed_at": "2021-05-19T02:57:26Z", "merged_at": "2021-05-19T02:57:26Z", "merge_commit_sha": "e7b2626291040b78b9a2dbc2982ba72691fb1a0f", "assignee": null, "milestone": null, "draft": 0, "head": "cdeb7d7da77c6d1cd3554d08cb5659243cb3d55b", "base": "22f1d9e1999f70af4c5b0f880a820cd9eead3942", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/254", "merged_by": null, "auto_merge": null}
{"id": 768796296, "node_id": "PR_kwDOCGYnMM4t0uaI", "number": 333, "state": "closed", "locked": 0, "title": "Add functionality to read Parquet files.", "user": {"value": 2118708, "label": "Florents-Tselai"}, "body": "I needed this for a project of mine, and I thought it'd be useful to have it in sqlite-utils (It's also mentioned in #248 ).\r\nThe current implementation works (data is read & data types are inferred correctly.\r\nI've added a single straightforward test case, but @simonw please let me know if there are any non-obvious flags/combinations I should test too.", "created_at": "2021-10-28T23:43:19Z", "updated_at": "2021-11-25T19:47:35Z", "closed_at": "2021-11-25T19:47:35Z", "merged_at": null, "merge_commit_sha": "eda2b1f8d2670c6ca8512e3e7c0150866bd0bdc6", "assignee": null, "milestone": null, "draft": 0, "head": "50ec2e49dee3b09a48a7aef55eceaa3f752a52e7", "base": "fda4dad23a0494890267fbe8baf179e2b56ee914", "author_association": "NONE", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/333", "merged_by": null, "auto_merge": null}
{"id": 719998225, "node_id": "MDExOlB1bGxSZXF1ZXN0NzE5OTk4MjI1", "number": 322, "state": "closed", "locked": 0, "title": "Add dict type to be mapped as TEXT in sqllite", "user": {"value": 2496189, "label": "minaeid90"}, "body": "the library deal with Postgres type jsonb as dictionary, add dict type as a TEXT for mapping to sqlite\r\n\r\n", "created_at": "2021-08-25T20:54:26Z", "updated_at": "2021-11-15T00:27:40Z", "closed_at": "2021-11-15T00:27:40Z", "merged_at": "2021-11-15T00:27:40Z", "merge_commit_sha": "271b894af52eb6437ae6cd84eba9867ad8dd43f6", "assignee": null, "milestone": null, "draft": 0, "head": "69619f68c26478fdee479110e084fd22711013a3", "base": "77c240df56068341561e95e4a412cbfa24dc5bc7", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/322", "merged_by": null, "auto_merge": null}
{"id": 1535767490, "node_id": "PR_kwDOCGYnMM5bifPC", "number": 596, "state": "closed", "locked": 0, "title": "Fixes mapping for time fields related to mysql, closes #522", "user": {"value": 4420927, "label": "nezhar"}, "body": "Adds `COLUMN_TYPE_MAPPING` for `TIME` fields that are mapped as `datetime.timedelta` for MySQL and json represantation for `datetime.timedelta` in order to fix #522\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--596.org.readthedocs.build/en/596/\n\r\n", "created_at": "2023-09-29T13:41:48Z", "updated_at": "2023-11-04T00:49:50Z", "closed_at": "2023-11-04T00:49:50Z", "merged_at": "2023-11-04T00:49:50Z", "merge_commit_sha": "347fdc865e91b8d3410f49a5c9d5b499fbb594c1", "assignee": null, "milestone": null, "draft": 0, "head": "ab12125b3f7d0ef327b08d95f27c6c4cec6fdfb3", "base": "622c3a5a7dd53a09c029e2af40c2643fe7579340", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/596", "merged_by": null, "auto_merge": null}
{"id": 623755723, "node_id": "MDExOlB1bGxSZXF1ZXN0NjIzNzU1NzIz", "number": 258, "state": "closed", "locked": 0, "title": "Fixing insert from JSON containing strings with non-ascii characters \u2026", "user": {"value": 6586811, "label": "dylan-wu"}, "body": "\u2026are escaped aps unicode for lists, tuples, dicts\r\n\r\nFix of #257 ", "created_at": "2021-04-26T20:50:00Z", "updated_at": "2021-05-19T02:47:44Z", "closed_at": "2021-05-19T02:47:44Z", "merged_at": "2021-05-19T02:47:44Z", "merge_commit_sha": "6f4f9a3effeb16de0348d3cf136664f7531f498d", "assignee": null, "milestone": null, "draft": 0, "head": "27e7c1ae1913e2127361af4fc664c2061fb76627", "base": "22f1d9e1999f70af4c5b0f880a820cd9eead3942", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/258", "merged_by": null, "auto_merge": null}
{"id": 1040691996, "node_id": "PR_kwDOCGYnMM4-B7Mc", "number": 480, "state": "closed", "locked": 0, "title": "search_sql add include_rank option", "user": {"value": 7908073, "label": "chapmanjacobd"}, "body": "I haven't tested this yet but wanted to get a heads-up whether this kind of change would be useful or if I should just duplicate the function and tweak it within my code\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--480.org.readthedocs.build/en/480/\n\r\n", "created_at": "2022-08-30T09:10:29Z", "updated_at": "2022-08-31T03:40:35Z", "closed_at": "2022-08-31T03:40:35Z", "merged_at": "2022-08-31T03:40:35Z", "merge_commit_sha": "ecf1d40112e52a8f4e509c39b98caae996b7bc36", "assignee": null, "milestone": null, "draft": 0, "head": "3dbfca6fdd878ba381f83157fdae2042eeaed371", "base": "087753cd42c406f1e060c1822dcd9b5fda3d60f4", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/480", "merged_by": null, "auto_merge": null}
{"id": 1082769953, "node_id": "PR_kwDOCGYnMM5AicIh", "number": 498, "state": "closed", "locked": 0, "title": "fix: enable-fts permanently save triggers", "user": {"value": 7908073, "label": "chapmanjacobd"}, "body": "I was wondering why my all my databases were giving wild search results. Turns out create_trigger was not sticking!\r\n\r\nRunning `sqlite-utils triggers x.db` shows `[]` after running `enable-fts` using the python api. Looking at the counts trigger it seems that is the right way to save triggers. triggers show up now\r\n\r\n\r\n----\n:books: Documentation preview :books:: https://sqlite-utils--498.org.readthedocs.build/en/498/\n\r\n", "created_at": "2022-10-11T05:10:51Z", "updated_at": "2022-10-15T04:33:08Z", "closed_at": "2022-10-11T06:34:31Z", "merged_at": null, "merge_commit_sha": "ab24d21e9c2aa9c514c3b65f05cfc0bafa7621c1", "assignee": null, "milestone": null, "draft": 0, "head": "75577e595d9177177a48030a1c39573af010df51", "base": "d792dad1cf5f16525da81b1e162fb71d469995f3", "author_association": "CONTRIBUTOR", "repo": {"value": 140912432, "label": "sqlite-utils"}, "url": "https://github.com/simonw/sqlite-utils/pull/498", "merged_by": null, "auto_merge": null}