github
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
488293926 | MDU6SXNzdWU0ODgyOTM5MjY= | 58 | Support enabling FTS on views | 49260 | closed | 0 | 1 | 2019-09-02T18:56:36Z | 2020-10-16T18:39:36Z | 2020-10-16T18:39:31Z | CONTRIBUTOR | Right now enable_fts() is only implemented for Table(). Technically sqlite supports enabling fts on views. But it requires deeper thought since views don't have `rowid` and the current implementation of enable_fts() relies on the presence of `rowid` column. It is possible to provide an alternative rowid using the `content_rowid` option to the FTS5() function. Ref: https://sqlite.org/fts5.html#fts5_table_creation_and_initialization > The "content_rowid" option, used to set the rowid field of an external content table. This will further complicate `enable_fts()` function by adding an extra argument. I'm wondering if that is outside the scope of this tool or should I work on that feature and send a PR? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/58/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
546078359 | MDExOlB1bGxSZXF1ZXN0MzU5ODIyNzcz | 75 | Explicitly include tests and docs in sdist | 15092 | closed | 0 | 1 | 2020-01-07T04:53:20Z | 2020-01-31T00:21:27Z | 2020-01-31T00:21:27Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/75 | Also exclude 'tests' from runtime installation. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/75/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
652816158 | MDExOlB1bGxSZXF1ZXN0NDQ1ODMzOTA4 | 120 | Fix query command's support for DML | 79913 | closed | 0 | 1 | 2020-07-08T01:36:34Z | 2020-07-08T05:14:04Z | 2020-07-08T05:14:04Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/120 | See commit messages for details. I ran into this while investigating another feature/issue. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/120/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
697030843 | MDExOlB1bGxSZXF1ZXN0NDgzMDI3NTg3 | 156 | Typos in tests | 96218 | closed | 0 | 1 | 2020-09-09T18:00:58Z | 2020-09-09T18:24:50Z | 2020-09-09T18:21:23Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/156 | One of these is my fault, and the other is one I just happened to come across. They're harmless, but might as well be fixed. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/156/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
697203800 | MDExOlB1bGxSZXF1ZXN0NDgzMTc1NTA5 | 158 | Fix accidental mega long line in docs | 167319 | closed | 0 | 1 | 2020-09-09T22:31:23Z | 2020-09-16T06:21:43Z | 2020-09-16T06:21:43Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/158 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/158/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
709043182 | MDExOlB1bGxSZXF1ZXN0NDkzMTYyNzY3 | 178 | Update README.md | 19921 | closed | 0 | 1 | 2020-09-25T15:52:11Z | 2020-10-01T14:18:30Z | 2020-09-30T20:29:28Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/178 | The `sqlite-utils insert releases.db releases - --pk` is missing the pk field name, added ` "id"` to fix it. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/178/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
752888228 | MDExOlB1bGxSZXF1ZXN0NTI5MDkwNTYw | 204 | use jsonify_if_need for sql updates | 78035 | closed | 0 | 1 | 2020-11-29T10:49:00Z | 2020-12-08T17:49:42Z | 2020-12-08T17:49:42Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/204 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/204/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
766156875 | MDU6SXNzdWU3NjYxNTY4NzU= | 209 | Test failure with sqlite 3.34 in test_cli.py::test_optimize | 191622 | closed | 0 | 1 | 2020-12-14T08:58:18Z | 2021-01-01T23:52:46Z | 2021-01-01T23:52:46Z | CONTRIBUTOR | pytest output: ``` ... ============================== short test summary info =============================== FAILED tests/test_cli.py::test_optimize[tables0] - assert 1662976 < 1662976 FAILED tests/test_cli.py::test_optimize[tables1] - assert 1667072 < 1662976 ===================== 2 failed, 538 passed, 3 skipped in 34.32s ====================== ``` Came across this while packaging `sqlite-utils` for NixOS, but it can be recreated it using the `alpine:edge` docker image as well as follows: ``` docker run --rm -it alpine:edge /bin/sh # apk update && apk add git sqlite python3 gcc python3-dev musl-dev && python3 -m ensurepip # git clone https://github.com/simonw/sqlite-utils.git # cd sqlite-utils/ # pip3 install -e .[test] # pytest ``` This definitely works on sqlite v3.33. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/209/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
820468864 | MDExOlB1bGxSZXF1ZXN0NTgzNDA3OTg5 | 244 | Typo in upsert example | 387669 | closed | 0 | 1 | 2021-03-02T23:14:14Z | 2021-05-19T02:58:21Z | 2021-05-19T02:58:21Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/244 | Remove extra `[` | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/244/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
830803173 | MDExOlB1bGxSZXF1ZXN0NTkyMjg5MzI0 | 245 | Correct some typos | 1076745 | closed | 0 | 1 | 2021-03-13T04:26:56Z | 2021-05-19T02:58:04Z | 2021-05-19T02:58:04Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/245 | Noticed a typo in the docs and followed that up with a spellcheck. Had to bite my tongue at some of the British spellings. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/245/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
857280617 | MDExOlB1bGxSZXF1ZXN0NjE0NzI3MDM2 | 254 | Fix incorrect create-table cli description | 1935268 | closed | 0 | 1 | 2021-04-13T20:03:15Z | 2021-05-19T04:43:46Z | 2021-05-19T02:57:26Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/254 | The description for `create-table` was duplicated from `create-index`. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/254/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
868191959 | MDExOlB1bGxSZXF1ZXN0NjIzNzU1NzIz | 258 | Fixing insert from JSON containing strings with non-ascii characters … | 6586811 | closed | 0 | 1 | 2021-04-26T20:50:00Z | 2021-05-19T02:47:44Z | 2021-05-19T02:47:44Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/258 | …are escaped aps unicode for lists, tuples, dicts Fix of #257 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/258/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
979612115 | MDExOlB1bGxSZXF1ZXN0NzE5OTk4MjI1 | 322 | Add dict type to be mapped as TEXT in sqllite | 2496189 | closed | 0 | 1 | 2021-08-25T20:54:26Z | 2021-11-15T00:27:40Z | 2021-11-15T00:27:40Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/322 | the library deal with Postgres type jsonb as dictionary, add dict type as a TEXT for mapping to sqlite | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/322/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
988013247 | MDExOlB1bGxSZXF1ZXN0NzI3MDEyOTk2 | 324 | Use python-dateutil package instead of dateutils | 191622 | closed | 0 | 1 | 2021-09-03T18:31:19Z | 2021-11-14T23:25:40Z | 2021-11-14T23:25:40Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/324 | While working on updating `sqlite-utils` for NixOS/Nixpkgs, I came a cross the following: In 5ec6686153e29ae10d4921a1ad4c841f192f20e2, a new dependency was added on `dateutils` (https://pypi.org/project/dateutils/). I believe this is unintentional, and instead `python-dateutil` (https://pypi.org/project/python-dateutil/) was intended. My reasoning is: - `python-dateutil` is imported here in [recipes.py](https://github.com/simonw/sqlite-utils/blob/5ec6686153e29ae10d4921a1ad4c841f192f20e2/sqlite_utils/recipes.py#L1) - The `mypy` `type-python-dateutil` dependency in [setup.py](https://github.com/simonw/sqlite-utils/blob/5ec6686153e29ae10d4921a1ad4c841f192f20e2/setup.py#L36) - `python-dateutil` is a dependency of `dateutils` as seen in the output in [docs/tutorial.ipynb](https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/docs/tutorial.ipynb#L43) Seems like the trailing "s" seems to be the source of confusion 😅 I've swapped the dependencies out, hope this helps. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/324/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1044267332 | I_kwDOCGYnMM4-PkFE | 336 | sqlite-util tranform --column-order mangles columns of type "timestamp" | 536941 | closed | 0 | 1 | 2021-11-04T01:15:38Z | 2023-05-08T21:13:38Z | 2023-05-08T21:13:38Z | CONTRIBUTOR | Reproducible code below: ```bash > echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db > sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP); sqlite> .exit > sqlite-utils transform foo.db bar --column-order baz sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT 'CURRENT_TIMESTAMP' ); sqlite> .exit > sqlite-utils transform foo.db bar --column-order baz > sqlite3 foo.db SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT '''CURRENT_TIMESTAMP''' ); ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/336/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1149729902 | PR_kwDOCGYnMM4zbaJy | 410 | Correct spelling mistakes (found with codespell) | 3818 | closed | 0 | 1 | 2022-02-24T20:44:18Z | 2022-03-06T08:48:29Z | 2022-03-01T21:05:29Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/410 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/410/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
1178484369 | PR_kwDOCGYnMM405rPe | 419 | Ignore common generated files | 25778 | closed | 0 | 1 | 2022-03-23T18:06:22Z | 2022-03-24T21:01:44Z | 2022-03-24T21:01:44Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/419 | Closes #418 This adds four files to `.gitignore`: .hypothesis/ Pipfile Pipfile.lock pyproject.toml Those are all generated in the course of development and testing. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/419/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1244294227 | PR_kwDOCGYnMM44P4GG | 437 | docs to dogs | 114388 | closed | 0 | 1 | 2022-05-22T15:50:33Z | 2022-05-30T21:32:41Z | 2022-05-30T21:32:41Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/437 | Fixes a typo. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/437/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1299760627 | PR_kwDOCGYnMM47JUun | 452 | Add duplicate table feature | 1690072 | closed | 0 | 1 | 2022-07-09T20:24:31Z | 2022-07-15T21:21:37Z | 2022-07-15T21:21:36Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/452 | This PR addresses a feature request raised in issue #449. Specifically this PR adds a functionality that lets users duplicate a table via: ```python table_new = db["my_table"].duplicate("new_table") ``` Test added in file `tests/test_duplicate.py`. Happy to make changes to meet maintainers' feedback, if any. | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/452/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1334416486 | PR_kwDOCGYnMM488n6D | 463 | Use Read the Docs action v1 | 244656 | closed | 0 | 1 | 2022-08-10T10:31:47Z | 2022-08-18T08:30:14Z | 2022-08-17T23:11:16Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/463 | Read the Docs repository was renamed from `readthedocs/readthedocs-preview` to `readthedocs/actions/`. Now, the `preview` action is under `readthedocs/actions/preview` and is tagged as `v1` <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--463.org.readthedocs.build/en/463/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/463/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1361355564 | I_kwDOCGYnMM5RJKMs | 482 | balanced table default column_order | 7908073 | closed | 0 | 1 | 2022-09-05T03:00:18Z | 2022-10-10T17:43:02Z | 2022-09-06T20:17:27Z | CONTRIBUTOR | Is there any performance or size difference with column order in SQLITE ? similar to this https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/ It might be interesting to have an option to create with an optimized column order. I'm assuming this would look something like INTEGER columns, REAL columns, BLOB columns, TEXT columns, NULL columns. NULL columns at the end because they are more likely to be TEXT and it is impossible to know if they will become INTEGER (Of course, any schema evolution would reduce optimization but maybe column order could also be re-evaluated when schema changes) edit: this is easy to accomplish with the existing `transform` method: ``` int_columns = [k for k, v in table_columns.items() if v == int] db[table].transform(column_order=[*int_columns]) ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/482/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1366423176 | I_kwDOCGYnMM5RcfaI | 485 | Progressbar not shown when inserting/upserting jsonlines file | 99098079 | closed | 0 | 1 | 2022-09-08T14:13:18Z | 2022-09-15T20:39:52Z | 2022-09-15T20:37:52Z | CONTRIBUTOR | When inserting or upserting a jsonlines file, no progressbar is shown. Expected behavior is that, just like with .csv/.tsv files, also for a jsonlines file (--nl), unless --silent is provided, a progressbar is shown. ```bash sql-utils upsert mydb.db posts posts.jl --nl --pk post_id (silence) ``` Currently `file_progress` is only called within the tsv/csv logic, however I think it can be safely wrapped around all the all the input formats that use `decoded`: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py#L963 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/485/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1393212964 | I_kwDOCGYnMM5TCr4k | 497 | column_names | 7908073 | closed | 0 | 1 | 2022-10-01T03:34:21Z | 2022-10-25T21:09:28Z | 2022-10-25T21:09:28Z | CONTRIBUTOR | It would be nice to have a `column_names`. Similar to `table_names`. Or if you could get one or all of the following syntax to work for both Database and Table that might be even better: Style 1 - `if 'table1' in db` - `if 'col1' in db['table1']` Style 2 - `if 'table1' in db.tables` - `if 'col1' in db['table1'].columns` maybe the table ones actually work but I'm too lazy to check. I just know that I have to do: `[c.name for c in db['table1'].columns]` Edit: This is possible with `columns_dict`. I have actually used that before but I forgot about it. Feel free to close, but I do think accessing this data could be more consistent and intuitive. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/497/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1430325103 | I_kwDOCGYnMM5VQQdv | 507 | conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character | 7908073 | closed | 0 | 1 | 2022-10-31T18:49:51Z | 2022-11-01T00:40:17Z | 2022-11-01T00:40:16Z | CONTRIBUTOR | I'm not really sure what caused this and it happened in the middle of my program (after running for 35775 seconds). ``` Extracting metadata 49.9% (chunk 9893 of 19831) ... File "/home/xk/.local/lib/python3.10/site-packages/xklb/fs_extract.py", line 90, in extract_chunk args.db["media"].insert_all(utils.list_dict_filter_bool(media), pk="path", alter=True, replace=True) File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 3107, in insert_all self.insert_chunk( File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 2872, in insert_chunk result = self.db.execute(query, params) File "/home/xk/.local/lib/python3.10/site-packages/sqlite_utils/db.py", line 483, in execute return self.conn.execute(sql, parameters) UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 62: surrogates not allowed ``` This might be relevant: https://stackoverflow.com/questions/31898353/python-cant-encode-with-surrogateescape I'm going to try re-running with ```py def execute( self, sql: str, parameters: Optional[Union[Iterable, dict]] = None ) -> sqlite3.Cursor: """ Execute SQL query and return a ``sqlite3.Cursor``. :param sql: SQL query to execute :param parameters: Parameters to use in that query - an iterable for ``where id = ?`` parameters, or a dictionary for ``where id = :id`` """ try: if self._tracer: self._tracer(sql, parameters) if parameters is not None: return self.conn.execute(sql, parameters) else: return self.conn.execute(sql) except UnicodeEncodeError: sql = sql.encode('utf-8', 'surrogatepass').decode('utf-8') if parameters is not None: parameters = parameters.encode('utf-8', 'surrogatepass').decode('utf-8') return self.execute(sql, parameters) ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/507/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1505568103 | PR_kwDOCGYnMM5F609a | 519 | Fixes breaking DEFAULT values | 13819005 | closed | 0 | 1 | 2022-12-21T01:27:52Z | 2023-05-08T21:13:37Z | 2023-05-08T21:13:37Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/519 | Fixes #509, Fixes #336 Thanks for the great library! I fixed a bug that `sqlite-utils transform` breaks DEFAULT values. All tests already present passed with no changes, and I added some tests for this PR. In #509 case, fixed here. ```shell $ sqlite3 test.db << EOF CREATE TABLE mytable ( col1 TEXT DEFAULT 'foo', col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) ) EOF $ sqlite3 test.db "SELECT sql FROM sqlite_master WHERE name = 'mytable';" CREATE TABLE mytable ( col1 TEXT DEFAULT 'foo', col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) ) $ sqlite3 test.db "INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;" foo|2022-12-21 01:15:39.669 $ sqlite-utils transform test.db mytable --rename col1 renamedcol1 $ sqlite3 test.db "SELECT sql FROM sqlite_master WHERE name = 'mytable';" CREATE TABLE "mytable" ( [renamedcol1] TEXT DEFAULT 'foo', [col2] TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) # ← Non-String Value ) $ sqlite3 test.db "INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;" foo|2022-12-21 01:15:39.669 foo|2022-12-21 01:15:56.432 ``` And #336 case also fixed. Special values are described [here](https://www.sqlite.org/lang_createtable.html). > 3.2. The DEFAULT clause > ... A default value may also be one of the special case-independent keywords CURRENT_TIME, CURRENT_DATE or CURRENT_TIMESTAMP. ```shell $ echo 'create table bar (baz text, created_at timestamp default CURRENT_TIMESTAMP)' | sqlite3 foo.db $ sqlite3 foo.db SQLite version 3.39.5 2022-10-14 20:58:05 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE bar (baz text, created_at timestamp default CURRENT_TIMESTAMP); sqlite> .exit $ sqlite-utils transform foo.db bar --column-order baz $ sqlite3 foo.db SQLite version 3.39.5 2022-10-14 20:58:05 Enter ".help" for usage hints. sqlite> .schema bar CREATE TABLE IF NOT EXISTS "bar" ( [baz] TEXT, [created_at] FLOAT DEFAULT CURRENT_TIMESTAMP ); sqlite> .exit $ sqlite… | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/519/reactions", "total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1560651350 | I_kwDOCGYnMM5dBaZW | 523 | Feature request: trim all leading and trailing white space for all columns for all tables in a database | 536941 | open | 0 | 1 | 2023-01-28T02:40:10Z | 2023-01-28T02:41:14Z | CONTRIBUTOR | It's pretty common that i need to trim leading or trailing white space from lots of columns in a database a part of an initial ETL. I use the following recipe a lot, and it would be great to include this functionality into sqlite-utils `trimify.sql` ```sql select 'select group_concat(''update [' || name || '] set ['' || name || ''] = trim(['' || name || ''])'', ''; '') || ''; '' as sql_to_run from pragma_table_info('''||name||''');' from sqlite_schema; ``` then something like: ```bash sqlite3 example.db < scripts/trimify.sql > table_trim.sql && \ sqlite3 $example.db < table_trim.sql > trim.sql && \ sqlite3 $example.db < trim.sql ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/523/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1578793661 | PR_kwDOCGYnMM5Jqn1u | 528 | Enable `Table.convert()` on falsey values | 167893 | closed | 0 | 1 | 2023-02-10T00:04:09Z | 2023-05-08T21:08:23Z | 2023-05-08T21:08:23Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/528 | Fixes #527 <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--528.org.readthedocs.build/en/528/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/528/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1581090327 | I_kwDOCGYnMM5ePYYX | 529 | Microsoft line endings | 7908073 | closed | 0 | 1 | 2023-02-12T02:20:48Z | 2023-06-14T23:12:12Z | 2023-06-14T23:11:47Z | CONTRIBUTOR | sqlite-utils prints `\r\n` but [it should probably](https://devblogs.microsoft.com/commandline/extended-eol-in-notepad/) print `\n` (unless the platform is detected as Windows?) It has tripped me up a few times when piping the output of sqlite-utils to other programs: ``` $ sqlite-utils --no-headers --csv ~/lb/fs/d.db 'select path from media limit 1' | cat -A /mnt/d7/file^M$ $ sqlite-utils --no-headers --csv ~/lb/fs/d.db 'select path from media limit 1' | tr -d '\r' | cat -A /mnt/d7/file$ ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/529/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1659525418 | PR_kwDOCGYnMM5N35VZ | 536 | Add paths for homebrew on Apple silicon | 25778 | closed | 0 | 1 | 2023-04-08T13:34:21Z | 2023-04-13T01:44:43Z | 2023-04-13T01:44:43Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/536 | Does what it says and nothing else. This is the same set of paths as Datasette uses. <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--536.org.readthedocs.build/en/536/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/536/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1740026046 | I_kwDOCGYnMM5ntrC- | 556 | Support storing incrementally piped values | 601708 | open | 0 | 1 | 2023-06-04T00:45:23Z | 2023-06-04T01:21:15Z | CONTRIBUTOR | I'm trying to use sqlite-utils to data generated incrementally. There are a few aspects of this that I don't currently know how to handle. I would like an option to apply writes incrementally, line-by-line as they are received. I would like an option to echo incremental progress. And, it would be nice to have In particular, I'm using CoreLocationCLI -w -j to generate, newline-delimited JSON. One variant of the command `stdbuf -oL CoreLocationCLI -w -j | pee 'sqlite-utils insert loc.db loc -' nl` `pee`, from `moreutils`, is like `tee` but spawns and pipes to the processes created by invoking each of its arguments, so, for gratuitous demonstration, `pee 'sponge out.log' cat` would behave like `tee`. It looks like I can get what I want with: `stdbuf -oL CoreLocationCLI -w -j | while read line; do <<<"$line" sqlite-utils insert loc.db loc -; echo "$line"; done | nl` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/556/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1919296686 | PR_kwDOCGYnMM5bifPC | 596 | Fixes mapping for time fields related to mysql, closes #522 | 4420927 | closed | 0 | 1 | 2023-09-29T13:41:48Z | 2023-11-04T00:49:50Z | 2023-11-04T00:49:50Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/596 | Adds `COLUMN_TYPE_MAPPING` for `TIME` fields that are mapped as `datetime.timedelta` for MySQL and json represantation for `datetime.timedelta` in order to fix #522 <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--596.org.readthedocs.build/en/596/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/596/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1976986318 | I_kwDOCGYnMM511mrO | 599 | Cannot find spatialite on arm64 linux | 37802088 | closed | 0 | 1 | 2023-11-03T22:05:51Z | 2023-11-04T01:06:31Z | 2023-11-04T00:33:28Z | CONTRIBUTOR | Initially, I found an issue in `datasette` where it wouldn’t find `spatialite` when running on my Radxa Rock 5B - an RK3588 powered SBC, running the arm64 build of Debian Bullseye. I confirmed the same behaviour on my Raspberry Pi 4 - a BCM2711 powered SBC, running the arm64 build of Debian Bookworm. ``` $ datasette --load-extension=spatialite example.db Error: Could not find SpatiaLite extension ``` I did some digging and realised the issue originates in this project. Even with the `libsqlite3-mod-spatialite` package installed, `pytest` skips all of the GIS tests in the project. ``` $ apt list --installed | grep spatial […] libsqlite3-mod-spatialite/stable,now 5.0.1-3 arm64 [installed] $ ls -l /usr/lib/*/*spatial* lrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0 lrwxrwxrwx 1 root root 23 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0 -rw-r--r-- 1 root root 7348584 Dec 1 2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0 ``` ``` $ pytest tests/test_get.py ...... [ 73%] tests/test_gis.py ssssssssssss [ 75%] tests/test_hypothesis.py .... [ 75%] ``` I tracked the issue down to the [`find_sqlite()` function in the `utils.py`](https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/utils.py#L60) file. The [`SPATIALITE_PATHS`](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L34-L39) array doesn’t have an entry for the location of this module on arm64 linux. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/599/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
411066700 | MDU6SXNzdWU0MTEwNjY3MDA= | 10 | Error in upsert if column named 'order' | 82988 | closed | 0 | 1 | 2019-02-16T12:05:18Z | 2019-02-24T16:55:38Z | 2019-02-24T16:55:37Z | NONE | The following works fine: ``` connX = sqlite3.connect('DELME.db', timeout=10) dfX=pd.DataFrame({'col1':range(3),'col2':range(3)}) DBX = Database(connX) DBX['test'].upsert_all(dfX.to_dict(orient='records')) ``` But if a column is named `order`: ``` connX = sqlite3.connect('DELME.db', timeout=10) dfX=pd.DataFrame({'order':range(3),'col2':range(3)}) DBX = Database(connX) DBX['test'].upsert_all(dfX.to_dict(orient='records')) ``` it throws an error: ``` --------------------------------------------------------------------------- OperationalError Traceback (most recent call last) <ipython-input-130-7dba33cd806c> in <module> 3 dfX=pd.DataFrame({'order':range(3),'col2':range(3)}) 4 DBX = Database(connX) ----> 5 DBX['test'].upsert_all(dfX.to_dict(orient='records')) /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order) 347 foreign_keys=foreign_keys, 348 upsert=True, --> 349 column_order=column_order, 350 ) 351 /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, upsert, batch_size, column_order) 327 jsonify_if_needed(record.get(key, None)) for key in all_columns 328 ) --> 329 result = self.db.conn.execute(sql, values) 330 self.db.conn.commit() 331 self.last_id = result.lastrowid OperationalError: near "order": syntax error ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
432727685 | MDU6SXNzdWU0MzI3Mjc2ODU= | 20 | JSON column values get extraneously quoted | 649467 | closed | 0 | 4348046 | 1 | 2019-04-12T20:15:30Z | 2019-05-25T00:57:19Z | 2019-05-25T00:57:19Z | NONE | If the input to `sqlite-utils insert` includes a column that is a JSON array or object, `sqlite-utils query` will introduce an extra level of quoting on output: ``` # echo '[{"key": ["one", "two", "three"]}]' | sqlite-utils insert t.db t - # sqlite-utils t.db 'select * from t' [{"key": "[\"one\", \"two\", \"three\"]"}] # sqlite3 t.db 'select * from t' ["one", "two", "three"] ``` This might require an imperfect solution, since sqlite3 doesn't have a JSON type. Perhaps fields that start with `["` or `{"` and end with `"]` or `"}` could be detected, with a flag to turn off that behavior for weird text fields (or vice versa). | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/20/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
577302229 | MDU6SXNzdWU1NzczMDIyMjk= | 91 | Enable ordering FTS results by rank | 416374 | closed | 0 | 6079500 | 1 | 2020-03-07T08:43:51Z | 2020-11-06T23:53:26Z | 2020-11-06T23:53:25Z | NONE | According to https://www.sqlite.org/fts5.html (not sure about FTS4) results can be sorted by relevance. At the moment results are returned by default by `rowid`. Perhaps a flag can be added to the `search` method? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/91/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
723708310 | MDU6SXNzdWU3MjM3MDgzMTA= | 188 | About loading spatialite | 30607 | closed | 0 | 1 | 2020-10-17T08:47:02Z | 2022-02-05T00:04:26Z | 2020-10-17T08:52:58Z | NONE | Hi @simonw , If I run ``` sqlite3 .load /usr/local/lib/mod_spatialite.so select spatialite_version(); ``` I have `5.0.0`. ![image](https://user-images.githubusercontent.com/30607/96332706-d8cd3300-1065-11eb-906b-daf99963198e.png) If I run ``` sqlite-utils :memory: "select spatialite_version()" --load-extension=spatialite ``` I have ``` Traceback (most recent call last): File "/home/aborruso/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 936, in query _load_extensions(db, load_extension) File "/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 1326, in _load_extensions db.conn.load_extension(ext) TypeError: argument 1 must be str, not None ``` How to load properly spatialite extension in sqlite-utils? Thank you very muc | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/188/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
767685961 | MDU6SXNzdWU3Njc2ODU5NjE= | 210 | Support of RData files | 23739126 | closed | 0 | 1 | 2020-12-15T15:04:14Z | 2021-01-02T00:02:40Z | 2021-01-02T00:02:40Z | NONE | Hi Simon, Would be great if you could ingest RData files! I could do this in a few lines of code but I am too lazy - sorry! Peter | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/210/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
836829560 | MDU6SXNzdWU4MzY4Mjk1NjA= | 248 | support for Apache Arrow / parquet files I/O | 649467 | open | 0 | 1 | 2021-03-20T14:59:30Z | 2021-10-28T23:46:48Z | NONE | I just started looking at Apache Arrow using pyarrow for import and export of tabular datasets, and it looks quite compelling. It might be worth looking at for sqlite-utils and/or datasette. As a test, I took a random jsonl data dump of a dataset I have with floats, strings, and ints and converted it to arrow's parquet format using the naive `pyarrow.parquet.write_file()` command, which has automatic type inferrence. It compressed down to 7% of the original size. Conversion of a 26MB JSON file and serializing it to parquet was eyeblink instantaneous. Parquet files are portable and can be directly imported into pandas and other analytics software. The only hangup is the automatic type inference of the naive reader. It's great for general laziness and for parsing JSON columns (it correctly interpreted a table of mine with a JSON array). However, I did get an exception for a string column where most entries looked integer-like but had a couple values that weren't -- the reader tried to coerce all of them for some reason, even though the JSON type is string. Since the writer optionally takes a schema, it shouldn't be too hard to grab the sqlite header types. With some additional hinting, you might get datetime columns and JSON, which are native Arrow types. Somewhat tangentially, someone even wrote an sqlite vfs extension for Parquet: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/248/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
842062949 | MDU6SXNzdWU4NDIwNjI5NDk= | 252 | Support json-line files | 279769 | closed | 0 | 1 | 2021-03-26T15:19:39Z | 2021-03-26T15:21:38Z | 2021-03-26T15:21:38Z | NONE | It's common for many processes to create flat files where each line is a JSON object. So the file isn't a json array. Many tools (like jq) support this natively, it'd be great for sqlite-utils to also! | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/252/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
907795562 | MDU6SXNzdWU5MDc3OTU1NjI= | 265 | Using enable_fts before search term | 36287 | open | 0 | 1 | 2021-06-01T01:43:34Z | 2023-04-01T17:27:18Z | NONE | Many thanks for the sqlite-utils suite of utilities. Has made my life much much easier. I used this to create a table and enable FTS. All works fine. The datasette utility detects FTS and shows a text box. Searching for a term using that interface works well. However, when I start to use features by following https://www.sqlite.org/fts5.html section **"3. Full-text Query Syntax"** I seem to run into issues that I suspect is due to `escape_fts` wrapper function. As an example, if i search for the term `"^குகை" `on the text box in datasette it produces 140 results. However, when i tweak the query produced by datasette to not use "escape_fts" it produces 5 results. Similarly, when I try to restrict the search to a single column in FTS using a spec like `{title : ^குகை}` it returns no rows. The same thing pulls results when used without `escape_fts`. The text in the table is in Tamil language and the search term is a Tamil word. ``` ... where posts_fts match escape_fts(:search) ``` vs ``` ... where posts_fts match (:search) ``` Any ideas why? How can I get the benefits of both escaping as well as utilizing different facets of providing / controlling search terms? Thanks. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/265/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
934123448 | MDU6SXNzdWU5MzQxMjM0NDg= | 295 | Insert with --tsv and --no-headers give error about --nl arguments | 7288187 | closed | 0 | 1 | 2021-06-30T21:01:01Z | 2021-08-18T20:19:04Z | 2021-08-18T20:18:57Z | NONE | Not quite sure if this is a bug, or just an assumption I made but I thought `--tsv` and `--no-headers` would work together when inserting from a file, and currently they seem not to (sqlite-utils, version 3.12, installed on Mac OS X via brew) Instead it says: `Error: Use just one of --nl, --csv or --tsv` As if it has interpreted the --no-headers as --nl. The --help does specifically say CSV: `--no-headers CSV file has no header row` And this heading in the documentation also only refers to CSV, but the text does mention TSV in passing, and I'd generally expect them to behave the same in most cases. https://sqlite-utils.datasette.io/en/stable/cli.html#csv-files-without-a-header-row | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/295/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
956832836 | MDU6SXNzdWU5NTY4MzI4MzY= | 300 | Returning underlying cause for User Defined Functions | 71236 | closed | 0 | 1 | 2021-07-30T15:08:21Z | 2021-08-02T21:53:50Z | 2021-08-02T21:53:50Z | NONE | The sqlite3 client takes user defined functions and replaces the text with "user-defined function raised exception`" so it's not apparent what's gone wrong: ``` Unexpected error: user-defined function raised exception ``` As mentioned in https://code.djangoproject.com/ticket/29500 and https://stackoverflow.com/questions/45824209/how-to-get-an-error-kind-from-sqlite-create-function/45834923#45834923 the workaround for this is to enable callback tracebacks: ``` sqlite3.enable_callback_tracebacks(True) ``` It would be nice if https://sqlite-utils.datasette.io/en/stable/python-api.html#registering-custom-sql-functions either included a reference to `enable_callback_tracebacks` or if registering a user defined function set this flag automatically. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/300/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1028056713 | I_kwDOCGYnMM49RuaJ | 332 | `sqlite-utils memory --flatten` option to flatten nested JSON | 22523840 | closed | 0 | 1 | 2021-10-16T14:04:42Z | 2021-11-14T23:05:05Z | 2021-11-14T23:05:05Z | NONE | currently --flatten option works only for `insert` command, it would be cool if it worked for `memory` as well to query nested json | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/332/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1046271107 | PR_kwDOCGYnMM4uK5z2 | 337 | Default values for `--attach` and `--param` options | 771193 | closed | 0 | 1 | 2021-11-05T21:57:53Z | 2021-11-05T22:33:03Z | 2021-11-05T22:33:02Z | NONE | simonw/sqlite-utils/pulls/337 | It seems that `click` 8.x uses `None` as the default value for `multiple=True` options. This change makes the code forward-compatible with `click` 8.x. See this build failure for more info: https://hydra.nixos.org/build/156926608 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/337/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1079422215 | I_kwDOCGYnMM5AVq0H | 357 | pytest-runner is not required | 4067843 | closed | 0 | 1 | 2021-12-14T07:51:24Z | 2021-12-16T20:43:19Z | 2021-12-16T20:43:13Z | NONE | Deprecated pytest-runner is not necessary for running the testsuite. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/357/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1091819089 | I_kwDOCGYnMM5BE9ZR | 360 | MemoryError | 559453 | closed | 0 | 1 | 2022-01-01T13:39:17Z | 2022-03-21T04:22:46Z | 2022-03-21T04:22:46Z | NONE | HI, when dealing with large json file (~170GB) i got the following error ``` Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 1126, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 1051, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 1393, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 752, in invoke return __callback(*args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/cli.py", line 1300, in memory rows, format_used = rows_from_file(csv_fp, format=format, encoding=encoding) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py", line 185, in rows_from_file return rows_from_file(buffered, format=Format.JSON) File "/usr/local/lib/python3.9/dist-packages/sqlite_utils/utils.py", line 156, in rows_from_file decoded = json.load(fp) File "/usr/lib/python3.9/json/__init__.py", line 293, in load return loads(fp.read(), MemoryError ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/360/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1199158210 | I_kwDOCGYnMM5HebPC | 423 | .extract() doesn't set foreign key when extracted columns contain NULL value | 37447552 | closed | 0 | 1 | 2022-04-10T20:05:30Z | 2022-08-27T14:45:04Z | 2022-08-27T14:45:04Z | NONE | I've run into an issue with `extract` and I don't believe this is the intended behaviour. I'm working with a database with music listening information. Currently it has one large table `listens` that contains all information. I'm trying to normalize the database by extracting relevant columns to separate tables (`artists`, `tracks`, `albums`). Not every track has an album. A simplified demonstration with just `track_title` and `album_title` columns: ```ipython In [1]: import sqlite_utils In [2]: db = sqlite_utils.Database(memory=True) In [3]: db["listens"].insert_all([ ...: {"id": 1, "track_title": "foo", "album_title": "bar"}, ...: {"id": 2, "track_title": "baz", "album_title": None} ...: ], pk="id") Out[3]: <Table listens (id, track_title, album_title)> ``` The track in the first row has an album, the second track doesn't. Now I extract album information into a separate column: ```ipython In [4]: db["listens"].extract(columns=["album_title"], table="albums", fk_column="album_id") Out[4]: <Table listens (id, track_title, album_id)> In [5]: list(db["albums"].rows) Out[5]: [{'id': 1, 'album_title': 'bar'}, {'id': 2, 'album_title': None}] In [6]: list(db["listens"].rows) Out[6]: [{'id': 1, 'track_title': 'foo', 'album_id': 1}, {'id': 2, 'track_title': 'baz', 'album_id': None}] ``` This behaves as expected -- the `album` table contains entries for both the existing album and the NULL album. The `listens` table has a foreign key only for the first row (since the album in the second row was empty). Now I want to extract the track information as well. Album information belongs to the track so I want to extract both columns to a new table. ```ipython In [7]: db["listens"].extract(columns=["track_title", "album_id"], table="tracks", fk_column="track_id") Out[7]: <Table listens (id, track_id)> In [8]: list(db["tracks"].rows) Out[8]: [{'id': 1, 'track_title': 'foo', 'album_id': 1}, {'id': 2, 'track_title': 'baz', 'album_id': None}] In [9]: list(db["… | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/423/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1212701569 | I_kwDOCGYnMM5ISFuB | 427 | sqlite-utils convert date parsing recipe complains about trying to parse "*" | 1385831 | closed | 0 | 1 | 2022-04-22T19:27:10Z | 2022-07-02T13:59:59Z | 2022-07-02T13:59:32Z | NONE | Missing values in my dataset are denoted by a single asterisk. I am trying to parse string dates into dates. This works fine for columns without missing values, but, when the column contains "*", I get the following: ``` $ sqlite-utils convert ${dbfile} details dob 'r.parsedate(value)' [------------------------------------] 0%Traceback (most recent call last): File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2508, in convert_value return fn(v) File "<string>", line 2, in fn File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 8, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: * Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils==3.25.1', 'console_scripts', 'sqlite-utils')()) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, … | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/427/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1303169663 | I_kwDOCGYnMM5NrMp_ | 453 | 'unclosed file' warning when using insert_upsert_implementation from Python | 311257 | closed | 0 | 1 | 2022-07-13T09:34:35Z | 2022-07-15T21:52:25Z | 2022-07-15T21:52:21Z | NONE | I'm using the `[insert_upsert_implementation](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py)` function directly in my Python code to import a csv file with all the bells and whistles `sqlite-utils` provides, but I'm getting a resource warning that a io.TextWrapper object is not closed. The warning goes away when wrapping the code from [this line](https://github.com/simonw/sqlite-utils/blob/42440d6345c242ee39778045e29143fb550bd2c2/sqlite_utils/cli.py#L924) in a try/finally block like: ``` try: ... ... finally: decoded.close() ``` (might be that `sniff_buffer` must also be closed if non null, but I might be wrong) I suspect Python closes the reference automatically when the sqlite-utils cli run is done, but since my code doesn't exit, I'm getting the warning. Alternatively, it'd be cool if the 'import csv/tsv' functionality could be added directly to the Database class. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/453/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1465194930 | PR_kwDOCGYnMM5DvZxa | 515 | upsert new rows with constraints, fixes #514 | 193185 | closed | 0 | 1 | 2022-11-26T16:15:21Z | 2023-05-08T21:27:11Z | 2023-05-08T21:27:10Z | NONE | simonw/sqlite-utils/pulls/515 | This fixes #514 by making the initial insert for upserts include all columns, so that new rows can be added to tables with non-pkey columns that have constraints. (aside: I'm not a python programmer. `pip`? `pipenv`? `venv`? These are mystical incantations to me. The process to set up this repo for local development and testing was _so easy_. Thank you for the excellent contributing documentation!) <!-- readthedocs-preview sqlite-utils start --> ---- :books: Documentation preview :books:: https://sqlite-utils--515.org.readthedocs.build/en/515/ <!-- readthedocs-preview sqlite-utils end --> | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/515/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1622640374 | I_kwDOCGYnMM5gt4b2 | 534 | ResourceWarning: unclosed file | 1244826 | closed | 0 | 1 | 2023-03-14T03:02:18Z | 2023-05-08T19:56:29Z | 2023-05-08T19:56:29Z | NONE | Issuing either ``` py -Wdefault -m sqlite_utils insert dogs.db dogs dogs0.csv --csv [#############-----------------------] 36% [####################################] 100%C:\Users\Doug\AppData\Local\Programs\Python\Python311\Lib\site-packages\sqlite_utils\cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs0.csv' encoding='utf-8-sig'> insert_upsert_implementation( ResourceWarning: Enable tracemalloc to get the object allocation traceback ``` or ``` set pythonwarnings=default sqlite-utils insert dogs.db dogs dogs0.csv --csv [#############-----------------------] 36% [####################################] 100%C:\Users\Doug\AppData\Local\Programs\Python\Python311\Lib\site-packages\sqlite_utils\cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs0.csv' encoding='utf-8-sig'> insert_upsert_implementation( ResourceWarning: Enable tracemalloc to get the object allocation traceback ``` exhibits a ResourceWarning indicating that the CSV file being loaded is not closed. sqlite-utils --version sqlite-utils, version 3.30 py --version Python 3.11.2 Windows Version 10.0.19045 Build 19045 SQLite version 3.41.0 2023-02-21 18:09:37 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/534/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1720096994 | I_kwDOCGYnMM5mhpji | 554 | `IndexError` when doing `.insert(..., pk='id')` after `insert_all` | 1231935 | open | 0 | 1 | 2023-05-22T17:13:02Z | 2023-05-22T17:18:33Z | NONE | I believe this is related to https://github.com/simonw/sqlite-utils/issues/98. When `pk` is specified by table A's `insert` call, it throws an index error if a different table has written a row with a higher rowid than exists in the first table. Here's a basic example: ```py from sqlite_utils import Database def test_pk_for_insert(fresh_db): user = {"id": "abc", "name": "david"} fresh_db["users"].insert(user, pk="id") fresh_db["comments"].insert_all( [ {"id": "def", "text": "ok"}, {"id": "ghi", "text": "great"}, ], ) fresh_db["users"].insert( user, ignore=True, # BUG: when specifying pk on the second insert call # db.py goes into a block it doesn't expect and we get the error pk="id", ) if __name__ == "__main__": db = Database("bug.db") if db["users"].exists(): raise ValueError( "bug only shows on a new database - remove bug.db before running the script" ) test_pk_for_insert(db) ``` The error is: ```py File "/Users/david/projects/reddit-to-sqlite/.venv/lib/python3.11/site-packages/sqlite_utils/db.py", line 2960, in insert_chunk row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^ IndexError: list index out of range ``` The issue is in this block: https://github.com/simonw/sqlite-utils/blob/2747257a3334d55e890b40ec58fada57ae8cfbfd/sqlite_utils/db.py#L2954-L2958 relevant locals are: - `pk`: `'id'` - `result.lastrowid`: `2` What's most interesting is the comment `# self.last_rowid will be 0 if a "INSERT OR IGNORE" happened`, which doesn't seem to be the case here. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/554/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1733198948 | I_kwDOCGYnMM5nToRk | 555 | Filter table by a large bunch of ids | 10843208 | open | 0 | 1 | 2023-05-31T00:29:51Z | 2023-06-14T22:01:57Z | NONE | Hi! this might be a question related to both SQLite & sqlite-utils, and you might be more experienced with them. I have a large bunch of ids, and I'm wondering which is the best way to query them in terms of performance, and simplicity if possible. The naive approach would be something like `select * from table where rowid in (?, ?, ?...)` but that wouldn't scale if ids are >1k. Another approach might be creating a temp table, or in-memory db table, insert all ids in that table and then join with the target one. I failed to attach an in-memory db both using sqlite-utils, and plain sql's execute(), so my closest approach is something like, ```python def filter_existing_video_ids(video_ids): db = get_db() # contains a "videos" table db.execute("CREATE TEMPORARY TABLE IF NOT EXISTS tmp (video_id TEXT NOT NULL PRIMARY KEY)") db["tmp"].insert_all([{"video_id": video_id} for video_id in video_ids]) for row in db["tmp"].rows_where("video_id not in (select video_id from videos)"): yield row["video_id"] db["tmp"].drop() ``` That kinda worked, I couldn't find an option in sqlite-utils's `create_table()` to tell it's a temporary table. Also, `tmp` table is not dropped finally, neither using `.drop()` despite being created with the keyword `TEMPORARY`. I believe it should be automatically dropped after connection/session ends though I read. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/555/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1907281675 | I_kwDOCGYnMM5xrs8L | 595 | Cascading DELETE not working with Table.delete(pk) | 123451970 | closed | 0 | 1 | 2023-09-21T15:46:41Z | 2023-09-25T09:38:57Z | 2023-09-25T09:38:13Z | NONE | Hi ! I noticed that when I am trying to use the delete method of the Table object, the record get properly deleted from the table, but the cascading delete triggers on foreign keys do not activate. `self.db["contact"].delete(contact_id)` I tried querying the database directly via DB Browser and the triggers work without any issue. Looked up the source code and behind the scene this method is just querying the database normally so I'm not exactly sure where this behavior comes from. Thank you in advance for your time ! | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/595/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1988525411 | I_kwDOCGYnMM52hn1j | 603 | Pyhton 3.12 Bug report | 1324252 | open | 0 | 1 | 2023-11-10T22:57:48Z | 2023-12-08T05:10:31Z | NONE | I start with new python3 verison 3.12.0 Also have the error where connect DataBase ``` Traceback (most recent call last): File "/home/t/Development/python/FKPJ/ClinicSYS/run.py", line 1, in <module> import re, os, io, json, sqlite_utils, requests, pytz, logging File "/home/t/.local/lib/python3.12/site-packages/sqlite_utils/__init__.py", line 1, in <module> from .db import Database File "/home/t/.local/lib/python3.12/site-packages/sqlite_utils/db.py", line 277, in <module> class Database: File "/home/t/.local/lib/python3.12/site-packages/sqlite_utils/db.py", line 306, in Database filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None, ^^^^^^^^^^^^^^^^^^ ``` This bug come from `sqlite-utils` since's v3.33. Anyone get the same ? As well now of the resolved plan just keep the sqlite-utils version in python3.12 with v3.32.1 [tested] but where are the sqlite3.Connection problem.... This won't happen on python version down to 3.11[tested] Just the python3.12.0, I have test this error are come from the sqlite3 connection The error say from `sqlite_utils` and with the sqlite3 Connection, what can I do. Let fix together. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/603/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
2007893839 | I_kwDOCGYnMM53rgdP | 605 | Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here? | 12229877 | closed | 0 | 1 | 2023-11-23T10:19:46Z | 2023-12-08T05:07:54Z | 2023-12-08T05:07:54Z | NONE | I'm currently working on a new feature for Hypothesis, where we can dump a tidy jsonlines table of all the test cases we tried - including arguments, outcomes, timings, coverage, etc. Exploring this seems like a perfect cases for `sqlite-utils` and `datasette`, but I pretty quickly ran into an integer overflow problem and don't want to recommend that experience to my users. I originally went to report this as a bug... and then found https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038 almost exactly matched my repro 😅 https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063 suggests that using `NUMERIC` would avoid this overflow error, although "If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL." suggests that this would come at the cost of rounding to the nearest float value. Maybe I should just convert large integers to float before writing out my json? After a bit more hacking, "manually cast large integers to float" seems like a decent solution for my particular case, but having written it up I thought I might as well post this issue anyway - I hope it's useful feedback, and won't mind at all if you close as wontfix if it's not. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/605/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
349850687 | MDU6SXNzdWUzNDk4NTA2ODc= | 2 | Mechanism for adding foreign keys to an existing table | 9599 | closed | 0 | 1 | 2018-08-12T22:50:56Z | 2019-02-24T21:34:41Z | 2019-02-24T21:34:41Z | OWNER | SQLite does not have ALTER TABLE support for adding new foreign keys... but it turns out it's possible to make these changes without having to duplicate the entire table by carefully running `UPDATE sqlite_master SET sql=... WHERE type='table' AND name='X';` Here's how Django does it: https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/db/backends/sqlite3/schema.py#L103-L125 And here's the official documentation about this: https://sqlite.org/lang_altertable.html#otheralter (scroll to the very bottom of the page) | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
351845423 | MDU6SXNzdWUzNTE4NDU0MjM= | 3 | Experiment with contentless FTS tables | 9599 | closed | 0 | 1 | 2018-08-18T19:31:01Z | 2019-07-22T20:58:55Z | 2019-07-22T20:58:55Z | OWNER | Could greatly reduce size of resulting database for large datasets: http://cocoamine.net/blog/2015/09/07/contentless-fts4-for-large-immutable-documents/ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
403624090 | MDU6SXNzdWU0MDM2MjQwOTA= | 6 | "sqlite-utils insert" should support newline-delimited JSON | 9599 | closed | 0 | 1 | 2019-01-28T02:00:02Z | 2019-01-28T02:17:45Z | 2019-01-28T02:17:45Z | OWNER | We can already export newline delimited JSON. We should learn to import it as well. The neat thing about importing it is that you can import GBs of data without having to read the whole lot into memory in order to decode the wrapping JSON array. Datasette can export it now: https://github.com/simonw/datasette/issues/405 Demo: https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on It should be possible to do this: $ curl "https://latest.datasette.io/fixtures/facetable.json?_shape=array&_nl=on" \ | sqlite-utils insert data.db facetable - --nl | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
413779210 | MDU6SXNzdWU0MTM3NzkyMTA= | 13 | Ability to automatically create IDs from content hash of row | 9599 | closed | 0 | 1 | 2019-02-24T04:07:08Z | 2019-02-24T04:36:48Z | 2019-02-24T04:36:48Z | OWNER | Sometimes when you are importing data the underlying source provides records without IDs that can be uniquely identified by their contents. A utility mechanism for calculating a sha1 hash of the contents and using that as a unique ID would be useful. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
455996809 | MDU6SXNzdWU0NTU5OTY4MDk= | 28 | Rearrange the docs by area, not CLI vs Python | 9599 | closed | 0 | 1 | 2019-06-13T23:33:35Z | 2019-07-15T02:37:20Z | 2019-07-15T02:37:20Z | OWNER | The docs for eg inserting data should live on the same page, rather than being split across the API and CLI pages. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
461215118 | MDU6SXNzdWU0NjEyMTUxMTg= | 30 | Option to open database in read-only mode | 9599 | closed | 0 | 1 | 2019-06-26T22:50:38Z | 2020-05-11T19:17:17Z | 2020-05-11T19:17:17Z | OWNER | Would this make it 100% safe to run reads against a database file that is being written to by another process? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
462094937 | MDExOlB1bGxSZXF1ZXN0MjkyODc5MjA0 | 32 | db.add_foreign_keys() method | 9599 | closed | 0 | 1 | 2019-06-28T15:40:33Z | 2019-06-29T06:27:39Z | 2019-06-29T06:27:39Z | OWNER | simonw/sqlite-utils/pulls/32 | Refs #31. Still TODO: - [x] Unit tests - [x] Documentation | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
467928674 | MDExOlB1bGxSZXF1ZXN0Mjk3NDU5Nzk3 | 40 | .get() method plus support for compound primary keys | 9599 | closed | 0 | 1 | 2019-07-15T03:43:13Z | 2019-07-15T04:28:57Z | 2019-07-15T04:28:52Z | OWNER | simonw/sqlite-utils/pulls/40 | - [x] Tests for the `NotFoundError` exception - [x] Documentation for `.get()` method - [x] Support `--pk` multiple times to define CLI compound primary keys - [x] Documentation for compound primary keys | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/40/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
473733752 | MDExOlB1bGxSZXF1ZXN0MzAxODI0MDk3 | 51 | Fix for too many SQL variables, closes #50 | 9599 | closed | 0 | 1 | 2019-07-28T11:30:30Z | 2019-07-28T11:59:32Z | 2019-07-28T11:59:32Z | OWNER | simonw/sqlite-utils/pulls/51 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/51/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
476413293 | MDU6SXNzdWU0NzY0MTMyOTM= | 52 | Throws error if .insert_all() / .upsert_all() called with empty list | 9599 | closed | 0 | 1 | 2019-08-03T04:09:00Z | 2019-11-07T04:32:39Z | 2019-11-07T04:32:39Z | OWNER | See also https://github.com/simonw/db-to-sqlite/issues/18 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/52/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
481887482 | MDExOlB1bGxSZXF1ZXN0MzA4MjkyNDQ3 | 55 | Ability to introspect and run queries against views | 9599 | closed | 0 | 1 | 2019-08-17T13:40:56Z | 2019-08-23T12:19:42Z | 2019-08-23T12:19:42Z | OWNER | simonw/sqlite-utils/pulls/55 | See #54 | 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/55/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
517241040 | MDU6SXNzdWU1MTcyNDEwNDA= | 63 | ensure_index() method | 9599 | closed | 0 | 1 | 2019-11-04T15:51:22Z | 2019-11-04T16:20:36Z | 2019-11-04T16:20:35Z | OWNER | ```python db["table"].ensure_index(["col1", "col2"]) ``` This will do the following: - if the specified table or column does not exist, do nothing - if they exist and already have an index, do nothing - otherwise, create the index I want this for tools like [twitter-to-sqlite search](https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches) where the `search_runs` table may or not have been created yet but, if it IS created, I want to put an index on the `hash` column. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/63/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
531583658 | MDU6SXNzdWU1MzE1ODM2NTg= | 68 | Add support for porter stemming in FTS | 9599 | closed | 0 | 1 | 2019-12-02T22:35:52Z | 2020-09-20T04:25:53Z | 2020-09-20T04:25:47Z | OWNER | FTS5 can have porter stemming enabled. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/68/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
559374410 | MDU6SXNzdWU1NTkzNzQ0MTA= | 83 | Make db["table"].exists a documented API | 9599 | closed | 0 | 1 | 2020-02-03T22:31:44Z | 2020-02-08T23:58:35Z | 2020-02-08T23:56:23Z | OWNER | Right now it's a static thing which might get out-of-sync with the database. It should probably be a live check. Maybe call it `.exists()` instead? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/83/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
562911863 | MDU6SXNzdWU1NjI5MTE4NjM= | 85 | Create index doesn't work for columns containing spaces | 9599 | closed | 0 | 1 | 2020-02-11T00:34:46Z | 2020-02-11T05:13:20Z | 2020-02-11T05:13:20Z | OWNER | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/85/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
610853576 | MDU6SXNzdWU2MTA4NTM1NzY= | 105 | "sqlite-utils views" command | 9599 | closed | 0 | 1 | 2020-05-01T16:56:11Z | 2020-05-01T20:40:07Z | 2020-05-01T20:38:36Z | OWNER | Similar to `sqlite-utils tables`. See also #104. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/105/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
611216862 | MDU6SXNzdWU2MTEyMTY4NjI= | 106 | create_view(..., ignore=True, replace=True) parameters | 9599 | closed | 0 | 1 | 2020-05-02T15:45:21Z | 2020-05-02T16:04:51Z | 2020-05-02T16:02:10Z | OWNER | Two new parameters which specify what should happen if the view already exists. I want this for https://github.com/dogsheep/github-to-sqlite/issues/37 Here's the current `create_view()` implementation: https://github.com/simonw/sqlite-utils/blob/b4d953d3ccef28bb81cea40ca165a647b59971fa/sqlite_utils/db.py#L325-L332 `ignore=True` will not do anything if the view exists already. `replace=True` will drop and redefine the view - but only if its SQL definition differs, otherwise it will be left alone. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/106/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
612658444 | MDU6SXNzdWU2MTI2NTg0NDQ= | 109 | table.create_index(..., ignore=True) | 9599 | closed | 0 | 1 | 2020-05-05T14:44:21Z | 2020-05-05T14:46:53Z | 2020-05-05T14:46:53Z | OWNER | Option to silently do nothing if the index already exists. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/109/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
637889964 | MDU6SXNzdWU2Mzc4ODk5NjQ= | 115 | Ability to execute insert/update statements with the CLI | 9599 | closed | 0 | 1 | 2020-06-12T17:01:17Z | 2020-06-12T17:51:11Z | 2020-06-12T17:41:10Z | OWNER | ``` $ sqlite-utils github.db "update stars set starred_at = ''" Traceback (most recent call last): File "/Users/simon/.local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/Users/simon/.local/pipx/venvs/sqlite-utils/lib/python3.8/site-packages/sqlite_utils/cli.py", line 673, in query headers = [c[0] for c in cursor.description] TypeError: 'NoneType' object is not iterable ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/115/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
665802405 | MDU6SXNzdWU2NjU4MDI0MDU= | 124 | sqlite-utils query should support named parameters | 9599 | closed | 0 | 1 | 2020-07-26T15:25:10Z | 2020-07-30T22:57:51Z | 2020-07-27T03:53:58Z | OWNER | To help out with escaping - so you can run this: sqlite-utils query "insert into foo (blah) values (:blah)" --param blah `something here` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/124/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
666639051 | MDU6SXNzdWU2NjY2MzkwNTE= | 128 | Support UUID and memoryview types | 9599 | closed | 0 | 1 | 2020-07-27T23:08:34Z | 2020-07-30T01:10:43Z | 2020-07-30T01:10:43Z | OWNER | `psycopg2` can return data from PostgreSQL as `uuid.UUID` or `memoryview` objects. These should to be supported by `sqlite-utils` - mainly for https://github.com/simonw/db-to-sqlite | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/128/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
677839979 | MDU6SXNzdWU2Nzc4Mzk5Nzk= | 133 | Release a sdist to PyPI | 9599 | closed | 0 | 1 | 2020-08-12T16:55:09Z | 2020-08-12T17:05:06Z | 2020-08-12T17:05:06Z | OWNER | https://pypi.org/project/sqlite-utils/#files currently just has a wheel. I need this to package for homebrew: https://github.com/simonw/homebrew-datasette/issues/10 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/133/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
683830416 | MDU6SXNzdWU2ODM4MzA0MTY= | 137 | --load-extension for other sqlite-utils commands | 9599 | closed | 0 | 1 | 2020-08-21T21:12:56Z | 2020-10-16T19:14:32Z | 2020-10-16T19:14:32Z | OWNER | e.g. for this: ``` calands-datasette % sqlite-utils tables calands.db --counts [{"table": "spatial_ref_sys", "count": 4924}, {"table": "spatialite_history", "count": 14}, {"table": "sqlite_sequence", "count": 1}, {"table": "geometry_columns", "count": 2}, {"table": "spatial_ref_sys_aux", "count": 4873}, {"table": "views_geometry_columns", "count": 0}, {"table": "virts_geometry_columns", "count": 0}, {"table": "geometry_columns_statistics", "count": 2}, {"table": "views_geometry_columns_statistics", "count": 0}, {"table": "virts_geometry_columns_statistics", "count": 0}, {"table": "geometry_columns_field_infos", "count": 0}, {"table": "views_geometry_columns_field_infos", "count": 0}, {"table": "virts_geometry_columns_field_infos", "count": 0}, {"table": "geometry_columns_time", "count": 2}, {"table": "geometry_columns_auth", "count": 2}, {"table": "views_geometry_columns_auth", "count": 0}, {"table": "virts_geometry_columns_auth", "count": 0}, Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/usr/local/Cellar/sqlite-utils/2.15.1/libexec/lib/python3.8/site-packages/sqlite_utils/cli.py", line 143, in tables … | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/137/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
688351054 | MDU6SXNzdWU2ODgzNTEwNTQ= | 140 | Idea: insert-files mechanism for adding extra columns with fixed values | 9599 | open | 0 | 1 | 2020-08-28T20:57:36Z | 2022-03-20T19:45:45Z | OWNER | Say for example you want to populate a `file_type` column with the value `gif`. That could work like this: ``` sqlite-utils insert-files gifs.db images *.gif \ -c path -c md5 -c last_modified:mtime \ -c file_type:text:gif --pk=path ``` So a column defined as a `text` column with a value that follows a second colon. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/140/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
688389933 | MDU6SXNzdWU2ODgzODk5MzM= | 143 | Move to GitHub Actions CI | 9599 | closed | 0 | 1 | 2020-08-28T22:34:11Z | 2020-08-28T22:41:35Z | 2020-08-28T22:41:35Z | OWNER | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/143/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
695276328 | MDU6SXNzdWU2OTUyNzYzMjg= | 148 | More attractive indentation of created FTS table schema | 9599 | closed | 0 | 1 | 2020-09-07T16:49:30Z | 2020-09-07T18:12:50Z | 2020-09-07T18:12:50Z | OWNER | On https://github-to-sqlite.dogsheep.net/github/licenses_fts the create table SQL is displayed as: ```sql CREATE VIRTUAL TABLE [licenses_fts] USING FTS5 ( [name], content=[licenses] ); ``` It would be more aesthetically pleasing if it looked like this: ```sql CREATE VIRTUAL TABLE [licenses_fts] USING FTS5 ( [name], content=[licenses] ); ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/148/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
705190723 | MDU6SXNzdWU3MDUxOTA3MjM= | 160 | table.enable_fts(..., replace=True) | 9599 | closed | 0 | 5896742 | 1 | 2020-09-20T21:36:23Z | 2020-09-24T20:35:47Z | 2020-09-20T22:05:51Z | OWNER | I noticed that https://til.simonwillison.net/ search doesn't use porter stemming. I'd like to add that, but since [the build script](https://github.com/simonw/til/blob/9d3f0fca30e94df3970df52b0447907a077e4673/build_database.py) always operates on an existing database (to avoid re-rendering markdown and re-building image thumbnails) I'd like it to only add porter stemming if it's not there already. So I'd like to be able to say "set up FTS to look like this, and fix it if it doesn't". I think the neatest way to do that is with a `replace=True` argument to `.enable_fts()`, for consistency with `def .create_view(self, name, sql, replace=True)`. So the `replace=True` argument would check and see if the configured FTS exists already with the correct options (columns, stemming, triggers) - and if any of those are incorrect it would call `.disable_fts()` and then create a new FTS configuration with the correct options. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/160/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
706098005 | MDU6SXNzdWU3MDYwOTgwMDU= | 167 | Review the foreign key pragma stuff | 9599 | closed | 0 | 5897911 | 1 | 2020-09-22T05:55:20Z | 2020-09-23T00:13:02Z | 2020-09-23T00:13:02Z | OWNER | > It is not possible to enable or disable foreign key constraints in the middle of a multi-statement transaction (when SQLite is not in autocommit mode). Attempting to do so does not return an error; it simply has no effect. https://sqlite.org/foreignkeys.html | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/167/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
706768798 | MDU6SXNzdWU3MDY3Njg3OTg= | 170 | Release notes for 2.20 | 9599 | closed | 0 | 5897911 | 1 | 2020-09-23T00:13:22Z | 2020-09-23T00:31:25Z | 2020-09-23T00:31:25Z | OWNER | https://github.com/simonw/sqlite-utils/compare/2.19...b8e004 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/170/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
708301810 | MDU6SXNzdWU3MDgzMDE4MTA= | 177 | Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key | 9599 | closed | 0 | 1 | 2020-09-24T16:13:50Z | 2020-09-24T20:35:03Z | 2020-09-24T16:19:13Z | OWNER | These both currently require you to provide three strings, for `column`, `other_table`, `other_column`. Just providing `column` should be enough information. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/177/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
709861194 | MDU6SXNzdWU3MDk4NjExOTQ= | 180 | Try running some tests using Hypothesis | 9599 | closed | 0 | 1 | 2020-09-28T01:11:30Z | 2020-10-19T04:51:55Z | 2020-10-19T04:51:55Z | OWNER | Inspired by this Twitter conversation: https://twitter.com/simonw/status/1310386009465479168 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/180/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
709920027 | MDU6SXNzdWU3MDk5MjAwMjc= | 181 | pk=["id"] should have same effect as pk="id" | 9599 | closed | 0 | 1 | 2020-09-28T04:28:07Z | 2020-10-14T21:59:47Z | 2020-10-14T21:59:47Z | OWNER | ``` In [11]: db['one'].insert({"id": 1, "name": "oentuh"}, pk="id") Out[11]: <Table one (id, name)> In [12]: db['two'].insert({"id": 1, "name": "oentuh"}, pk=["id"]) Out[12]: <Table two (id, name)> In [13]: db['one'].schema Out[13]: 'CREATE TABLE [one] (\n [id] INTEGER PRIMARY KEY,\n [name] TEXT\n)' In [14]: db['two'].schema Out[14]: 'CREATE TABLE [two] (\n [id] INTEGER,\n [name] TEXT\n)' ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/181/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
712316959 | MDU6SXNzdWU3MTIzMTY5NTk= | 183 | Try out GitHub code scanning | 9599 | closed | 0 | 1 | 2020-09-30T22:16:14Z | 2020-09-30T22:23:44Z | 2020-09-30T22:23:44Z | OWNER | https://github.blog/2020-09-30-code-scanning-is-now-available/ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/183/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
737855731 | MDU6SXNzdWU3Mzc4NTU3MzE= | 199 | @db.register_function(..., replace=False) to avoid double-registering custom functions | 9599 | closed | 0 | 1 | 2020-11-06T15:39:21Z | 2020-11-06T18:30:44Z | 2020-11-06T18:30:44Z | OWNER | I'd like a mechanism to optionally avoid registering a custom function if it has already been registered. SQLite doesn't seem to offer a way to introspect registered custom functions so I'll need to track what has already been registered in `sqlite-utils` instead. > Should I register the custom `rank_bm25` SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed. _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383_ | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/199/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
738115165 | MDU6SXNzdWU3MzgxMTUxNjU= | 200 | sqlite-utils rows -c option | 9599 | closed | 0 | 6079500 | 1 | 2020-11-07T00:22:12Z | 2020-11-07T00:28:48Z | 2020-11-07T00:28:47Z | OWNER | To let you specify the exact columns you want. Based on the `-c` option to `sqlite-utils search` in #192. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/200/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
738128913 | MDU6SXNzdWU3MzgxMjg5MTM= | 201 | .search(columns=) and sqlite-utils search -c ... bug | 9599 | closed | 0 | 6079500 | 1 | 2020-11-07T01:27:26Z | 2020-11-08T16:54:15Z | 2020-11-08T16:54:15Z | OWNER | Both `table.search(columns=)` and the `sqlite-utils search -c` option do not work as expected - they always return both the `rowid` and the `rank` columns even if those have not been requested. This should be fixed before the 3.0 non-alpha release. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/201/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
777392020 | MDU6SXNzdWU3NzczOTIwMjA= | 212 | Mechanism for maintaining cache of table counts using triggers | 9599 | closed | 0 | 1 | 2021-01-02T02:58:53Z | 2021-01-02T21:40:27Z | 2021-01-02T21:40:27Z | OWNER | Counting all of the rows in a large table is expensive - this is one of the main causes of performance problems in Datasette when running against large databases. Carefully constructed SQL triggers could be used to maintain accurate cached counts for a table, by incrementing and decrementing a counter every time a row is inserted or deleted. `sqlite-utils` already has a mechanism for creating triggers for FTS - the `table.enable_fts(..., create_triggers=True)` method. How about a similar mechanism for setting up triggers to maintain a count of table rows? | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/212/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
777540352 | MDU6SXNzdWU3Nzc1NDAzNTI= | 216 | database.triggers_dict introspection property | 9599 | closed | 0 | 1 | 2021-01-02T23:13:00Z | 2021-01-03T04:27:14Z | 2021-01-03T04:25:36Z | OWNER | Following #211 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/216/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
777560474 | MDU6SXNzdWU3Nzc1NjA0NzQ= | 218 | "sqlite-utils triggers" command | 9599 | closed | 0 | 1 | 2021-01-03T02:34:50Z | 2021-01-03T03:49:51Z | 2021-01-03T03:03:35Z | OWNER | A command to list the triggers in the database. sqlite-utils triggers my.db Can optionally take one or more tables: sqlite-utils triggers my.db table1 table2 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/218/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
808046597 | MDU6SXNzdWU4MDgwNDY1OTc= | 234 | .insert_all() fails if subsequent chunks contain additional columns | 9599 | closed | 0 | 1 | 2021-02-14T21:01:51Z | 2021-02-14T21:03:40Z | 2021-02-14T21:03:40Z | OWNER | Reported by @nieuwenhoven in #225 along with a proposed fix. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/234/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
811680502 | MDU6SXNzdWU4MTE2ODA1MDI= | 236 | --attach command line option for attaching extra databases | 9599 | closed | 0 | 1 | 2021-02-19T04:38:30Z | 2021-02-19T05:10:41Z | 2021-02-19T05:08:43Z | OWNER | This will enable cross-database joins, as seen in https://github.com/simonw/datasette/issues/283 Also refs #113 | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/236/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
816523763 | MDU6SXNzdWU4MTY1MjM3NjM= | 238 | .add_foreign_key() corrupts database if column contains a space | 9599 | closed | 0 | 1 | 2021-02-25T15:07:20Z | 2021-02-25T16:54:02Z | 2021-02-25T16:54:02Z | OWNER | I ran this: db["Reports"].add_foreign_key("Reported by ID", "Reporters", "id") And got this: ``` ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in add_foreign_keys(self, foreign_keys) 616 # Have to VACUUM outside the transaction to ensure .foreign_keys property 617 # can see the newly created foreign key. --> 618 self.vacuum() 619 620 def index_foreign_keys(self): ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in vacuum(self) 629 630 def vacuum(self): --> 631 self.execute("VACUUM;") 632 633 ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in execute(self, sql, parameters) 234 return self.conn.execute(sql, parameters) 235 else: --> 236 return self.conn.execute(sql) 237 238 def executescript(self, sql): DatabaseError: database disk image is malformed ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/238/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
894948100 | MDU6SXNzdWU4OTQ5NDgxMDA= | 259 | Suggest the --alter option if a new column cannot be added | 9599 | closed | 0 | 1 | 2021-05-19T03:17:38Z | 2021-05-19T03:27:33Z | 2021-05-19T03:26:26Z | OWNER | Refs #256. | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/259/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
919702451 | MDU6SXNzdWU5MTk3MDI0NTE= | 271 | table.upsert_all() fails if input has a single column that should be a primary key | 9599 | closed | 0 | 1 | 2021-06-13T02:50:27Z | 2021-06-13T02:57:29Z | 2021-06-13T02:57:29Z | OWNER | This works: ```pycon >>> db['foo'].insert_all([{"name": "hello"}], pk="name") <Table foo (name)> ``` But this fails: ``` >>> db['foo3'].upsert_all([{"name": "hello"}], pk="name") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1837, in upsert_all return self.insert_all( File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1778, in insert_all self.insert_chunk( File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1588, in insert_chunk result = self.db.execute(query, params) File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 213, in execute return self.conn.execute(sql, parameters) sqlite3.OperationalError: near "WHERE": syntax error ``` With the debugger: ``` >>> import pdb; pdb.pm() > /Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py(213)execute() -> return self.conn.execute(sql, parameters) (Pdb) print(sql, parameters) UPDATE [foo3] SET WHERE [name] = ? ['hello'] ``` | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/271/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
922955697 | MDU6SXNzdWU5MjI5NTU2OTc= | 275 | Enable code coverage | 9599 | closed | 0 | 1 | 2021-06-16T18:33:49Z | 2021-06-17T00:12:12Z | 2021-06-17T00:12:12Z | OWNER | https://app.codecov.io/gh/simonw/sqlite-utils Same mechanism as Datasette. Need to copy across the token from that page and add an equivalent of this workflow: https://github.com/simonw/datasette/blob/main/.github/workflows/test-coverage.yml | 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/275/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed |