issues
604 rows where repo = 140912432 sorted by node_id
This data as json, CSV (advanced)
id | node_id ▼ | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1243704847 | I_kwDOCGYnMM5KIW4P | 435 | Switch to Furo documentation theme | simonw 9599 | closed | 0 | 2 | 2022-05-20T21:46:39Z | 2022-05-20T21:56:10Z | 2022-05-20T21:54:43Z | OWNER | sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/435/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
1243715381 | I_kwDOCGYnMM5KIZc1 | 436 | Add "copy to clipboard" button to code examples in documentation | simonw 9599 | closed | 0 | 0 | 2022-05-20T21:53:23Z | 2022-05-20T21:57:53Z | 2022-05-20T21:57:53Z | OWNER | Follows: - #435 Imitates: - https://github.com/simonw/datasette/issues/1748 I'll use https://github.com/executablebooks/sphinx-copybutton - here's the Datasette commit: https://github.com/simonw/datasette/commit/1465fea4798599eccfe7e8f012bd8d9adfac3039 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/436/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1250161887 | I_kwDOCGYnMM5Kg_Tf | 438 | illegal UTF-16 surrogate | frafra 4068 | closed | 0 | 2 | 2022-05-26T22:49:52Z | 2022-05-27T08:21:53Z | 2022-05-27T08:21:53Z | NONE | I am trying to insert ``` sqlite-utils insert --csv --delimiter ";" --encoding="utf-16-le" --pk "Id" csv fremmedart test.db [------------------------------------] 0% Error: 'utf-16-le' codec can't decode bytes in position 98-99: illegal UTF-16 surrogate The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` I tried to convert the file using ``` sqlite-utils insert --csv --delimiter ";" --encoding=utf-8 --pk "Id" csv_utf8 fremmedart test.db [------------------------------------] 0% Error: 'utf-8' codec can't decode byte 0xd9 in position 99: invalid continuation byte The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` I have no issues reading such file using this Python code:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/438/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1250495688 | I_kwDOCGYnMM5KiQzI | 439 | Misleading progress bar against utf-16-le CSV input | frafra 4068 | open | 0 | 12 | 2022-05-27T08:34:49Z | 2022-06-15T03:53:43Z | NONE | The program crashes without any error.
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1250629388 | I_kwDOCGYnMM5KixcM | 440 | CSV files with too many values in a row cause errors | frafra 4068 | closed | 0 | 20 | 2022-05-27T10:54:44Z | 2022-06-14T22:23:01Z | 2022-06-14T20:12:46Z | NONE | Original title: csv.DictReader can have None as key In some cases, ```python url="https://artsdatabanken.no/Fab2018/api/export/csv" db = sqlite_utils.Database(":memory") with urlopen(url) as fab:
reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le") Result:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1269886084 | I_kwDOCGYnMM5LsOyE | 442 | `maximize_csv_field_size_limit()` utility function | simonw 9599 | closed | 0 | 2 | 2022-06-13T19:54:54Z | 2022-06-14T21:55:15Z | 2022-06-14T21:31:49Z | OWNER | This code here runs only if I found myself needing the same fix in another library: It should be a documented utility function. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/442/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1269998342 | I_kwDOCGYnMM5LsqMG | 443 | Make `utils.rows_from_file()` a documented API | simonw 9599 | closed | 0 | 2 | 2022-06-13T21:53:24Z | 2022-06-20T19:49:37Z | 2022-06-14T20:12:46Z | OWNER |
Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/443/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1271426387 | I_kwDOCGYnMM5LyG1T | 444 | CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool | simonw 9599 | open | 0 | 5 | 2022-06-14T22:22:47Z | 2022-07-07T16:39:18Z | OWNER |
Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/444/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1277295119 | I_kwDOCGYnMM5MIfoP | 445 | `sqlite_utils.utils.TypeTracker` should be a documented API | simonw 9599 | closed | 0 | 3 | 2022-06-20T19:08:28Z | 2022-06-20T19:49:02Z | 2022-06-20T19:46:58Z | OWNER | I've used it in a couple of external places now: |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/445/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1277328147 | I_kwDOCGYnMM5MInsT | 446 | Use Just to automate running tests and linters locally | simonw 9599 | closed | 0 | 2 | 2022-06-20T19:51:09Z | 2022-06-21T19:28:35Z | 2022-06-20T19:54:50Z | OWNER | I keep committing code that fails additional tests like |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/446/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1278571700 | I_kwDOCGYnMM5MNXS0 | 447 | Incorrect syntax highlighting in docs CLI reference | simonw 9599 | closed | 0 | 3 | 2022-06-21T14:53:10Z | 2022-06-21T18:48:47Z | 2022-06-21T18:48:46Z | OWNER | https://sqlite-utils.datasette.io/en/stable/cli-reference.html#insert It looks like Python keywords are being incorrectly highlighted here. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/447/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1279144769 | I_kwDOCGYnMM5MPjNB | 448 | Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto' | mungewell 236907 | closed | 0 | 5 | 2022-06-21T21:48:27Z | 2023-05-08T22:01:00Z | 2023-05-08T22:01:00Z | NONE | Attempting to run the example given here (without extra bracket ;-): https://sqlite-utils.datasette.io/en/stable/python-api.html#reading-rows-from-a-file ``` from sqlite_utils.utils import rows_from_file import io rows, format = rows_from_file(io.StringIO("id,name\n1,Cleo")) print(list(rows), format) Outputs [{'id': '1', 'name': 'Cleo'}] Format.CSV``` Gives error ```
I am running Python on Windows. ```
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/448/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1279863844 | I_kwDOCGYnMM5MSSwk | 449 | Utilities for duplicating tables and creating a table with the results of a query | davidleejy 1690072 | closed | 0 | 4 | 2022-06-22T09:41:43Z | 2022-07-15T21:46:13Z | 2022-07-15T21:21:36Z | CONTRIBUTOR | is there a duplicate table functionality? Otherwise, I'd be happy to submit a PR. In sqlite3 it would look like: ```python import sqlite3 as sl con = sl.connect('prompt-tune.db') def db_duplicate_table(table_name, table_name_new, con=con):
# Duplicates table db_duplicate_table('orig_table', 'new_table') ``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/449/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1306548397 | I_kwDOCGYnMM5N4Fit | 454 | CLI command for duplicating tables | simonw 9599 | closed | 0 | 1 | 2022-07-15T21:31:27Z | 2022-07-15T21:48:23Z | 2022-07-15T21:45:51Z | OWNER | CLI equivalent of: - #449 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/454/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1292060682 | I_kwDOCGYnMM5NA0gK | 450 | Add --ignore option to more commands | simonw 9599 | closed | 0 | 9 | 2022-07-02T13:52:02Z | 2022-07-15T22:39:09Z | 2022-07-15T22:37:45Z | OWNER | As seen in https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key Could make this TIL trick unnecessary: https://til.simonwillison.net/bash/ignore-errors |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/450/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1298531653 | I_kwDOCGYnMM5NZgVF | 451 | Make sqlite_utils.utils.chunks a documented function | simonw 9599 | closed | 0 | 2 | 2022-07-08T06:01:04Z | 2022-07-15T22:09:34Z | 2022-07-15T21:59:33Z | OWNER | I want to use it in another project: https://github.com/simonw/sqlite-utils/blob/8a9fe6498faf783a1fdeb1793e661ad194a05267/sqlite_utils/utils.py#L471-L474 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/451/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1303169663 | I_kwDOCGYnMM5NrMp_ | 453 | 'unclosed file' warning when using insert_upsert_implementation from Python | makkus 311257 | closed | 0 | 1 | 2022-07-13T09:34:35Z | 2022-07-15T21:52:25Z | 2022-07-15T21:52:21Z | NONE | I'm using the The warning goes away when wrapping the code from this line in a try/finally block like:
I suspect Python closes the reference automatically when the sqlite-utils cli run is done, but since my code doesn't exit, I'm getting the warning. Alternatively, it'd be cool if the 'import csv/tsv' functionality could be added directly to the Database class. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/453/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1324659241 | I_kwDOCGYnMM5O9LIp | 459 | Single quoted transform recipes on Windows do not work as expected | shakeel 19921 | open | 0 | 0 | 2022-08-01T16:14:54Z | 2022-08-01T16:14:54Z | CONTRIBUTOR | Trying to follow the tutorial for sqlite-utils and datasette https://datasette.io/tutorials/clean-data on Windows 11 OS
In the step to transform dates into ISO dates the quoted value ``` sqlite-utils convert manatees.db locations \ REPDATE created_date last_edited_date \ 'r.parsedatetime(value)' --dry-run 1975/01/31 00:00:00+00 --- becomes: r.parsedatetime(value) Would affect 13568 rows ``` However, if I change the code from single quotes to double quotes, it works as expected. ``` sqlite-utils convert manatees.db locations \ REPDATE created_date last_edited_date \ "r.parsedatetime(value)" --dry-run 1975/01/31 00:00:00+00 --- becomes: 1975-01-31T00:00:00+00:00 Would affect 13568 rows ``` Specifying the transform code recipe should work with single quotes on Windows. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/459/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1310243385 | I_kwDOCGYnMM5OGLo5 | 456 | feature request: pivot command | fgregg 536941 | open | 0 | 5 | 2022-07-20T00:58:08Z | 2022-07-20T17:50:50Z | CONTRIBUTOR | pivoting long-format table to wide-format tables is pretty common and kind of pain. would love to see this feature in sqlite-utils! |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/456/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1320243134 | I_kwDOCGYnMM5OsU-- | 458 | Support custom names for registered functions | simonw 9599 | closed | 0 | 3.29 8355157 | 1 | 2022-07-28T00:13:00Z | 2022-08-27T03:56:01Z | 2022-07-28T00:13:57Z | OWNER | In this example: ```python @db.register_function def reverse_string(s): return "".join(reversed(list(s)))
``` There's currently no way to over-ride the automatically selected name for the SQL function. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/458/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1326349129 | I_kwDOCGYnMM5PDntJ | 461 | Consider including animated SVG console demos | simonw 9599 | open | 0 | 1 | 2022-08-02T20:10:04Z | 2022-08-02T20:12:14Z | OWNER | I recorded this one using https://github.com/nbedos/termtosvg - with
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/461/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1338001039 | I_kwDOCGYnMM5PwEaP | 464 | Link from documentation to source code | simonw 9599 | closed | 0 | 5 | 2022-08-13T16:19:57Z | 2022-08-17T23:38:03Z | 2022-08-17T23:38:03Z | OWNER | Twitter conversation asking for ways to automate this here: https://twitter.com/simonw/status/1558260492015046656 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/464/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1348169997 | I_kwDOCGYnMM5QW3EN | 467 | Mechanism for ensuring a table has all the columns | simonw 9599 | closed | 0 | 3.29 8355157 | 13 | 2022-08-23T15:50:23Z | 2022-08-27T23:19:41Z | 2022-08-27T23:17:56Z | OWNER | Suggested by @jefftriplett on Discord: https://discord.com/channels/823971286308356157/997738192360964156/1011655389063958600 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/467/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352932716 | I_kwDOCGYnMM5QpB1s | 471 | sqlite-utils query --functions mechanism for registering extra functions | simonw 9599 | closed | 0 | 3.29 8355157 | 12 | 2022-08-27T03:57:53Z | 2022-09-07T03:46:26Z | 2022-08-27T05:10:57Z | OWNER | It would be really cool if you could register additional custom SQL functions for use with the ``` sqlite-utils data.db 'update images set domain = extract_domain(url)' --functions ' from urllib.parse import urlparse def extract_domain(url): return urlparse(url).netloc ' ``` Every function defined in that code block would be registered with the connection, unless the name began with an underscore. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352931464 | I_kwDOCGYnMM5QpBiI | 469 | sqlite-utils rows --order option | simonw 9599 | closed | 0 | 3.29 8355157 | 1 | 2022-08-27T03:49:51Z | 2022-08-27T04:30:49Z | 2022-08-27T04:10:32Z | OWNER | For consistency with
I wanted to run |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/469/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352932038 | I_kwDOCGYnMM5QpBrG | 470 | Upgrade `--load-extension` to accept entrypoints like Datasette | simonw 9599 | closed | 0 | 3.29 8355157 | 6 | 2022-08-27T03:53:20Z | 2022-08-27T05:55:49Z | 2022-08-27T05:55:48Z | OWNER | Imitate: - https://github.com/simonw/datasette/pull/1789 ``` would load default entrypoint like beforedatasette data.db --load-extension ext loads the extensions with the "sqlite3_foo_init" entrpointdatasette data.db --load-extension ext:sqlite3_foo_init loads the extensions with the "sqlite3_bar_init" entrpointdatasette data.db --load-extension ext:sqlite3_bar_init ``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352946135 | I_kwDOCGYnMM5QpFHX | 472 | Reuse the locals/globals fix from --functions for other code accepting options | simonw 9599 | closed | 0 | 3.29 8355157 | 2 | 2022-08-27T05:12:05Z | 2022-08-27T05:20:12Z | 2022-08-27T05:20:12Z | OWNER | I figured out a workaround for the ugly |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/472/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1353074021 | I_kwDOCGYnMM5QpkVl | 474 | Add an option for specifying column names when inserting CSV data | hubgit 14294 | open | 0 | 3 | 2022-08-27T15:29:59Z | 2022-08-31T03:42:36Z | NONE | https://sqlite-utils.datasette.io/en/stable/cli.html#csv-files-without-a-header-row
It would be nice to be able to specify the column names when importing CSV/TSV without a header row, via an extra command line option. (renaming a column of a large table can take a long time, which makes it an inconvenient workaround) |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/474/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1353441389 | I_kwDOCGYnMM5Qq-Bt | 477 | Conda Forge | thewchan 49702524 | closed | 0 | 2 | 2022-08-28T19:03:08Z | 2022-09-07T03:46:55Z | 2022-09-07T03:46:55Z | NONE | Hello! I have successfully put this package on to Conda Forge, and I have extending the invitation for the owner/maintainers of this package to be maintainers on Conda Forge as well. Let me know if you are interested! Thanks. https://github.com/conda-forge/sqlite-utils-feedstock |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/477/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1353189941 | I_kwDOCGYnMM5QqAo1 | 475 | table.default_values introspection property | simonw 9599 | closed | 0 | 3.29 8355157 | 1 | 2022-08-27T22:33:31Z | 2022-08-27T22:44:46Z | 2022-08-27T22:43:02Z | OWNER |
Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/468#issuecomment-1229279539 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/475/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1353196970 | I_kwDOCGYnMM5QqCWq | 476 | Release notes for 3.29 | simonw 9599 | closed | 0 | 3.29 8355157 | 2 | 2022-08-27T23:21:21Z | 2022-08-28T04:07:15Z | 2022-08-28T04:07:03Z | OWNER | sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/476/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1353481513 | I_kwDOCGYnMM5QrH0p | 478 | `sqlite-utils tables data.db table1 table2` | simonw 9599 | open | 0 | 1 | 2022-08-28T22:05:53Z | 2022-08-28T22:22:35Z | OWNER | The If you have a huge table in there then running it with Would be useful if it could accept an optional list of tables that it should execute against, as an alternative to the default of all of them. This should be a backwards compatible change. Current design is: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#tables ``` Usage: sqlite-utils tables [OPTIONS] PATH List the tables in the database Example:
``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/478/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1355193529 | I_kwDOCGYnMM5Qxpy5 | 479 | OperationalError: cannot VACUUM from within a transaction | chapmanjacobd 7908073 | open | 0 | 0 | 2022-08-30T05:34:24Z | 2022-08-30T05:34:24Z | CONTRIBUTOR | Maybe when calling ``` 46 db["media"].optimize() # type: ignore ---> 47 db.vacuum() File ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:1047, in Database.vacuum(self)
1045 def vacuum(self):
1046 "Run a SQLite File ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:470, in Database.execute(self, sql, parameters) 468 return self.conn.execute(sql, parameters) 469 else: --> 470 return self.conn.execute(sql) OperationalError: cannot VACUUM from within a transaction ``` It might also be nice to add a sentence or two about how transactions are committed on the docs page. When I was swapping out my sqlite3 code for this library it was nice that everything was pretty much drop-in but I was/am unsure what to do about the places I explicitly call Related to https://github.com/simonw/sqlite-utils/issues/121 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/479/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1373224657 | I_kwDOCGYnMM5R2b7R | 488 | `sqlite-utils transform` should set empty strings to null when converting text columns to integer/float | simonw 9599 | open | 0 | 5 | 2022-09-14T15:51:30Z | 2022-12-23T17:38:55Z | OWNER |
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/488/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1374939463 | I_kwDOCGYnMM5R8-lH | 489 | Ability to load JSON records held in a file with a single top level key that is a list of objects | simonw 9599 | open | 0 | 9 | 2022-09-15T18:46:03Z | 2022-09-15T20:56:10Z | OWNER | It's very common for JSON to look like this:
Right now you can't import this into But since this is so common, it would be neat if |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1359604075 | I_kwDOCGYnMM5RCelr | 481 | Idea: `sqlite-utils create-table tablename --sql "select ..."` | simonw 9599 | open | 0 | 0 | 2022-09-02T01:41:24Z | 2022-09-02T01:42:08Z | OWNER | Could offer syntactic sugar for:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/481/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1361355564 | I_kwDOCGYnMM5RJKMs | 482 | balanced table default column_order | chapmanjacobd 7908073 | closed | 0 | 1 | 2022-09-05T03:00:18Z | 2022-10-10T17:43:02Z | 2022-09-06T20:17:27Z | CONTRIBUTOR | Is there any performance or size difference with column order in SQLITE ? similar to this https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/ It might be interesting to have an option to create with an optimized column order. I'm assuming this would look something like INTEGER columns, REAL columns, BLOB columns, TEXT columns, NULL columns. NULL columns at the end because they are more likely to be TEXT and it is impossible to know if they will become INTEGER (Of course, any schema evolution would reduce optimization but maybe column order could also be re-evaluated when schema changes) edit: this is easy to accomplish with the existing
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/482/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1363766973 | I_kwDOCGYnMM5RSW69 | 484 | Expose convert recipes to `sqlite-utils --functions` | simonw 9599 | open | 0 | 11 | 2022-09-06T20:15:08Z | 2022-09-07T19:09:52Z | OWNER |
It would be useful if the |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/484/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1363765916 | I_kwDOCGYnMM5RSWqc | 483 | `sqlite-utils install` command | simonw 9599 | closed | 0 | 2 | 2022-09-06T20:13:55Z | 2022-09-26T19:04:43Z | 2022-09-26T18:57:15Z | OWNER | With the addition of In addition to the existing This isn't easy if you installed the tool with Datasette solved this problem with the
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/483/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1366423176 | I_kwDOCGYnMM5RcfaI | 485 | Progressbar not shown when inserting/upserting jsonlines file | MischaU8 99098079 | closed | 0 | 1 | 2022-09-08T14:13:18Z | 2022-09-15T20:39:52Z | 2022-09-15T20:37:52Z | CONTRIBUTOR | When inserting or upserting a jsonlines file, no progressbar is shown. Expected behavior is that, just like with .csv/.tsv files, also for a jsonlines file (--nl), unless --silent is provided, a progressbar is shown.
Currently |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/485/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1367835380 | I_kwDOCGYnMM5Rh4L0 | 487 | Specify foreign key against compound key in other table | ryanfox 540968 | closed | 0 | 2 | 2022-09-09T13:32:09Z | 2022-09-11T04:00:44Z | 2022-09-11T04:00:44Z | NONE | When inserting rows via the library, is it possible to specify a foreign key to a compound primary key? For example, suppose I create a table:
And I want to add related data in another table:
Is it possible to specify a value for SQLite does support it: https://www.sqlite.org/foreignkeys.html#fk_composite |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/487/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1382457780 | I_kwDOCGYnMM5SZqG0 | 490 | Ability to insert multi-line files | jeqo 6180701 | closed | 0 | 4 | 2022-09-22T13:29:22Z | 2022-09-26T18:24:44Z | 2022-09-23T16:37:58Z | NONE | I was looking into how to parse application log files that contain multiline text (e.g. Java stack traces) into sqlite.
I can see that at the moment I wonder if this functionality would be useful for sqlite-utils. A similar approach to Elastic logstash/filebeat can be adopted: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html Potential changes:
Or if this is achievable in a different way, please share. Thanks! |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1383646615 | I_kwDOCGYnMM5SeMWX | 491 | Ability to merge databases and tables | sgraaf 8904453 | open | 0 | 7 | 2022-09-23T11:10:55Z | 2023-06-14T22:14:24Z | NONE | Hi! Let me firstly say that I am a big fan of your work -- I follow your tweets and blog posts with great interest 😄. Now onto the matter at hand: I think it would be great if This could look something like this:
I imagine this is rather straightforward if all databases involved in the merge contain differently named tables (i.e. no chance of conflicts), but things get slightly more complicated if two or more of the databases to be merged contain tables with the same name. Not only do you have to "do something" with the primary key(s), but these tables could also simply have different schemas (and therefore be incompatible for concatenation to begin with). Anyhow, I would love your thoughts on this, and, if you are open to it, work together on the design and implementation! |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1386530156 | I_kwDOCGYnMM5SpMVs | 492 | Idea: ability to pass extra variables to `--convert` scripts | simonw 9599 | open | 0 | 1 | 2022-09-26T18:30:45Z | 2022-09-26T18:33:19Z | OWNER | Got this idea from this example in https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/
This made me think: it might be neat if you could inject additional variable values into that script with extra command-line options, to make this kind of reuse easier. Something like this:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/492/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1386562662 | I_kwDOCGYnMM5SpURm | 493 | Tiny typographical error in install/uninstall docs | simonw 9599 | open | 0 | 3 | 2022-09-26T19:00:42Z | 2022-10-25T21:31:15Z | OWNER | Added in: - #483 I don't know how to fix this in Sphinx: I'm getting this: https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install
But I want it to display Here's the code: https://github.com/simonw/sqlite-utils/blob/85247038f70d7eb2f3e272cfeaa4c44459cafba8/docs/cli.rst#L2125 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/493/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1386593843 | I_kwDOCGYnMM5Spb4z | 494 | Document how to use Just | simonw 9599 | closed | 0 | 2 | 2022-09-26T19:25:12Z | 2022-09-26T19:32:36Z | 2022-09-26T19:26:39Z | OWNER | I'm using https://github.com/simonw/sqlite-utils/blob/afbd2b2cba45cccb305c3d4638d18db4dd3d4bbd/Justfile#L1-L24 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/494/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1392690202 | I_kwDOCGYnMM5TAsQa | 495 | Support JSON values returned from .convert() functions | mhalle 649467 | closed | 0 | 3 | 2022-09-30T16:33:49Z | 2022-10-25T21:23:37Z | 2022-10-25T21:23:28Z | NONE | When using the convert function on a JSON column, the result of the conversion function must be a string. If the return value is either a dict (object) or a list (array), the convert call will error out with an unhelpful user defined function exception. It makes sense that since the original column value was a string and required conversion to data structures, the result should be converted back into a JSON string as well. However, other functions auto-convert to JSON string representation, so the fact that convert doesn't could be surprising. At least the documentation should note this requirement, because the sqlite error messages won't readily reveal the issue. Jf only sqlite's JSON column type meant something :) |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/495/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1393202060 | I_kwDOCGYnMM5TCpOM | 496 | devrel/python api: Pylance type hinting | chapmanjacobd 7908073 | open | 0 | 4 | 2022-10-01T03:03:34Z | 2023-05-03T05:53:27Z | CONTRIBUTOR | Pylance is generally pretty good at figuring out stuff but For example:
I think this is because DEFAULT is an empty class? maybe a few small changes could be made to make the library more type-friendly The interim solution is of course to turn off type hints completely for the line
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/496/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1393212964 | I_kwDOCGYnMM5TCr4k | 497 | column_names | chapmanjacobd 7908073 | closed | 0 | 1 | 2022-10-01T03:34:21Z | 2022-10-25T21:09:28Z | 2022-10-25T21:09:28Z | CONTRIBUTOR | It would be nice to have a Or if you could get one or all of the following syntax to work for both Database and Table that might be even better: Style 1
- Style 2
- maybe the table ones actually work but I'm too lazy to check. I just know that I have to do:
Edit: This is possible with |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/497/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1422954582 | I_kwDOCGYnMM5U0JBW | 502 | Fix tests for Python 3.11 | simonw 9599 | closed | 0 | 1 | 2022-10-25T19:20:31Z | 2022-10-25T19:23:47Z | 2022-10-25T19:23:47Z | OWNER | The way errors are represented has changed: https://github.com/simonw/sqlite-utils/actions/runs/3323588047/jobs/5494127154 ``` ____ testquery_invalid_function ______ db_path = '/tmp/pytest-of-runner/pytest-0/test_query_invalid_function0/test.db'
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/502/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423000702 | I_kwDOCGYnMM5U0UR- | 503 | test_recreate failing on Windows Python 3.11 | simonw 9599 | closed | 0 | 10 | 2022-10-25T20:01:41Z | 2022-10-25T20:47:34Z | 2022-10-25T20:45:43Z | OWNER | https://github.com/simonw/sqlite-utils/actions/runs/3323672128/jobs/5494726927 Related: - #502
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/503/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423069384 | I_kwDOCGYnMM5U0lDI | 504 | db.close() method, calling db.conn.close() | simonw 9599 | closed | 0 | 1 | 2022-10-25T20:50:50Z | 2022-10-25T21:00:29Z | 2022-10-25T20:57:47Z | OWNER | I ended up needing to use I think |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/504/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423182778 | I_kwDOCGYnMM5U1Au6 | 505 | Release sqlite-utils 3.30 | simonw 9599 | closed | 0 | 2 | 2022-10-25T22:20:05Z | 2022-10-25T22:41:26Z | 2022-10-25T22:41:16Z | OWNER | sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/505/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
1413610718 | I_kwDOCGYnMM5UQfze | 500 | Turn --flatten into a documented utility function | simonw 9599 | closed | 0 | 4 | 2022-10-18T17:43:36Z | 2022-10-18T18:02:10Z | 2022-10-18T18:00:40Z | OWNER | The |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/500/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1413641049 | I_kwDOCGYnMM5UQnNZ | 501 | Tests failing due to updated tabulate library | simonw 9599 | closed | 0 | 4 | 2022-10-18T18:07:52Z | 2022-10-18T18:23:40Z | 2022-10-18T18:23:40Z | OWNER | Failure here: https://github.com/simonw/sqlite-utils/actions/runs/3275786702/jobs/5391063221 I figured out the problem:
+- |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/501/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1429029604 | I_kwDOCGYnMM5VLULk | 506 | Make `cursor.rowcount` accessible (wontfix) | simonw 9599 | closed | 0 | 3 | 2022-10-30T21:51:55Z | 2022-11-01T17:37:47Z | 2022-11-01T17:37:13Z | OWNER | In building this Datasette feature on top of This isn't currently exposed by |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/506/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1430325103 | I_kwDOCGYnMM5VQQdv | 507 | conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character | chapmanjacobd 7908073 | closed | 0 | 1 | 2022-10-31T18:49:51Z | 2022-11-01T00:40:17Z | 2022-11-01T00:40:16Z | CONTRIBUTOR | I'm not really sure what caused this and it happened in the middle of my program (after running for 35775 seconds).
This might be relevant: https://stackoverflow.com/questions/31898353/python-cant-encode-with-surrogateescape I'm going to try re-running with
``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/507/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1432377191 | I_kwDOCGYnMM5VYFdn | 509 | `sqlite-utils transform` breaks DEFAULT string values and STRFTIME() | kennysong 2199875 | closed | 0 | 0 | 2022-11-02T02:32:23Z | 2023-05-08T21:13:38Z | 2023-05-08T21:13:38Z | NONE | Very nice library! Our team found sqlite-utils through @simonw's comment on the "Simple declarative schema migration for SQLite" article, and we were excited to use it, but unfortunately Running
Here are steps to reproduce: Original database ``` $ sqlite3 test.db << EOF CREATE TABLE mytable ( col1 TEXT DEFAULT 'foo', col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) ) EOF $ sqlite3 test.db "SELECT sql FROM sqlite_master WHERE name = 'mytable';" CREATE TABLE mytable ( col1 TEXT DEFAULT 'foo', col2 TEXT DEFAULT (STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')) ) ``` Modified database after sqlite-utils ``` $ sqlite3 test.db "INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;" foo|2022-11-02 02:26:58.038 $ sqlite-utils transform test.db mytable --rename col1 renamedcol1 $ sqlite3 test.db "SELECT sql FROM sqlite_master WHERE name = 'mytable';" CREATE TABLE "mytable" ( [renamedcol1] TEXT DEFAULT '''foo''', [col2] TEXT DEFAULT 'STRFTIME(''%Y-%m-%d %H:%M:%f'', ''NOW'')' ) $ sqlite3 test.db "INSERT INTO mytable DEFAULT VALUES; SELECT * FROM mytable;" foo|2022-11-02 02:26:58.038 'foo'|STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW') ``` (Related: #336) |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/509/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1434911255 | I_kwDOCGYnMM5VhwIX | 510 | Cannot enable FTS5 despite it being available | ar-jan 1176293 | closed | 0 | 3 | 2022-11-03T16:03:49Z | 2022-11-18T18:37:52Z | 2022-11-17T10:36:28Z | NONE | When I do FTS5 is however available and Python/SQLite versions do not seem to be the issue. I can manually create the FTS5 virtual table, and then Datasette also works with it from this same Python environment.
Any ideas what's happening and how to fix? |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/510/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1436539554 | I_kwDOCGYnMM5Vn9qi | 511 | [insert_all, upsert_all] IntegrityError: constraint failed | chapmanjacobd 7908073 | closed | 0 | 2 | 2022-11-04T19:21:48Z | 2022-11-04T22:59:54Z | 2022-11-04T22:54:09Z | CONTRIBUTOR | My understand is that ``` import argparse from pathlib import Path from xklb import db, utils from xklb.utils import log def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument("database") parser.add_argument("dbs", nargs="*") parser.add_argument("--upsert") parser.add_argument("--db", "-db", help=argparse.SUPPRESS) parser.add_argument("--verbose", "-v", action="count", default=0) args = parser.parse_args()
def merge_db(args, source_db): source_db = str(Path(source_db).resolve())
def merge_dbs(): args = parse_args() for s_db in args.dbs: merge_db(args, s_db) if name == "main": merge_dbs() ``` ``` $ lb-dev merge video.db tube_71.db --upsert path -vv SQL: INSERT OR IGNORE INTO media VALUES(?); - params: ['https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz'] ... File ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:3122, in Table.insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, hash_id_columns, alter, ignore, replace, truncate, extracts, conversions, columns, upsert, analyze) 3116 all_columns += [ 3117 column for column in record if column not in all_columns 3118 ] 3120 first = False -> 3122 self.insert_chunk( 3123 alter, 3124 extracts, 3125 chunk, 3126 all_columns, 3127 hash_id, 3128 hash_id_columns, 3129 upsert, 3130 pk, 3131 conversions, 3132 num_records_processed, 3133 replace, 3134 ignore, 3135 ) 3137 if analyze: 3138 self.analyze() File ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:2887, in Table.insert_chunk(self, alter, extracts, chunk, all_columns, hash_id, hash_id_columns, upsert, pk, conversions, num_records_processed, replace, ignore) 2885 for query, params in queries_and_params: 2886 try: -> 2887 result = self.db.execute(query, params) 2888 except OperationalError as e: 2889 if alter and (" column" in e.args[0]): 2890 # Attempt to add any missing columns, then try again File ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:484, in Database.execute(self, sql, parameters) 482 self._tracer(sql, parameters) 483 if parameters is not None: --> 484 return self.conn.execute(sql, parameters) 485 else: 486 return self.conn.execute(sql) IntegrityError: constraint failed
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/511/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1450952393 | I_kwDOCGYnMM5We8bJ | 512 | mypy failures in CI | simonw 9599 | closed | 0 | 3 | 2022-11-16T06:22:48Z | 2022-11-16T07:49:51Z | 2022-11-16T07:49:50Z | OWNER | https://github.com/simonw/sqlite-utils/actions/runs/3472012235 failed on Python 3.11: Truncated output:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/512/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1453134846 | I_kwDOCGYnMM5WnRP- | 513 | Add or document streamlined workflow for importing Datasette csv / json exports | henry501 19328961 | open | 0 | 0 | 2022-11-17T10:54:47Z | 2022-11-17T10:54:47Z | NONE | I'm working on some small front-end enhancements to the laion-aesthetic-datasette project, and I wanted to partially populate a database directly using exports from the existing Datasette instance instead of downloading the parquet files and creating my own multi-GB database. There have been a number of small issues that are certainly related to my relative lack of familiarity with the toolkit, but that are still surprising. For example: a CSV export of the images table (http://laion-aesthetic.datasette.io/laion-aesthetic-6pls.csv?sql=select+rowid%2C+url%2C+text%2C+domain_id%2C+width%2C+height%2C+similarity%2C+punsafe%2C+pwatermark%2C+aesthetic%2C+hash%2C+index_level_0+from+images+order+by+random%28%29+limit+100) has nested single quotes, double quotes, and commas that aren't handled by rows_from_file. Similarly, the json output has to be manually transformed to add the column names and remove extraneous information before sqlite_utils can import it. I was able to work through these issues, but as an enhancement it would be really helpful to create or document a clear workflow that avoids the friction of this data transformation. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/513/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1465194249 | I_kwDOCGYnMM5XVRcJ | 514 | upsert of new row with check constraints fails | cldellow 193185 | closed | 0 | 5 | 2022-11-26T16:12:23Z | 2023-05-08T21:50:52Z | 2023-05-08T21:50:51Z | NONE | (I originally opened this in https://github.com/simonw/datasette-insert/issues/20, but I see that that library depends on sqlite-utils) In the case of a new row, upsert first adds the row, specifying only its pkeys: https://github.com/simonw/sqlite-utils/blob/965ca0d5f5bffe06cc02cd7741344d1ddddf9d56/sqlite_utils/db.py#L2783-L2787 This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/514/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1479914599 | I_kwDOCGYnMM5YNbRn | 516 | Feature request: output number of ignored/replaced rows for insert command | simonw 9599 | open | 0 | 4 | 2022-12-06T18:59:21Z | 2022-12-06T19:08:14Z | OWNER | https://hachyderm.io/@briandorsey/109468185742876820
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/516/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1487757143 | I_kwDOCGYnMM5YrV9X | 517 | Drop support for Python 3.6 | simonw 9599 | closed | 0 | 1 | 2022-12-10T01:23:31Z | 2022-12-10T01:36:36Z | 2022-12-10T01:36:36Z | OWNER | CI has started failing for Python 3.6: https://github.com/simonw/sqlite-utils/actions/runs/3576322798 It's fixable by swiching away from But https://endoflife.date/python says that 3.6 end of life was almost 6 years ago, and end of security support nearly 1 year ago. So I'm OK dropping support entirely - Python 3.6 users will still be able to install version 3.30, just not any releases that come next. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/517/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1487764628 | I_kwDOCGYnMM5YrXyU | 518 | flake8 ValueError: Error code '#' supplied to 'extend-ignore' option... | simonw 9599 | closed | 0 | 0 | 2022-12-10T01:30:24Z | 2022-12-10T01:36:46Z | 2022-12-10T01:36:46Z | OWNER |
https://github.com/simonw/sqlite-utils/actions/runs/3662011265/jobs/6190770361 I think from this: https://github.com/simonw/sqlite-utils/blob/e660635cea6c32f4022818380b1e1ee88e7c93a6/setup.cfg#L1-L3 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/518/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1516644980 | I_kwDOCGYnMM5aZip0 | 520 | rows_from_file() raises confusing error if file-like object is not in binary mode | simonw 9599 | closed | 0 | 3 | 2023-01-02T19:00:14Z | 2023-05-08T22:08:07Z | 2023-05-08T22:08:07Z | OWNER | I got this error:
@cli.command()
@click.argument(
"db_path",
type=click.Path(file_okay=True, dir_okay=False, allow_dash=False),
)
@click.option(
"-i",
"--input",
type=click.File("r"),
default="-",
)
def embeddings(db_path, input):
"Store embeddings for one or more text documents"
click.echo("Here is some output")
db = sqlite_utils.Database(db_path)
rows, _ = rows_from_file(input)
print(list(rows))
This should either be called out in the documentation or |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/520/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1550536442 | I_kwDOCGYnMM5ca076 | 521 | Custom JSON encoder | janrito 31504 | open | 0 | 0 | 2023-01-20T09:19:40Z | 2023-01-20T09:19:40Z | NONE | It would be nice if we could specify a custom encoder (and decoder) for types that will need extra deserialisation – e.g., sets, enums or sparse matrices – or even project-specific types |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/521/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1553425465 | I_kwDOCGYnMM5cl2Q5 | 522 | Add COLUMN_TYPE_MAPPING for timedelta | maport 81377 | closed | 0 | 0 | 2023-01-23T16:49:54Z | 2023-11-04T00:49:51Z | 2023-11-04T00:49:51Z | NONE | Currently trying to create a column with Python type ```
The reason this would be useful is that ```
So currently any attempt to convert a MySQL DB with a I was rather surprised that |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/522/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1575131737 | I_kwDOCGYnMM5d4ppZ | 525 | Repeated calls to `Table.convert()` fail | mcarpenter 167893 | closed | 0 | 4 | 2023-02-07T22:40:47Z | 2023-05-08T21:59:41Z | 2023-05-08T21:54:02Z | CONTRIBUTOR | SummaryWhen using the API, repeated calls to Example```python from sqlite_utils import Database db = Database(memory=True) table = db['table'] col = 'x' table.insert_all([{col: 1}]) print(table.get(1)) table.convert(col, lambda x: x*2) print(table.get(1)) def zeroize(x): return 0 zeroize = lambda x: 0zeroize.name = 'zeroize'table.convert(col, zeroize) print(table.get(1)) ``` Output:
ExplanationThis is some relevant documentation.
There's a mismatch between the comments and the code: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L404 but actually the existing function is returned/used instead (as the "registering custom sql functions" doc I linked above says too). Seems like this can be rectified to match the comment? Suggested fixI think there are four things:
1. The call to See also
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/525/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1560651350 | I_kwDOCGYnMM5dBaZW | 523 | Feature request: trim all leading and trailing white space for all columns for all tables in a database | fgregg 536941 | open | 0 | 1 | 2023-01-28T02:40:10Z | 2023-01-28T02:41:14Z | CONTRIBUTOR | It's pretty common that i need to trim leading or trailing white space from lots of columns in a database a part of an initial ETL. I use the following recipe a lot, and it would be great to include this functionality into sqlite-utils
then something like:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/523/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1572766460 | I_kwDOCGYnMM5dvoL8 | 524 | Transformation type `--type DATETIME` | 4l1fe 21095447 | closed | 0 | 15 | 2023-02-06T15:18:42Z | 2023-02-15T12:10:54Z | 2023-02-15T12:10:54Z | NONE | Hey. Currently i do transformation with the type Is it possible to alter a column type to |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/524/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1578790070 | I_kwDOCGYnMM5eGmy2 | 527 | `Table.convert()` skips falsey values | mcarpenter 167893 | closed | 0 | 5 | 2023-02-10T00:00:52Z | 2023-05-09T21:15:05Z | 2023-05-08T21:03:24Z | CONTRIBUTOR | SummaryBy design, ExampleIncrement a column of integers by one ``` python from sqlite_utils import Database db = Database(memory=True) table = db['table'] col = 'x' table.insert_all([{col: 0}, {col:1}]) print(table.get(1)) # 0 print(table.get(2)) # 1 print() table.convert(col, lambda x: x+1) print(table.get(1)) # got 0, expected 1 ⚠⚠⚠ print(table.get(2)) # got 2, expected 2 ``` Another example might be, say, transforming cells containing empty string to DiscussionThis was, I think, a pragmatic choice so that consumers can skip writing guard clauses for these falsey values (particularly from the CLI). But this surprising undocumented behavior can lead to incorrect data. I don't think this is a good trade-off between convenience and correctness. In the absence of this convenience users will either have to write guard clauses into their conversion expressions (or adapt the called function to do the same), so:
Such a change will certainly inconvenience some existing consumers; there will be some breakage. But I think this is worth it to avoid quietly not converting some values by default, which can lead to quietly bad data. I have a PR that I will attach, please take a look and see what you think. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/527/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1581090327 | I_kwDOCGYnMM5ePYYX | 529 | Microsoft line endings | chapmanjacobd 7908073 | closed | 0 | 1 | 2023-02-12T02:20:48Z | 2023-06-14T23:12:12Z | 2023-06-14T23:11:47Z | CONTRIBUTOR | sqlite-utils prints It has tripped me up a few times when piping the output of sqlite-utils to other programs:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/529/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1595340692 | I_kwDOCGYnMM5fFveU | 530 | add ability to configure "on delete" and "on update" attributes of foreign keys: | fgregg 536941 | open | 0 | 2 | 2023-02-22T15:44:14Z | 2023-05-08T20:39:01Z | CONTRIBUTOR | sqlite supports these, and it would be quite nice to be able to add them with sqlite-utils. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/530/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1620254998 | I_kwDOCGYnMM5gkyEW | 532 | Show more information when JSON can't be imported with sqlite-utils insert | voltagex 83080728 | closed | 0 | 2 | 2023-03-12T06:41:44Z | 2023-05-08T20:32:16Z | 2023-05-08T20:32:02Z | NONE | I am currently trying to import the JSON export of my data from Discord, specifically
Please show more information as to why this is invalid, if possible. I am using version 3.30 with Python 3.10 on Windows 11. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/532/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1620516340 | I_kwDOCGYnMM5glx30 | 533 | ReadTheDocs error: not all arguments converted during string formatting | simonw 9599 | closed | 0 | 2 | 2023-03-12T21:21:05Z | 2023-03-12T21:25:33Z | 2023-03-12T21:25:33Z | OWNER | This came up as a failure running tests for: - #531 Traceback on https://readthedocs.org/projects/sqlite-utils/builds/19749348/ ``` File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 889, in interpreted nodes, messages2 = role_fn(role, rawsource, text, lineno, self) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting Exception occurred: File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/531/lib/python3.8/site-packages/sphinx/ext/extlinks.py", line 103, in role title = caption % part TypeError: not all arguments converted during string formatting ``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/533/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1622640374 | I_kwDOCGYnMM5gt4b2 | 534 | ResourceWarning: unclosed file | djhenderson 1244826 | closed | 0 | 1 | 2023-03-14T03:02:18Z | 2023-05-08T19:56:29Z | 2023-05-08T19:56:29Z | NONE | Issuing either
exhibits a ResourceWarning indicating that the CSV file being loaded is not closed. sqlite-utils --version sqlite-utils, version 3.30 py --version Python 3.11.2 Windows Version 10.0.19045 Build 19045 SQLite version 3.41.0 2023-02-21 18:09:37 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/534/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1655860104 | I_kwDOCGYnMM5ismuI | 535 | rows: --transpose or psql extended view-like functionality | chapmanjacobd 7908073 | closed | 0 | 2 | 2023-04-05T15:37:33Z | 2023-06-15T08:39:49Z | 2023-06-14T22:05:28Z | CONTRIBUTOR | It would be nice if the rows subcommand had a flag, perhaps called In other words instead of this:
| track_id | title | song_id | release | artist_id | artist_mbid | artist_name | duration | artist_familiarity | artist_hotttnesss | year | track_7digitalid | shs_perf | shs_work | |--------------------|-------------------|--------------------|--------------------------------------|--------------------|--------------------------------------|------------------|------------|----------------------|---------------------|--------|--------------------|------------|------------| | TRMMMYQ128F932D901 | Silent Night | SOQMMHC12AB0180CB8 | Monster Ballads X-Mas | ARYZTJS1187B98C555 | 357ff05d-848a-44cf-b608-cb34b5701ae5 | Faster Pussy cat | 252.055 | 0.649822 | 0.394032 | 2003 | 7032331 | -1 | 0 | | TRMMMKD128F425225D | Tanssi vaan | SOVFVAK12A8C1350D9 | Karkuteillä | ARMVN3U1187FB3A1EB | 8d7ef530-a6fd-4f8f-b2e2-74aec765e0f9 | Karkkiautomaatti | 156.551 | 0.439604 | 0.356992 | 1995 | 1514808 | -1 | 0 | | TRMMMRX128F93187D9 | No One Could Ever | SOGTUKN12AB017F4F1 | Butter | ARGEKB01187FB50750 | 3d403d44-36ce-465c-ad43-ae877e65adc4 | Hudson Mohawke | 138.971 | 0.643681 | 0.437504 | 2006 | 6945353 | -1 | 0 | | TRMMMCH128F425532C | Si Vos Querés | SOBNYVR12A8C13558C | De Culo | ARNWYLR1187B9B2F9C | 12be7648-7094-495f-90e6-df4189d68615 | Yerba Brava | 145.058 | 0.448501 | 0.372349 | 2003 | 2168257 | -1 | 0 | | TRMMMWA128F426B589 | Tangle Of Aspens | SOHSBXH12A8C13B0DF | Rene Ablaze Presents Winter Sessions | AREQDTE1269FB37231 | | Der Mystic | 514.298 | 0 | 0 | 0 | 2264873 | -1 | 0 | The output would look something like this:
| track_id | |--------------------| | TRYYYVU12903CD01E3 | | TRYYYDJ128F9310A21 | | TRYYYMG128F4260ECA | | TRYYYJO128F426DA37 | | TRYYYUS12903CD2DF0 | | title | |-------------------------------------| | Fernweh feat. Sektion Kuchikäschtli | | Faraday | | Novemba | | Jago Chhadeo | | O Samba Da Vida | | song_id | |--------------------| | SOWXJXQ12AB0189F43 | | SOLXGOR12A81C21EB7 | | SOHODZI12A8C137BB3 | | SOXQYIQ12A8C137FBB | | SOTXAME12AB018F136 | | release | |---------------------------------| | So Oder So | | The Trance Collection Vol. 2 | | Dub_Connected: electronic music | | Naale Baba Lassi Pee Gya | | Pacha V.I.P. | | artist_id | |--------------------| | AR7PLM21187B990D08 | | ARCMCOK1187B9B1073 | | ARZ3R6M1187B9AF750 | | ART5FZD1187B9A7FCF | | AR7Z4J81187FB3FC59 | | artist_mbid | |--------------------------------------| | 3af2b07e-c91c-4160-9bda-f0b9e3144ed3 | | 4ac5f3de-c5ad-475e-ad50-41f1ef9dba20 | | 8b97e9c8-61f5-4615-9a96-276f24204e34 | | 2357c400-9109-42b6-b3fe-9e2d9f8e3872 | | 9d50cb20-7e42-45cc-b0dd-154c3e92a577 | | artist_name | |----------------| | Texta | | Elude | | Gabriel Le Mar | | Kuldeep Manak | | Kiko Navarro | | duration | |------------| | 295.079 | | 484.519 | | 553.038 | | 244.166 | | 217.443 | | artist_familiarity | |----------------------| | 0.552977 | | 0.403668 | | 0.556918 | | 0.4015 | | 0.528617 | | artist_hotttnesss | |---------------------| | 0.454869 | | 0.256935 | | 0.336914 | | 0.374866 | | 0.411595 | | year | |--------| | 2004 | | 0 | | 0 | | 0 | | 0 | | track_7digitalid | |--------------------| | 8486723 | | 5472456 | | 2219291 | | 1632096 | | 7522478 | | shs_perf | |------------| | -1 | | -1 | | -1 | | -1 | | -1 | | shs_work | |------------| | 0 | | 0 | | 0 | | 0 | | 0 | |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/535/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1695428235 | I_kwDOCGYnMM5lDi6L | 538 | `table.upsert_all` fails to write rows when `not_null` is present | xavdid 1231935 | closed | 0 | 9 | 2023-05-04T07:30:38Z | 2023-05-08T20:06:35Z | 2023-05-08T19:27:02Z | NONE | I found an odd bug today, where calls to Repro Example```py from sqlite_utils import Database db = Database("upsert-test.db") db["comments"].upsert_all( [{"id": 1, "name": "david"}], pk="id", not_null=["name"], ) assert list(db["comments"].rows) # err! ``` The schema is correctly created:
But no rows are created. Removing either the Version Info
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1699174055 | I_kwDOCGYnMM5lR1an | 539 | `--raw-lines` option, like `--raw` for multiple lines | simonw 9599 | closed | 0 | 4 | 2023-05-07T18:07:46Z | 2023-05-07T18:43:24Z | 2023-05-07T18:26:18Z | OWNER | I wanted to output newline-separated output of the first column of every row in the results - like |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/539/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1699184583 | I_kwDOCGYnMM5lR3_H | 540 | sphinx.builders.linkcheck build error | simonw 9599 | closed | 0 | 4 | 2023-05-07T18:37:09Z | 2023-05-08T04:56:13Z | 2023-05-07T18:42:36Z | OWNER | https://readthedocs.org/projects/sqlite-utils/builds/20512693/ ``` Running Sphinx v6.2.1 Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py", line 442, in load_extension mod = import_module(extname) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1014, in _gcd_import File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 783, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/builders/linkcheck.py", line 20, in <module> from requests import Response File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/requests/init.py", line 43, in <module> import urllib3 File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/urllib3/init.py", line 38, in <module> raise ImportError( ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main app = Sphinx(args.sourcedir, args.confdir, args.outputdir, File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py", line 225, in init self.setup_extension(extension) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py", line 404, in setup_extension self.registry.load_extension(self, extname) File "/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py", line 445, in load_extension raise ExtensionError(__('Could not import extension %s') % extname, sphinx.errors.ExtensionError: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168) Extension error: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168) ``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/540/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1701018909 | I_kwDOCGYnMM5lY30d | 543 | Tests broken on Windows due to new convert() lambda names | simonw 9599 | closed | 0 | 0 | 2023-05-08T22:11:29Z | 2023-05-08T22:19:04Z | 2023-05-08T22:19:04Z | OWNER | https://github.com/simonw/sqlite-utils/actions/runs/4920084038/jobs/8788501314
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/543/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1700840265 | I_kwDOCGYnMM5lYMNJ | 541 | Get tests to pass with `pytest -Werror` | simonw 9599 | open | 0 | 1 | 2023-05-08T19:57:23Z | 2023-05-08T19:59:35Z | OWNER | Inspired by: - #534 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/541/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1700936245 | I_kwDOCGYnMM5lYjo1 | 542 | Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0 | simonw 9599 | open | 0 | 4.0 backwards incomatible changes 9374594 | 1 | 2023-05-08T21:04:28Z | 2023-05-08T21:07:41Z | OWNER | Following: - #527 The only reason I didn't remove fix this mis-feature entirely is that it represents a backwards incompatible change. I'll make that change in 4.0. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/542/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
|||||||
1718607907 | I_kwDOCGYnMM5mb-Aj | 551 | Make as many examples in the CLI docs as possible copy-and-pastable | simonw 9599 | closed | 0 | 6 | 2023-05-21T19:04:10Z | 2023-05-21T21:04:04Z | 2023-05-21T20:57:24Z | OWNER | e.g. in this section: https://sqlite-utils.datasette.io/en/stable/cli.html#running-queries-directly-against-csv-or-json The little copy button will also copy the |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/551/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1718572201 | I_kwDOCGYnMM5mb1Sp | 547 | No need to show common values if everything is null | simonw 9599 | closed | 0 | 1 | 2023-05-21T17:05:07Z | 2023-05-21T17:19:21Z | 2023-05-21T17:19:21Z | OWNER | Noticed this: ``` % sqlite-utils analyze-tables content.db repos -c delete_branch_on_merge --common-limit 20 --no-least repos.delete_branch_on_merge: (1/1) Total rows: 158 Null rows: 158 Blank rows: 0 Distinct values: 0 Most common: 158: None ``` The |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/547/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1718576761 | I_kwDOCGYnMM5mb2Z5 | 548 | analyze-tables should validate provide --column names | simonw 9599 | closed | 0 | 1 | 2023-05-21T17:20:24Z | 2023-05-21T17:35:52Z | 2023-05-21T17:35:52Z | OWNER | Noticed this while testing: - #547 If you pass a non-existent column to |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/548/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1718595700 | I_kwDOCGYnMM5mb7B0 | 550 | AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7 | simonw 9599 | closed | 0 | 3 | 2023-05-21T18:24:39Z | 2023-05-21T18:42:25Z | 2023-05-21T18:41:58Z | OWNER | https://github.com/simonw/sqlite-utils/actions/runs/5039064797/jobs/9036965488
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/550/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1718612569 | I_kwDOCGYnMM5mb_JZ | 552 | Document how to setup shell auto-completion | simonw 9599 | closed | 0 | 1 | 2023-05-21T19:20:41Z | 2023-05-21T21:05:16Z | 2023-05-21T21:03:40Z | OWNER | https://click.palletsprojects.com/en/8.1.x/shell-completion/ This works for
This will probably work for
Need to add this to the installation docs here: https://sqlite-utils.datasette.io/en/stable/installation.html - along with the pattern for adding that to |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/552/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1718515590 | I_kwDOCGYnMM5mbneG | 544 | New options for analyze-tables --common-limit --no-most and --no-least | simonw 9599 | closed | 0 | 2 | 2023-05-21T14:03:19Z | 2023-05-21T17:03:06Z | 2023-05-21T16:19:31Z | OWNER | The "least common" section is frequently uninteresting, especially for huge tables with a large number of repeated-once values.
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/544/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1718517882 | I_kwDOCGYnMM5mboB6 | 545 | Try out Trogon for a tui interface | simonw 9599 | closed | 0 | 6 | 2023-05-21T14:08:25Z | 2023-05-21T19:33:13Z | 2023-05-21T18:41:58Z | OWNER | sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/545/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
1720096994 | I_kwDOCGYnMM5mhpji | 554 | `IndexError` when doing `.insert(..., pk='id')` after `insert_all` | xavdid 1231935 | open | 0 | 1 | 2023-05-22T17:13:02Z | 2023-05-22T17:18:33Z | NONE | I believe this is related to https://github.com/simonw/sqlite-utils/issues/98. When ```py from sqlite_utils import Database def test_pk_for_insert(fresh_db): user = {"id": "abc", "name": "david"}
if name == "main": db = Database("bug.db") if db["users"].exists(): raise ValueError( "bug only shows on a new database - remove bug.db before running the script" ) test_pk_for_insert(db) ``` The error is:
The issue is in this block: relevant locals are:
What's most interesting is the comment |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/554/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1733198948 | I_kwDOCGYnMM5nToRk | 555 | Filter table by a large bunch of ids | redraw 10843208 | open | 0 | 1 | 2023-05-31T00:29:51Z | 2023-06-14T22:01:57Z | NONE | Hi! this might be a question related to both SQLite & sqlite-utils, and you might be more experienced with them. I have a large bunch of ids, and I'm wondering which is the best way to query them in terms of performance, and simplicity if possible. The naive approach would be something like Another approach might be creating a temp table, or in-memory db table, insert all ids in that table and then join with the target one. I failed to attach an in-memory db both using sqlite-utils, and plain sql's execute(), so my closest approach is something like,
That kinda worked, I couldn't find an option in sqlite-utils's |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/555/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1740026046 | I_kwDOCGYnMM5ntrC- | 556 | Support storing incrementally piped values | mcint 601708 | open | 0 | 1 | 2023-06-04T00:45:23Z | 2023-06-04T01:21:15Z | CONTRIBUTOR | I'm trying to use sqlite-utils to data generated incrementally. There are a few aspects of this that I don't currently know how to handle. I would like an option to apply writes incrementally, line-by-line as they are received. I would like an option to echo incremental progress. And, it would be nice to have In particular, I'm using CoreLocationCLI -w -j to generate, newline-delimited JSON. One variant of the command
It looks like I can get what I want with:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/556/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1740150327 | I_kwDOCGYnMM5nuJY3 | 557 | Aliased ROWID option for tables created from alter=True commands | chapmanjacobd 7908073 | closed | 0 | 2 | 2023-06-04T05:29:28Z | 2023-06-14T06:09:21Z | 2023-06-05T19:26:26Z | CONTRIBUTOR |
ROWID should never be used with foreign keys but the simple act of aliasing rowid to id (which is what happens when one does It would be convenient if there were more options to use a string column (eg. filepath) as the PK, and be able to use it during upserts, but when creating a foreign key, to create an integer column which aliases rowid I made an attempt to switch to integer primary keys here but it is not going well... In my usecase the path column is a business key. Yes, it should be as simple as including the https://github.com/chapmanjacobd/library/actions/runs/5173602136/jobs/9319024777 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/557/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1754174496 | I_kwDOCGYnMM5ojpQg | 558 | Ability to define unique columns when creating a table | aguinane 1910303 | open | 0 | 0 | 2023-06-13T06:56:19Z | 2023-08-18T01:06:03Z | NONE | When creating a new table, it would be good to have an option to set unique columns similar to how not_null is set. ```python from sqlite_utils import Database columns = {"mRID": str, "name": str} db = Database("example.db") db["ExampleTable"].create(columns, pk="mRID", not_null=["mRID"], if_not_exists=True) db["ExampleTable"].create_index(["mRID"], unique=True, if_not_exists=True) ``` So something like this would add the UNIQUE flag to the table definition.
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/558/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1777548699 | I_kwDOCGYnMM5p8z2b | 561 | `--stop-after` option for `insert` and `upsert` commands | simonw 9599 | closed | 0 | 1 | 2023-06-27T18:44:15Z | 2023-06-27T18:50:09Z | 2023-06-27T18:50:08Z | OWNER | I found myself wanting to insert rows from a 849MB CSV file without processing the whole thing: https://huggingface.co/datasets/jerpint-org/HackAPrompt-Playground-Submissions/tree/main |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/561/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1773450152 | I_kwDOCGYnMM5ptLOo | 559 | sqlean support | simonw 9599 | closed | 0 | 0 | 2023-06-25T19:27:26Z | 2023-06-25T23:25:53Z | 2023-06-25T23:25:53Z | OWNER | If sqlean is available, use that. Refs: - https://github.com/nalgeon/sqlean.py/issues/1#issuecomment-1605707788 This will provide a good workaround for: - #235 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/559/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1784794489 | I_kwDOCGYnMM5qYc15 | 562 | Explore the intersection between sqlite-utils and dataclasses | simonw 9599 | open | 0 | 1 | 2023-07-02T19:23:08Z | 2023-07-02T19:26:39Z | OWNER |
Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1616742529 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/562/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issues] ( [id] INTEGER PRIMARY KEY, [node_id] TEXT, [number] INTEGER, [title] TEXT, [user] INTEGER REFERENCES [users]([id]), [state] TEXT, [locked] INTEGER, [assignee] INTEGER REFERENCES [users]([id]), [milestone] INTEGER REFERENCES [milestones]([id]), [comments] INTEGER, [created_at] TEXT, [updated_at] TEXT, [closed_at] TEXT, [author_association] TEXT, [pull_request] TEXT, [body] TEXT, [repo] INTEGER REFERENCES [repos]([id]), [type] TEXT , [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT); CREATE INDEX [idx_issues_repo] ON [issues] ([repo]); CREATE INDEX [idx_issues_milestone] ON [issues] ([milestone]); CREATE INDEX [idx_issues_assignee] ON [issues] ([assignee]); CREATE INDEX [idx_issues_user] ON [issues] ([user]);