issues
604 rows where repo = 140912432 sorted by number descending
This data as json, CSV (advanced)
id | node_id | number ▲ | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | pull_request | body | repo | type | active_lock_reason | performed_via_github_app | reactions | draft | state_reason |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1429029604 | I_kwDOCGYnMM5VLULk | 506 | Make `cursor.rowcount` accessible (wontfix) | simonw 9599 | closed | 0 | 3 | 2022-10-30T21:51:55Z | 2022-11-01T17:37:47Z | 2022-11-01T17:37:13Z | OWNER | In building this Datasette feature on top of This isn't currently exposed by |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/506/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423182778 | I_kwDOCGYnMM5U1Au6 | 505 | Release sqlite-utils 3.30 | simonw 9599 | closed | 0 | 2 | 2022-10-25T22:20:05Z | 2022-10-25T22:41:26Z | 2022-10-25T22:41:16Z | OWNER | sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/505/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
1423069384 | I_kwDOCGYnMM5U0lDI | 504 | db.close() method, calling db.conn.close() | simonw 9599 | closed | 0 | 1 | 2022-10-25T20:50:50Z | 2022-10-25T21:00:29Z | 2022-10-25T20:57:47Z | OWNER | I ended up needing to use I think |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/504/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1423000702 | I_kwDOCGYnMM5U0UR- | 503 | test_recreate failing on Windows Python 3.11 | simonw 9599 | closed | 0 | 10 | 2022-10-25T20:01:41Z | 2022-10-25T20:47:34Z | 2022-10-25T20:45:43Z | OWNER | https://github.com/simonw/sqlite-utils/actions/runs/3323672128/jobs/5494726927 Related: - #502
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/503/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1422954582 | I_kwDOCGYnMM5U0JBW | 502 | Fix tests for Python 3.11 | simonw 9599 | closed | 0 | 1 | 2022-10-25T19:20:31Z | 2022-10-25T19:23:47Z | 2022-10-25T19:23:47Z | OWNER | The way errors are represented has changed: https://github.com/simonw/sqlite-utils/actions/runs/3323588047/jobs/5494127154 ``` ____ testquery_invalid_function ______ db_path = '/tmp/pytest-of-runner/pytest-0/test_query_invalid_function0/test.db'
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/502/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1413641049 | I_kwDOCGYnMM5UQnNZ | 501 | Tests failing due to updated tabulate library | simonw 9599 | closed | 0 | 4 | 2022-10-18T18:07:52Z | 2022-10-18T18:23:40Z | 2022-10-18T18:23:40Z | OWNER | Failure here: https://github.com/simonw/sqlite-utils/actions/runs/3275786702/jobs/5391063221 I figured out the problem:
+- |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/501/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1413610718 | I_kwDOCGYnMM5UQfze | 500 | Turn --flatten into a documented utility function | simonw 9599 | closed | 0 | 4 | 2022-10-18T17:43:36Z | 2022-10-18T18:02:10Z | 2022-10-18T18:00:40Z | OWNER | The |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/500/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1405196044 | PR_kwDOCGYnMM5AmYzy | 499 | feat: recreate fts triggers after table transform | chapmanjacobd 7908073 | open | 0 | 2 | 2022-10-11T20:35:39Z | 2022-10-26T17:54:51Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/499 | https://github.com/simonw/sqlite-utils/pull/498 :books: Documentation preview :books:: https://sqlite-utils--499.org.readthedocs.build/en/499/ alternatively, |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/499/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
1404013495 | PR_kwDOCGYnMM5AicIh | 498 | fix: enable-fts permanently save triggers | chapmanjacobd 7908073 | closed | 0 | 2 | 2022-10-11T05:10:51Z | 2022-10-15T04:33:08Z | 2022-10-11T06:34:31Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/498 | I was wondering why my all my databases were giving wild search results. Turns out create_trigger was not sticking! Running :books: Documentation preview :books:: https://sqlite-utils--498.org.readthedocs.build/en/498/ |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/498/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1393212964 | I_kwDOCGYnMM5TCr4k | 497 | column_names | chapmanjacobd 7908073 | closed | 0 | 1 | 2022-10-01T03:34:21Z | 2022-10-25T21:09:28Z | 2022-10-25T21:09:28Z | CONTRIBUTOR | It would be nice to have a Or if you could get one or all of the following syntax to work for both Database and Table that might be even better: Style 1
- Style 2
- maybe the table ones actually work but I'm too lazy to check. I just know that I have to do:
Edit: This is possible with |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/497/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1393202060 | I_kwDOCGYnMM5TCpOM | 496 | devrel/python api: Pylance type hinting | chapmanjacobd 7908073 | open | 0 | 4 | 2022-10-01T03:03:34Z | 2023-05-03T05:53:27Z | CONTRIBUTOR | Pylance is generally pretty good at figuring out stuff but For example:
I think this is because DEFAULT is an empty class? maybe a few small changes could be made to make the library more type-friendly The interim solution is of course to turn off type hints completely for the line
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/496/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1392690202 | I_kwDOCGYnMM5TAsQa | 495 | Support JSON values returned from .convert() functions | mhalle 649467 | closed | 0 | 3 | 2022-09-30T16:33:49Z | 2022-10-25T21:23:37Z | 2022-10-25T21:23:28Z | NONE | When using the convert function on a JSON column, the result of the conversion function must be a string. If the return value is either a dict (object) or a list (array), the convert call will error out with an unhelpful user defined function exception. It makes sense that since the original column value was a string and required conversion to data structures, the result should be converted back into a JSON string as well. However, other functions auto-convert to JSON string representation, so the fact that convert doesn't could be surprising. At least the documentation should note this requirement, because the sqlite error messages won't readily reveal the issue. Jf only sqlite's JSON column type meant something :) |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/495/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1386593843 | I_kwDOCGYnMM5Spb4z | 494 | Document how to use Just | simonw 9599 | closed | 0 | 2 | 2022-09-26T19:25:12Z | 2022-09-26T19:32:36Z | 2022-09-26T19:26:39Z | OWNER | I'm using https://github.com/simonw/sqlite-utils/blob/afbd2b2cba45cccb305c3d4638d18db4dd3d4bbd/Justfile#L1-L24 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/494/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1386562662 | I_kwDOCGYnMM5SpURm | 493 | Tiny typographical error in install/uninstall docs | simonw 9599 | open | 0 | 3 | 2022-09-26T19:00:42Z | 2022-10-25T21:31:15Z | OWNER | Added in: - #483 I don't know how to fix this in Sphinx: I'm getting this: https://sqlite-utils.datasette.io/en/latest/cli.html#cli-install
But I want it to display Here's the code: https://github.com/simonw/sqlite-utils/blob/85247038f70d7eb2f3e272cfeaa4c44459cafba8/docs/cli.rst#L2125 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/493/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1386530156 | I_kwDOCGYnMM5SpMVs | 492 | Idea: ability to pass extra variables to `--convert` scripts | simonw 9599 | open | 0 | 1 | 2022-09-26T18:30:45Z | 2022-09-26T18:33:19Z | OWNER | Got this idea from this example in https://jeqo.github.io/notes/2022-09-24-ingest-logs-sqlite/
This made me think: it might be neat if you could inject additional variable values into that script with extra command-line options, to make this kind of reuse easier. Something like this:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/492/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1383646615 | I_kwDOCGYnMM5SeMWX | 491 | Ability to merge databases and tables | sgraaf 8904453 | open | 0 | 7 | 2022-09-23T11:10:55Z | 2023-06-14T22:14:24Z | NONE | Hi! Let me firstly say that I am a big fan of your work -- I follow your tweets and blog posts with great interest 😄. Now onto the matter at hand: I think it would be great if This could look something like this:
I imagine this is rather straightforward if all databases involved in the merge contain differently named tables (i.e. no chance of conflicts), but things get slightly more complicated if two or more of the databases to be merged contain tables with the same name. Not only do you have to "do something" with the primary key(s), but these tables could also simply have different schemas (and therefore be incompatible for concatenation to begin with). Anyhow, I would love your thoughts on this, and, if you are open to it, work together on the design and implementation! |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/491/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1382457780 | I_kwDOCGYnMM5SZqG0 | 490 | Ability to insert multi-line files | jeqo 6180701 | closed | 0 | 4 | 2022-09-22T13:29:22Z | 2022-09-26T18:24:44Z | 2022-09-23T16:37:58Z | NONE | I was looking into how to parse application log files that contain multiline text (e.g. Java stack traces) into sqlite.
I can see that at the moment I wonder if this functionality would be useful for sqlite-utils. A similar approach to Elastic logstash/filebeat can be adopted: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html Potential changes:
Or if this is achievable in a different way, please share. Thanks! |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/490/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1374939463 | I_kwDOCGYnMM5R8-lH | 489 | Ability to load JSON records held in a file with a single top level key that is a list of objects | simonw 9599 | open | 0 | 9 | 2022-09-15T18:46:03Z | 2022-09-15T20:56:10Z | OWNER | It's very common for JSON to look like this:
Right now you can't import this into But since this is so common, it would be neat if |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/489/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1373224657 | I_kwDOCGYnMM5R2b7R | 488 | `sqlite-utils transform` should set empty strings to null when converting text columns to integer/float | simonw 9599 | open | 0 | 5 | 2022-09-14T15:51:30Z | 2022-12-23T17:38:55Z | OWNER |
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/488/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1367835380 | I_kwDOCGYnMM5Rh4L0 | 487 | Specify foreign key against compound key in other table | ryanfox 540968 | closed | 0 | 2 | 2022-09-09T13:32:09Z | 2022-09-11T04:00:44Z | 2022-09-11T04:00:44Z | NONE | When inserting rows via the library, is it possible to specify a foreign key to a compound primary key? For example, suppose I create a table:
And I want to add related data in another table:
Is it possible to specify a value for SQLite does support it: https://www.sqlite.org/foreignkeys.html#fk_composite |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/487/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1366512990 | PR_kwDOCGYnMM4-nBs9 | 486 | progressbar for inserts/upserts of all fileformats, closes #485 | MischaU8 99098079 | closed | 0 | 7 | 2022-09-08T14:58:02Z | 2022-09-15T20:40:03Z | 2022-09-15T20:37:51Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/486 | :books: Documentation preview :books:: https://sqlite-utils--486.org.readthedocs.build/en/486/ |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/486/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1366423176 | I_kwDOCGYnMM5RcfaI | 485 | Progressbar not shown when inserting/upserting jsonlines file | MischaU8 99098079 | closed | 0 | 1 | 2022-09-08T14:13:18Z | 2022-09-15T20:39:52Z | 2022-09-15T20:37:52Z | CONTRIBUTOR | When inserting or upserting a jsonlines file, no progressbar is shown. Expected behavior is that, just like with .csv/.tsv files, also for a jsonlines file (--nl), unless --silent is provided, a progressbar is shown.
Currently |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/485/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1363766973 | I_kwDOCGYnMM5RSW69 | 484 | Expose convert recipes to `sqlite-utils --functions` | simonw 9599 | open | 0 | 11 | 2022-09-06T20:15:08Z | 2022-09-07T19:09:52Z | OWNER |
It would be useful if the |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/484/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1363765916 | I_kwDOCGYnMM5RSWqc | 483 | `sqlite-utils install` command | simonw 9599 | closed | 0 | 2 | 2022-09-06T20:13:55Z | 2022-09-26T19:04:43Z | 2022-09-26T18:57:15Z | OWNER | With the addition of In addition to the existing This isn't easy if you installed the tool with Datasette solved this problem with the
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/483/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1361355564 | I_kwDOCGYnMM5RJKMs | 482 | balanced table default column_order | chapmanjacobd 7908073 | closed | 0 | 1 | 2022-09-05T03:00:18Z | 2022-10-10T17:43:02Z | 2022-09-06T20:17:27Z | CONTRIBUTOR | Is there any performance or size difference with column order in SQLITE ? similar to this https://www.cybertec-postgresql.com/en/column-order-in-postgresql-does-matter/ It might be interesting to have an option to create with an optimized column order. I'm assuming this would look something like INTEGER columns, REAL columns, BLOB columns, TEXT columns, NULL columns. NULL columns at the end because they are more likely to be TEXT and it is impossible to know if they will become INTEGER (Of course, any schema evolution would reduce optimization but maybe column order could also be re-evaluated when schema changes) edit: this is easy to accomplish with the existing
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/482/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1359604075 | I_kwDOCGYnMM5RCelr | 481 | Idea: `sqlite-utils create-table tablename --sql "select ..."` | simonw 9599 | open | 0 | 0 | 2022-09-02T01:41:24Z | 2022-09-02T01:42:08Z | OWNER | Could offer syntactic sugar for:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/481/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1355433619 | PR_kwDOCGYnMM4-B7Mc | 480 | search_sql add include_rank option | chapmanjacobd 7908073 | closed | 0 | 4 | 2022-08-30T09:10:29Z | 2022-08-31T03:40:35Z | 2022-08-31T03:40:35Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/480 | I haven't tested this yet but wanted to get a heads-up whether this kind of change would be useful or if I should just duplicate the function and tweak it within my code :books: Documentation preview :books:: https://sqlite-utils--480.org.readthedocs.build/en/480/ |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/480/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1355193529 | I_kwDOCGYnMM5Qxpy5 | 479 | OperationalError: cannot VACUUM from within a transaction | chapmanjacobd 7908073 | open | 0 | 0 | 2022-08-30T05:34:24Z | 2022-08-30T05:34:24Z | CONTRIBUTOR | Maybe when calling ``` 46 db["media"].optimize() # type: ignore ---> 47 db.vacuum() File ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:1047, in Database.vacuum(self)
1045 def vacuum(self):
1046 "Run a SQLite File ~/.local/lib/python3.10/site-packages/sqlite_utils/db.py:470, in Database.execute(self, sql, parameters) 468 return self.conn.execute(sql, parameters) 469 else: --> 470 return self.conn.execute(sql) OperationalError: cannot VACUUM from within a transaction ``` It might also be nice to add a sentence or two about how transactions are committed on the docs page. When I was swapping out my sqlite3 code for this library it was nice that everything was pretty much drop-in but I was/am unsure what to do about the places I explicitly call Related to https://github.com/simonw/sqlite-utils/issues/121 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/479/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1353481513 | I_kwDOCGYnMM5QrH0p | 478 | `sqlite-utils tables data.db table1 table2` | simonw 9599 | open | 0 | 1 | 2022-08-28T22:05:53Z | 2022-08-28T22:22:35Z | OWNER | The If you have a huge table in there then running it with Would be useful if it could accept an optional list of tables that it should execute against, as an alternative to the default of all of them. This should be a backwards compatible change. Current design is: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#tables ``` Usage: sqlite-utils tables [OPTIONS] PATH List the tables in the database Example:
``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/478/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1353441389 | I_kwDOCGYnMM5Qq-Bt | 477 | Conda Forge | thewchan 49702524 | closed | 0 | 2 | 2022-08-28T19:03:08Z | 2022-09-07T03:46:55Z | 2022-09-07T03:46:55Z | NONE | Hello! I have successfully put this package on to Conda Forge, and I have extending the invitation for the owner/maintainers of this package to be maintainers on Conda Forge as well. Let me know if you are interested! Thanks. https://github.com/conda-forge/sqlite-utils-feedstock |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/477/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1353196970 | I_kwDOCGYnMM5QqCWq | 476 | Release notes for 3.29 | simonw 9599 | closed | 0 | 3.29 8355157 | 2 | 2022-08-27T23:21:21Z | 2022-08-28T04:07:15Z | 2022-08-28T04:07:03Z | OWNER | sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/476/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1353189941 | I_kwDOCGYnMM5QqAo1 | 475 | table.default_values introspection property | simonw 9599 | closed | 0 | 3.29 8355157 | 1 | 2022-08-27T22:33:31Z | 2022-08-27T22:44:46Z | 2022-08-27T22:43:02Z | OWNER |
Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/468#issuecomment-1229279539 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/475/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1353074021 | I_kwDOCGYnMM5QpkVl | 474 | Add an option for specifying column names when inserting CSV data | hubgit 14294 | open | 0 | 3 | 2022-08-27T15:29:59Z | 2022-08-31T03:42:36Z | NONE | https://sqlite-utils.datasette.io/en/stable/cli.html#csv-files-without-a-header-row
It would be nice to be able to specify the column names when importing CSV/TSV without a header row, via an extra command line option. (renaming a column of a large table can take a long time, which makes it an inconvenient workaround) |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/474/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1352953535 | PR_kwDOCGYnMM4950Az | 473 | Support entrypoints for `--load-extension` | simonw 9599 | closed | 0 | 1 | 2022-08-27T05:53:59Z | 2022-08-27T05:55:52Z | 2022-08-27T05:55:47Z | OWNER | simonw/sqlite-utils/pulls/473 | Refs #470 :books: Documentation preview :books:: https://sqlite-utils--473.org.readthedocs.build/en/473/ |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/473/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1352946135 | I_kwDOCGYnMM5QpFHX | 472 | Reuse the locals/globals fix from --functions for other code accepting options | simonw 9599 | closed | 0 | 3.29 8355157 | 2 | 2022-08-27T05:12:05Z | 2022-08-27T05:20:12Z | 2022-08-27T05:20:12Z | OWNER | I figured out a workaround for the ugly |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/472/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352932716 | I_kwDOCGYnMM5QpB1s | 471 | sqlite-utils query --functions mechanism for registering extra functions | simonw 9599 | closed | 0 | 3.29 8355157 | 12 | 2022-08-27T03:57:53Z | 2022-09-07T03:46:26Z | 2022-08-27T05:10:57Z | OWNER | It would be really cool if you could register additional custom SQL functions for use with the ``` sqlite-utils data.db 'update images set domain = extract_domain(url)' --functions ' from urllib.parse import urlparse def extract_domain(url): return urlparse(url).netloc ' ``` Every function defined in that code block would be registered with the connection, unless the name began with an underscore. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/471/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352932038 | I_kwDOCGYnMM5QpBrG | 470 | Upgrade `--load-extension` to accept entrypoints like Datasette | simonw 9599 | closed | 0 | 3.29 8355157 | 6 | 2022-08-27T03:53:20Z | 2022-08-27T05:55:49Z | 2022-08-27T05:55:48Z | OWNER | Imitate: - https://github.com/simonw/datasette/pull/1789 ``` would load default entrypoint like beforedatasette data.db --load-extension ext loads the extensions with the "sqlite3_foo_init" entrpointdatasette data.db --load-extension ext:sqlite3_foo_init loads the extensions with the "sqlite3_bar_init" entrpointdatasette data.db --load-extension ext:sqlite3_bar_init ``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/470/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1352931464 | I_kwDOCGYnMM5QpBiI | 469 | sqlite-utils rows --order option | simonw 9599 | closed | 0 | 3.29 8355157 | 1 | 2022-08-27T03:49:51Z | 2022-08-27T04:30:49Z | 2022-08-27T04:10:32Z | OWNER | For consistency with
I wanted to run |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/469/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1348294436 | PR_kwDOCGYnMM49qP2V | 468 | db[table].create(..., transform=True) and create-table --transform | simonw 9599 | closed | 0 | 3.29 8355157 | 6 | 2022-08-23T17:27:58Z | 2022-08-27T23:17:55Z | 2022-08-27T23:17:55Z | OWNER | simonw/sqlite-utils/pulls/468 | Work in progress. Still needs documentation and tests (and to cover more cases of things that might have changed). Refs: - #467 :books: Documentation preview :books:: https://sqlite-utils--468.org.readthedocs.build/en/468/ |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/468/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||
1348169997 | I_kwDOCGYnMM5QW3EN | 467 | Mechanism for ensuring a table has all the columns | simonw 9599 | closed | 0 | 3.29 8355157 | 13 | 2022-08-23T15:50:23Z | 2022-08-27T23:19:41Z | 2022-08-27T23:17:56Z | OWNER | Suggested by @jefftriplett on Discord: https://discord.com/channels/823971286308356157/997738192360964156/1011655389063958600 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/467/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1342374388 | PR_kwDOCGYnMM49Wv9T | 466 | Use Read the Docs action v1 (#463) | simonw 9599 | closed | 0 | 0 | 2022-08-17T23:11:50Z | 2022-08-17T23:11:54Z | 2022-08-17T23:11:54Z | OWNER | simonw/sqlite-utils/pulls/466 | Read the Docs repository was renamed from |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/466/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1342357149 | PR_kwDOCGYnMM49Wsnq | 465 | beanbag-docutils>=2.0 | simonw 9599 | closed | 0 | 2 | 2022-08-17T22:41:39Z | 2022-08-17T23:38:07Z | 2022-08-17T23:38:02Z | OWNER | simonw/sqlite-utils/pulls/465 | Refs #464 |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/465/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1338001039 | I_kwDOCGYnMM5PwEaP | 464 | Link from documentation to source code | simonw 9599 | closed | 0 | 5 | 2022-08-13T16:19:57Z | 2022-08-17T23:38:03Z | 2022-08-17T23:38:03Z | OWNER | Twitter conversation asking for ways to automate this here: https://twitter.com/simonw/status/1558260492015046656 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/464/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1334416486 | PR_kwDOCGYnMM488n6D | 463 | Use Read the Docs action v1 | humitos 244656 | closed | 0 | 1 | 2022-08-10T10:31:47Z | 2022-08-18T08:30:14Z | 2022-08-17T23:11:16Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/463 | Read the Docs repository was renamed from :books: Documentation preview :books:: https://sqlite-utils--463.org.readthedocs.build/en/463/ |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/463/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1326391841 | PR_kwDOCGYnMM48iLGF | 462 | Discord badge | simonw 9599 | closed | 0 | 2 | 2022-08-02T20:56:04Z | 2022-08-02T21:15:57Z | 2022-08-02T21:15:52Z | OWNER | simonw/sqlite-utils/pulls/462 | Also testing fix for: - https://github.com/readthedocs/readthedocs-preview/issues/10 :books: Documentation preview :books:: https://sqlite-utils--462.org.readthedocs.build/en/462/ |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/462/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1326349129 | I_kwDOCGYnMM5PDntJ | 461 | Consider including animated SVG console demos | simonw 9599 | open | 0 | 1 | 2022-08-02T20:10:04Z | 2022-08-02T20:12:14Z | OWNER | I recorded this one using https://github.com/nbedos/termtosvg - with
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/461/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1326087800 | PR_kwDOCGYnMM48hI-_ | 460 | Cross-link CLI to Python docs | simonw 9599 | closed | 0 | 4 | 2022-08-02T16:18:28Z | 2022-08-18T21:58:10Z | 2022-08-18T21:58:07Z | OWNER | simonw/sqlite-utils/pulls/460 | Work in progress, partly to test the ReadTheDocs preview link action. Refs: - #426 :books: Documentation preview :books:: https://readthedocs-preview--460.org.readthedocs.build/en/460/ |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/460/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1324659241 | I_kwDOCGYnMM5O9LIp | 459 | Single quoted transform recipes on Windows do not work as expected | shakeel 19921 | open | 0 | 0 | 2022-08-01T16:14:54Z | 2022-08-01T16:14:54Z | CONTRIBUTOR | Trying to follow the tutorial for sqlite-utils and datasette https://datasette.io/tutorials/clean-data on Windows 11 OS
In the step to transform dates into ISO dates the quoted value ``` sqlite-utils convert manatees.db locations \ REPDATE created_date last_edited_date \ 'r.parsedatetime(value)' --dry-run 1975/01/31 00:00:00+00 --- becomes: r.parsedatetime(value) Would affect 13568 rows ``` However, if I change the code from single quotes to double quotes, it works as expected. ``` sqlite-utils convert manatees.db locations \ REPDATE created_date last_edited_date \ "r.parsedatetime(value)" --dry-run 1975/01/31 00:00:00+00 --- becomes: 1975-01-31T00:00:00+00:00 Would affect 13568 rows ``` Specifying the transform code recipe should work with single quotes on Windows. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/459/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1320243134 | I_kwDOCGYnMM5OsU-- | 458 | Support custom names for registered functions | simonw 9599 | closed | 0 | 3.29 8355157 | 1 | 2022-07-28T00:13:00Z | 2022-08-27T03:56:01Z | 2022-07-28T00:13:57Z | OWNER | In this example: ```python @db.register_function def reverse_string(s): return "".join(reversed(list(s)))
``` There's currently no way to over-ride the automatically selected name for the SQL function. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/458/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1319881016 | PR_kwDOCGYnMM48Mmde | 457 | Link to installation instructions | simonw 9599 | closed | 0 | 3.29 8355157 | 2 | 2022-07-27T17:38:36Z | 2022-08-27T03:55:52Z | 2022-07-27T17:57:50Z | OWNER | simonw/sqlite-utils/pulls/457 | sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/457/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1310243385 | I_kwDOCGYnMM5OGLo5 | 456 | feature request: pivot command | fgregg 536941 | open | 0 | 5 | 2022-07-20T00:58:08Z | 2022-07-20T17:50:50Z | CONTRIBUTOR | pivoting long-format table to wide-format tables is pretty common and kind of pain. would love to see this feature in sqlite-utils! |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/456/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1309542173 | PR_kwDOCGYnMM47pwAb | 455 | in extract code, check equality with IS instead of = for nulls | fgregg 536941 | closed | 0 | 3 | 2022-07-19T13:40:25Z | 2022-08-27T14:45:03Z | 2022-08-27T14:45:03Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/455 | sqlite "IS" is equivalent to SQL "IS NOT DISTINCT FROM" closes #423 |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/455/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1306548397 | I_kwDOCGYnMM5N4Fit | 454 | CLI command for duplicating tables | simonw 9599 | closed | 0 | 1 | 2022-07-15T21:31:27Z | 2022-07-15T21:48:23Z | 2022-07-15T21:45:51Z | OWNER | CLI equivalent of: - #449 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/454/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1303169663 | I_kwDOCGYnMM5NrMp_ | 453 | 'unclosed file' warning when using insert_upsert_implementation from Python | makkus 311257 | closed | 0 | 1 | 2022-07-13T09:34:35Z | 2022-07-15T21:52:25Z | 2022-07-15T21:52:21Z | NONE | I'm using the The warning goes away when wrapping the code from this line in a try/finally block like:
I suspect Python closes the reference automatically when the sqlite-utils cli run is done, but since my code doesn't exit, I'm getting the warning. Alternatively, it'd be cool if the 'import csv/tsv' functionality could be added directly to the Database class. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/453/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1299760627 | PR_kwDOCGYnMM47JUun | 452 | Add duplicate table feature | davidleejy 1690072 | closed | 0 | 1 | 2022-07-09T20:24:31Z | 2022-07-15T21:21:37Z | 2022-07-15T21:21:36Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/452 | This PR addresses a feature request raised in issue #449. Specifically this PR adds a functionality that lets users duplicate a table via:
Test added in file Happy to make changes to meet maintainers' feedback, if any. |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/452/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1298531653 | I_kwDOCGYnMM5NZgVF | 451 | Make sqlite_utils.utils.chunks a documented function | simonw 9599 | closed | 0 | 2 | 2022-07-08T06:01:04Z | 2022-07-15T22:09:34Z | 2022-07-15T21:59:33Z | OWNER | I want to use it in another project: https://github.com/simonw/sqlite-utils/blob/8a9fe6498faf783a1fdeb1793e661ad194a05267/sqlite_utils/utils.py#L471-L474 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/451/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1292060682 | I_kwDOCGYnMM5NA0gK | 450 | Add --ignore option to more commands | simonw 9599 | closed | 0 | 9 | 2022-07-02T13:52:02Z | 2022-07-15T22:39:09Z | 2022-07-15T22:37:45Z | OWNER | As seen in https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key Could make this TIL trick unnecessary: https://til.simonwillison.net/bash/ignore-errors |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/450/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1279863844 | I_kwDOCGYnMM5MSSwk | 449 | Utilities for duplicating tables and creating a table with the results of a query | davidleejy 1690072 | closed | 0 | 4 | 2022-06-22T09:41:43Z | 2022-07-15T21:46:13Z | 2022-07-15T21:21:36Z | CONTRIBUTOR | is there a duplicate table functionality? Otherwise, I'd be happy to submit a PR. In sqlite3 it would look like: ```python import sqlite3 as sl con = sl.connect('prompt-tune.db') def db_duplicate_table(table_name, table_name_new, con=con):
# Duplicates table db_duplicate_table('orig_table', 'new_table') ``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/449/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1279144769 | I_kwDOCGYnMM5MPjNB | 448 | Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto' | mungewell 236907 | closed | 0 | 5 | 2022-06-21T21:48:27Z | 2023-05-08T22:01:00Z | 2023-05-08T22:01:00Z | NONE | Attempting to run the example given here (without extra bracket ;-): https://sqlite-utils.datasette.io/en/stable/python-api.html#reading-rows-from-a-file ``` from sqlite_utils.utils import rows_from_file import io rows, format = rows_from_file(io.StringIO("id,name\n1,Cleo")) print(list(rows), format) Outputs [{'id': '1', 'name': 'Cleo'}] Format.CSV``` Gives error ```
I am running Python on Windows. ```
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/448/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1278571700 | I_kwDOCGYnMM5MNXS0 | 447 | Incorrect syntax highlighting in docs CLI reference | simonw 9599 | closed | 0 | 3 | 2022-06-21T14:53:10Z | 2022-06-21T18:48:47Z | 2022-06-21T18:48:46Z | OWNER | https://sqlite-utils.datasette.io/en/stable/cli-reference.html#insert It looks like Python keywords are being incorrectly highlighted here. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/447/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1277328147 | I_kwDOCGYnMM5MInsT | 446 | Use Just to automate running tests and linters locally | simonw 9599 | closed | 0 | 2 | 2022-06-20T19:51:09Z | 2022-06-21T19:28:35Z | 2022-06-20T19:54:50Z | OWNER | I keep committing code that fails additional tests like |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/446/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1277295119 | I_kwDOCGYnMM5MIfoP | 445 | `sqlite_utils.utils.TypeTracker` should be a documented API | simonw 9599 | closed | 0 | 3 | 2022-06-20T19:08:28Z | 2022-06-20T19:49:02Z | 2022-06-20T19:46:58Z | OWNER | I've used it in a couple of external places now: |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/445/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1271426387 | I_kwDOCGYnMM5LyG1T | 444 | CSV `extras_key=` and `ignore_extras=` equivalents for CLI tool | simonw 9599 | open | 0 | 5 | 2022-06-14T22:22:47Z | 2022-07-07T16:39:18Z | OWNER |
Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1155767915 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/444/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1269998342 | I_kwDOCGYnMM5LsqMG | 443 | Make `utils.rows_from_file()` a documented API | simonw 9599 | closed | 0 | 2 | 2022-06-13T21:53:24Z | 2022-06-20T19:49:37Z | 2022-06-14T20:12:46Z | OWNER |
Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/440#issuecomment-1154385916 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/443/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1269886084 | I_kwDOCGYnMM5LsOyE | 442 | `maximize_csv_field_size_limit()` utility function | simonw 9599 | closed | 0 | 2 | 2022-06-13T19:54:54Z | 2022-06-14T21:55:15Z | 2022-06-14T21:31:49Z | OWNER | This code here runs only if I found myself needing the same fix in another library: It should be a documented utility function. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/442/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1257724585 | I_kwDOCGYnMM5K91qp | 441 | Combining `rows_where()` and `search()` to limit which rows are searched | betatim 1448859 | closed | 0 | 4 | 2022-06-02T06:01:55Z | 2022-06-14T21:57:57Z | 2022-06-14T21:54:38Z | NONE | What is the right way to limit a full text search query to some rows of a table? For example, I have a table that contains the following columns: I tried to combine My two questions:
1. is adding a Right now I am thinking I will make my own version of Bonus question: is this generally useful/something to add to sqlite-utils or too niche? |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/441/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1250629388 | I_kwDOCGYnMM5KixcM | 440 | CSV files with too many values in a row cause errors | frafra 4068 | closed | 0 | 20 | 2022-05-27T10:54:44Z | 2022-06-14T22:23:01Z | 2022-06-14T20:12:46Z | NONE | Original title: csv.DictReader can have None as key In some cases, ```python url="https://artsdatabanken.no/Fab2018/api/export/csv" db = sqlite_utils.Database(":memory") with urlopen(url) as fab:
reader, _ = sqlite_utils.utils.rows_from_file(fab, encoding="utf-16le") Result:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/440/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1250495688 | I_kwDOCGYnMM5KiQzI | 439 | Misleading progress bar against utf-16-le CSV input | frafra 4068 | open | 0 | 12 | 2022-05-27T08:34:49Z | 2022-06-15T03:53:43Z | NONE | The program crashes without any error.
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/439/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1250161887 | I_kwDOCGYnMM5Kg_Tf | 438 | illegal UTF-16 surrogate | frafra 4068 | closed | 0 | 2 | 2022-05-26T22:49:52Z | 2022-05-27T08:21:53Z | 2022-05-27T08:21:53Z | NONE | I am trying to insert ``` sqlite-utils insert --csv --delimiter ";" --encoding="utf-16-le" --pk "Id" csv fremmedart test.db [------------------------------------] 0% Error: 'utf-16-le' codec can't decode bytes in position 98-99: illegal UTF-16 surrogate The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` I tried to convert the file using ``` sqlite-utils insert --csv --delimiter ";" --encoding=utf-8 --pk "Id" csv_utf8 fremmedart test.db [------------------------------------] 0% Error: 'utf-8' codec can't decode byte 0xd9 in position 99: invalid continuation byte The input you provided uses a character encoding other than utf-8. You can fix this by passing the --encoding= option with the encoding of the file. If you do not know the encoding, running 'file filename.csv' may tell you. It's often worth trying: --encoding=latin-1 ``` I have no issues reading such file using this Python code:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/438/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1244294227 | PR_kwDOCGYnMM44P4GG | 437 | docs to dogs | yurivish 114388 | closed | 0 | 1 | 2022-05-22T15:50:33Z | 2022-05-30T21:32:41Z | 2022-05-30T21:32:41Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/437 | Fixes a typo. |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/437/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1243715381 | I_kwDOCGYnMM5KIZc1 | 436 | Add "copy to clipboard" button to code examples in documentation | simonw 9599 | closed | 0 | 0 | 2022-05-20T21:53:23Z | 2022-05-20T21:57:53Z | 2022-05-20T21:57:53Z | OWNER | Follows: - #435 Imitates: - https://github.com/simonw/datasette/issues/1748 I'll use https://github.com/executablebooks/sphinx-copybutton - here's the Datasette commit: https://github.com/simonw/datasette/commit/1465fea4798599eccfe7e8f012bd8d9adfac3039 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/436/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1243704847 | I_kwDOCGYnMM5KIW4P | 435 | Switch to Furo documentation theme | simonw 9599 | closed | 0 | 2 | 2022-05-20T21:46:39Z | 2022-05-20T21:56:10Z | 2022-05-20T21:54:43Z | OWNER | sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/435/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||||
1243151184 | I_kwDOCGYnMM5KGPtQ | 434 | `detect_fts()` identifies the wrong table if tables have names that are subsets of each other | ryascott 559711 | closed | 0 | 3 | 2022-05-20T13:28:31Z | 2022-06-14T23:24:09Z | 2022-06-14T23:24:09Z | NONE | Windows 10 Python 3.9.6 When I was running a full text search through the Python library, I noticed that the query was being run on a different full text search table than the one I was trying to search. I took a look at the following function and noticed:
My database contains tables with similar names and %{table}% was matching another table that ended differently in its name. I have included a sample test that shows this occurring: I search for Marsupials in db["books"] and The Clue of the Broken Blade is returned. This occurs since the search for Marsupials was "successfully" done against db["booksb"] and rowid 1 is returned. "The Clue of the Broken Blade" has a rowid of 1 in db["books"] and this is what is returned from the search. ```python def test_fts_search_with_similar_table_names(fresh_db): db = Database(memory=True) db["books"].insert_all( [ { "title": "The Clue of the Broken Blade", "author": "Franklin W. Dixon", }, { "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", }, ] ) db["booksb"].insert( { "title": "Habits of Australian Marsupials", "author": "Marlee Hawkins", } )
``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/434/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1239034903 | I_kwDOCGYnMM5J2iwX | 433 | CLI eats my cursor | chapmanjacobd 7908073 | closed | 0 | 10 | 2022-05-17T18:52:52Z | 2023-11-04T00:46:30Z | 2023-11-04T00:46:30Z | CONTRIBUTOR | I'm not sure why this happens but I can still type commands after it runs but the text cursor is invisible |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/433/reactions", "total_count": 5, "+1": 5, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1236693079 | I_kwDOCGYnMM5JtnBX | 432 | Support `rows_where()`, `delete_where()` etc for attached alias databases | luxint 11597658 | open | 0 | 5 | 2022-05-16T06:38:58Z | 2022-06-14T22:16:48Z | NONE | Hi, I noticed Besides, |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/432/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1227571375 | I_kwDOCGYnMM5JK0Cv | 431 | Allow making m2m relation of a table to itself | rafguns 738408 | open | 0 | 3 | 2022-05-06T08:30:43Z | 2022-06-23T14:12:51Z | NONE | I am building a database, in which one of the tables has a many-to-many relationship to itself. As far as I can see, this is not (yet) possible using Example: suppose I have a table of people, and I want to store the information that John and Mary have two children, Michael and Suzy. It would be neat if I could do something like this: ```python from sqlite_utils import Database db = Database(memory=True) db["people"].insert({"name": "John"}, pk="name").m2m( "people", [{"name": "Michael"}, {"name": "Suzy"}], m2m_table="parent_child", pk="name" ) db["people"].insert({"name": "Mary"}, pk="name").m2m( "people", [{"name": "Michael"}, {"name": "Suzy"}], m2m_table="parent_child", pk="name" ) ``` But if I do that, the many-to-many table This could be solved by adding one or two keyword_arguments to |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/431/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1224112817 | I_kwDOCGYnMM5I9nqx | 430 | Document how to use `PRAGMA temp_store` to avoid errors when running VACUUM against huge databases | rayvoelker 9308268 | open | 0 | 2 | 2022-05-03T13:33:58Z | 2022-06-14T23:26:37Z | NONE | I'm trying to figure out a way to get the Here's the bit that's causing the error, and the resulting error output: ```python combine these columns into 1 table "bib_properties" :best_titlebib_level_codemat_typematerial_codebest_authordb["circ_trans"].extract( ["best_title", "bib_level_code", "mat_type", "material_code", "best_author"], table="bib_properties", fk_column="bib_properties_id" ) db["circ_trans"].extract( ["call_number"], table="call_number", fk_column="call_number_id", rename={"call_number": "value"} ) ``` ```pythonOperationalError Traceback (most recent call last) Input In [17], in <cell line: 7>() 1 # combine these columns into 1 table "bib_properties" : 2 # best_title 3 # bib_level_code 4 # mat_type 5 # material_code 6 # best_author ----> 7 db["circ_trans"].extract( 8 ["best_title", "bib_level_code", "mat_type", "material_code", "best_author"], 9 table="bib_properties", 10 fk_column="bib_properties_id" 11 ) 13 db["circ_trans"].extract( 14 ["call_number"], 15 table="call_number", 16 fk_column="call_number_id", 17 rename={"call_number": "value"} 18 ) File ~/jupyter/venv/lib/python3.10/site-packages/sqlite_utils/db.py:1764, in Table.extract(self, columns, table, fk_column, rename) 1761 column_order.append(c.name) 1763 # Drop the unnecessary columns and rename lookup column -> 1764 self.transform( 1765 drop=set(columns), 1766 rename={magic_lookup_column: fk_column}, 1767 column_order=column_order, 1768 ) 1770 # And add the foreign key constraint 1771 self.add_foreign_key(fk_column, table, "id") File ~/jupyter/venv/lib/python3.10/site-packages/sqlite_utils/db.py:1526, in Table.transform(self, types, rename, drop, pk, not_null, defaults, drop_foreign_keys, column_order) 1524 with self.db.conn: 1525 for sql in sqls: -> 1526 self.db.execute(sql) 1527 # Run the foreign_key_check before we commit 1528 if pragma_foreign_keys_was_on: File ~/jupyter/venv/lib/python3.10/site-packages/sqlite_utils/db.py:465, in Database.execute(self, sql, parameters) 463 return self.conn.execute(sql, parameters) 464 else: --> 465 return self.conn.execute(sql) OperationalError: database or disk is full ``` This database is about 17G in total size, so I'm assuming the error is coming from the vacuum ... where i'm assuming it's maybe trying to do the temp storage in a location that doesn't have sufficient room. The disk space is more than ample on the host in question (1.8T is free in the directory where the sqlite db resides) The I'm trying to think if there's a way to set the ```python SET the temp file store to be a file ...print(db.execute('PRAGMA temp_store').fetchall()) print(db.execute('PRAGMA temp_store=FILE').fetchall()) print(db.execute('PRAGMA temp_store').fetchall()) the users home directory ...print(db.execute("PRAGMA temp_store_directory='/home/plchuser/'").fetchall()) print(db.execute("PRAGMA sqlite3_temp_directory='/home/plchuser/'").fetchall()) print(db.execute("PRAGMA temp_store_directory").fetchall())
print(db.execute("PRAGMA sqlite3_temp_directory").fetchall())
Here's the docs on the Temporary File Storage Locations https://www.sqlite.org/tempfiles.html |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/430/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1223177069 | PR_kwDOCGYnMM43LrKB | 429 | Depend on click-default-group-wheel | simonw 9599 | closed | 0 | 2 | 2022-05-02T18:03:10Z | 2022-05-02T18:52:42Z | 2022-05-02T18:05:00Z | OWNER | simonw/sqlite-utils/pulls/429 | Trying to get this to work with Pyodide. Refs: https://github.com/simonw/click-default-group-wheel/issues/3 |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/429/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1215216249 | I_kwDOCGYnMM5Ibrp5 | 428 | Research adding support for savepoints | simonw 9599 | open | 0 | 1 | 2022-04-26T01:04:01Z | 2022-04-26T01:05:29Z | OWNER | https://www.sqlite.org/lang_savepoint.html Savepoints are like regular transactions except they have names and can be nested. Would there be any value in adding support to them to |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/428/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1212701569 | I_kwDOCGYnMM5ISFuB | 427 | sqlite-utils convert date parsing recipe complains about trying to parse "*" | wdccdw 1385831 | closed | 0 | 1 | 2022-04-22T19:27:10Z | 2022-07-02T13:59:59Z | 2022-07-02T13:59:32Z | NONE | Missing values in my dataset are denoted by a single asterisk. I am trying to parse string dates into dates. This works fine for columns without missing values, but, when the column contains "*", I get the following: ``` $ sqlite-utils convert ${dbfile} details dob 'r.parsedate(value)' [------------------------------------] 0%Traceback (most recent call last): File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2508, in convert_value return fn(v) File "<string>", line 2, in fn File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/recipes.py", line 8, in parsedate parser.parse(value, dayfirst=dayfirst, yearfirst=yearfirst).date().isoformat() File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 1368, in parse return DEFAULTPARSER.parse(timestr, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/dateutil/parser/_parser.py", line 643, in parse raise ParserError("Unknown string format: %s", timestr) dateutil.parser._parser.ParserError: Unknown string format: * Traceback (most recent call last): File "/usr/local/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils==3.25.1', 'console_scripts', 'sqlite-utils')()) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/click/core.py", line 754, in invoke return __callback(args, **kwargs) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/cli.py", line 2698, in convert db[table].convert( File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 2524, in convert self.db.execute(sql, where_args or []) File "/usr/local/Cellar/sqlite-utils/3.25.1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py", line 458, in execute return self.conn.execute(sql, parameters) sqlite3.OperationalError: user-defined function raised exception ``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/427/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1205687423 | I_kwDOCGYnMM5H3VR_ | 426 | CLI docs should link to Python docs and vice versa | simonw 9599 | closed | 0 | simonw 9599 | 1 | 2022-04-15T16:05:15Z | 2023-07-22T22:13:22Z | 2023-07-22T22:13:22Z | OWNER | For every command/API method there should be a link to the equivalent in the other form factor. Maybe also link to the API and CLI reference pages too. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/426/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | |||||
1203842656 | I_kwDOCGYnMM5HwS5g | 425 | `sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher | simonw 9599 | closed | 0 | 5 | 2022-04-13T22:16:53Z | 2023-04-15T20:14:58Z | 2022-04-13T22:48:57Z | OWNER | Got this error while investigating: - #421 Even though I was using the Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354 |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/425/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1200866134 | I_kwDOCGYnMM5Hk8NW | 424 | Better error message if you try to create a table with no columns | simonw 9599 | closed | 0 | 1 | 2022-04-12T02:43:20Z | 2022-04-13T22:40:15Z | 2022-04-13T22:40:10Z | OWNER | Seen here: Attempting to create a table with no columns produced this confusing error:
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/424/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1199158210 | I_kwDOCGYnMM5HebPC | 423 | .extract() doesn't set foreign key when extracted columns contain NULL value | jlieth 37447552 | closed | 0 | 1 | 2022-04-10T20:05:30Z | 2022-08-27T14:45:04Z | 2022-08-27T14:45:04Z | NONE | I've run into an issue with I'm working with a database with music listening information. Currently it has one large table A simplified demonstration with just In [2]: db = sqlite_utils.Database(memory=True) In [3]: db["listens"].insert_all([ ...: {"id": 1, "track_title": "foo", "album_title": "bar"}, ...: {"id": 2, "track_title": "baz", "album_title": None} ...: ], pk="id") Out[3]: <Table listens (id, track_title, album_title)> ``` The track in the first row has an album, the second track doesn't. Now I extract album information into a separate column: ```ipython In [4]: db["listens"].extract(columns=["album_title"], table="albums", fk_column="album_id") Out[4]: <Table listens (id, track_title, album_id)> In [5]: list(db["albums"].rows) Out[5]: [{'id': 1, 'album_title': 'bar'}, {'id': 2, 'album_title': None}] In [6]: list(db["listens"].rows) Out[6]: [{'id': 1, 'track_title': 'foo', 'album_id': 1}, {'id': 2, 'track_title': 'baz', 'album_id': None}] ``` This behaves as expected -- the Now I want to extract the track information as well. Album information belongs to the track so I want to extract both columns to a new table. ```ipython In [7]: db["listens"].extract(columns=["track_title", "album_id"], table="tracks", fk_column="track_id") Out[7]: <Table listens (id, track_id)> In [8]: list(db["tracks"].rows) Out[8]: [{'id': 1, 'track_title': 'foo', 'album_id': 1}, {'id': 2, 'track_title': 'baz', 'album_id': None}] In [9]: list(db["listens"].rows) Out[9]: [{'id': 1, 'track_id': 1}, {'id': 2, 'track_id': None}] ``` Extracting to the Changing the order of extracts doesn't help. I poked around in the source a bit and I believe this line (essentially comparing |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/423/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1181236173 | I_kwDOCGYnMM5GaDvN | 422 | Reconsider not running convert functions against null values | simonw 9599 | open | 0 | 1 | 2022-03-25T20:22:40Z | 2022-03-25T20:23:21Z | OWNER | I just got caught out by the fact that I had run this code while working on #420 and I wasn't sure why it didn't work: ``` $ sqlite-utils add-column content.db articles score float $ sqlite-utils convert content.db articles score ' import random random.seed(10) def convert(value):
global random
return random.random()
'
I fixed it by doing this instead:
But this indicates to me that the design of |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/422/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1180427792 | I_kwDOCGYnMM5GW-YQ | 421 | "Error: near "(": syntax error" when using sqlite-utils indexes CLI | learning4life 24938923 | closed | 0 | 8 | 2022-03-25T07:12:51Z | 2022-04-13T22:41:59Z | 2022-04-13T22:41:59Z | NONE | This bug relates to https://github.com/simonw/sqlite-utils/issues/408#issuecomment-1066139147 New error when using CLI: "sqlite-utils indexes global.db --table"
Dockerfile ``` FROM centos/python-38-centos7 USER root RUN yum update -y RUN yum upgrade -y epelRUN yum -y install epel-release && yum clean all SQLiteRUN yum -y install zlib-devel geos geos-devel proj proj-devel freexl freexl-devel libxml2-devel WORKDIR /build/ COPY sqlite-autoconf-3360000.tar.gz ./ RUN tar -zxf sqlite-autoconf-3360000.tar.gz WORKDIR /build/sqlite-autoconf-3360000 RUN ./configure RUN make RUN make install RUN /opt/app-root/bin/python3.8 -m pip install --upgrade pip RUN pip install sqlite-utils ``` |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1178546862 | I_kwDOCGYnMM5GPzKu | 420 | Document how to use a `--convert` function that runs initialization code first | strada 770231 | closed | 0 | 12 | 2022-03-23T19:07:36Z | 2022-08-28T11:34:37Z | 2022-03-25T20:07:33Z | NONE | When I have an insert command with transform like this:
I noticed as the number of rows increases the operation becomes quite slow, likely due to the creation of the |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/420/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1178484369 | PR_kwDOCGYnMM405rPe | 419 | Ignore common generated files | eyeseast 25778 | closed | 0 | 1 | 2022-03-23T18:06:22Z | 2022-03-24T21:01:44Z | 2022-03-24T21:01:44Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/419 | Closes #418 This adds four files to
Those are all generated in the course of development and testing. |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/419/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | |||||
1178456794 | I_kwDOCGYnMM5GPdLa | 418 | Add generated files to .gitignore | eyeseast 25778 | closed | 0 | 0 | 2022-03-23T17:48:12Z | 2022-03-24T21:01:44Z | 2022-03-24T21:01:44Z | CONTRIBUTOR | I end up with these in my local directory:
Might as well gitignore them. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/418/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1175744654 | I_kwDOCGYnMM5GFHCO | 417 | insert fails on JSONL with whitespace | blaine 9954 | closed | 0 | 3 | 2022-03-21T17:58:14Z | 2022-03-25T21:19:06Z | 2022-03-25T21:17:13Z | NONE | Any JSON that is newline-delimited and has whitespace (newlines) between the start of a JSON object and an attribute fails due to a parse error. e.g. given the valid JSONL:
I would expect that
It makes sense that since the file is intended to be newline separated, the thing being parsed is "{" (which obviously fails), however the default newline-separated output of Proposed solutions:
1. Default to a "loose" newline-separated parse; this could be implemented internally as [the equivalent of] a It might just have been too early in the morning when I was playing with this, but running pipes of data through sqlite-utils without the 'knack' of it led to some false starts. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/417/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1173023272 | I_kwDOCGYnMM5F6uoo | 416 | Options for how `r.parsedate()` should handle invalid dates | mattkiefer 638427 | closed | 0 | 11 | 2022-03-17T23:29:55Z | 2022-05-03T21:36:49Z | 2022-03-21T04:01:39Z | NONE | Exceptions are normal expected behavior when typecasting an invalid format. However, r.parsedate() is really just re-formatting strings and keeping the type as text. So it may be better to print-and-pass on exception so the user can see a complete list of invalid values -- while also allowing for the parser to reformat the remaining valid values.
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/416/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1171599874 | I_kwDOCGYnMM5F1TIC | 415 | Convert with `--multi` and `--dry-run` flag does not work | dotcs 3976183 | closed | 0 | 2 | 2022-03-16T21:59:46Z | 2022-03-21T04:18:24Z | 2022-03-21T04:18:24Z | NONE | It's not possible to combine Let's first create a simple database from JSON string
and then try to convert the "foo" column with a static value "bar" (see docs Converting a column into multiple columns)
But without the
|
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/415/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1166731361 | I_kwDOCGYnMM5Fiuhh | 414 | I forgot to include the changelog in the 3.25.1 release | simonw 9599 | closed | 0 | 7 | 2022-03-11T18:32:36Z | 2022-03-11T18:40:39Z | 2022-03-11T18:40:39Z | OWNER | I pushed a release for https://github.com/simonw/sqlite-utils/releases/tag/3.25.1 but forgot to include the release notes in This means https://sqlite-utils.datasette.io/en/stable/changelog.html isn't showing them. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/414/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1166587040 | I_kwDOCGYnMM5FiLSg | 413 | Display autodoc type information more legibly | simonw 9599 | closed | 0 | 5 | 2022-03-11T15:58:20Z | 2022-03-11T18:07:10Z | 2022-03-11T18:07:10Z | OWNER | https://sqlite-utils.datasette.io/en/3.25/reference.html#sqlite_utils.db.Table.insert looks like this at the moment: |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/413/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1160182768 | I_kwDOCGYnMM5FJvvw | 412 | Optional Pandas integration | simonw 9599 | open | 0 | 13 | 2022-03-05T01:49:27Z | 2022-06-14T15:36:29Z | OWNER | It would be neat if there was a way to use this more seamlessly with Pandas, in particular Pandas dataframes - but without making Pandas a required dependency. |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/412/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1160034488 | I_kwDOCGYnMM5FJLi4 | 411 | Support for generated columns | eyeseast 25778 | open | 0 | 8 | 2022-03-04T20:41:33Z | 2022-03-11T22:32:43Z | CONTRIBUTOR | This is a fairly new feature -- SQLite version 3.31.0 (2020-01-22) -- that I, admittedly, haven't gotten to work yet. But it looks incredibly useful: https://dgl.cx/2020/06/sqlite-json-support I'm not sure if this is an option on
More here: https://www.sqlite.org/gencol.html |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/411/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1149729902 | PR_kwDOCGYnMM4zbaJy | 410 | Correct spelling mistakes (found with codespell) | EdwardBetts 3818 | closed | 0 | 1 | 2022-02-24T20:44:18Z | 2022-03-06T08:48:29Z | 2022-03-01T21:05:29Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/410 | sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/410/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 | ||||||
1149661489 | I_kwDOCGYnMM5EhnEx | 409 | `with db:` for transactions | simonw 9599 | open | 0 | 3 | 2022-02-24T19:22:06Z | 2022-10-01T03:42:50Z | OWNER | This can be a documented wrapper around |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/409/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
||||||||
1145882578 | I_kwDOCGYnMM5ETMfS | 408 | `deterministic=True` fails on versions of SQLite prior to 3.8.3 | learning4life 24938923 | closed | 0 | 6 | 2022-02-21T14:36:43Z | 2022-03-13T16:54:09Z | 2022-03-02T00:38:11Z | NONE | Hi, love your work. I am unable to lookup indexes in a database using sqlite-utils:
or
Software sqlite-utils, version 3.24 sqlite3 --version: 3.36.0 Output: Traceback (most recent call last): File "/opt/app-root/bin/sqlite-utils", line 8, in <module> sys.exit(cli()) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/opt/app-root/lib64/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, kwargs) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py", line 2123, in indexes ctx.invoke( File "/opt/app-root/lib64/python3.8/site-packages/click/core.py", line 754, in invoke return __callback(args, kwargs) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py", line 1624, in query db.register_fts4_bm25() File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 403, in register_fts4_bm25 self.register_function(rank_bm25, deterministic=True) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 399, in register_function register(fn) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 392, in register self.conn.create_function(name, arity, fn, *kwargs) sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher |
sqlite-utils 140912432 | issue | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/408/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | ||||||
1138948786 | PR_kwDOCGYnMM4y3yW0 | 407 | Add SpatiaLite helpers to CLI | eyeseast 25778 | closed | 0 | 7 | 2022-02-15T16:50:17Z | 2022-02-16T01:49:40Z | 2022-02-16T00:58:08Z | CONTRIBUTOR | simonw/sqlite-utils/pulls/407 | Closes #398 This adds SpatiaLite helpers to the CLI. ```sh init spatialite when creating a databasesqlite-utils create database.db --enable-wal --init-spatialite add geometry columnsneeds a database, table, geometry column name, type, with optional SRID and not-nullthis will throw an error if the table doesn't already existsqlite-utils add-geometry-column database.db table-name geometry --srid 4326 --not-null spatial index an existing table/columnthis will throw an error it the table and column don't existsqlite-utils create-spatial-index database.db table-name geometry ``` Docs and tests are included. |
sqlite-utils 140912432 | pull | { "url": "https://api.github.com/repos/simonw/sqlite-utils/issues/407/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issues] ( [id] INTEGER PRIMARY KEY, [node_id] TEXT, [number] INTEGER, [title] TEXT, [user] INTEGER REFERENCES [users]([id]), [state] TEXT, [locked] INTEGER, [assignee] INTEGER REFERENCES [users]([id]), [milestone] INTEGER REFERENCES [milestones]([id]), [comments] INTEGER, [created_at] TEXT, [updated_at] TEXT, [closed_at] TEXT, [author_association] TEXT, [pull_request] TEXT, [body] TEXT, [repo] INTEGER REFERENCES [repos]([id]), [type] TEXT , [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT); CREATE INDEX [idx_issues_repo] ON [issues] ([repo]); CREATE INDEX [idx_issues_milestone] ON [issues] ([milestone]); CREATE INDEX [idx_issues_assignee] ON [issues] ([assignee]); CREATE INDEX [idx_issues_user] ON [issues] ([user]);