issue_comments
616 rows where author_association = "CONTRIBUTOR" sorted by html_url descending
This data as json, CSV (advanced)
Suggested facets: created_at (date), updated_at (date)
issue >30
- Upgrade to CodeMirror 6, add SQL autocomplete 21
- feat: Javascript Plugin API (Custom panels, column menu items with JS actions) 14
- Database page loads too slowly with many large tables (due to table counts) 13
- Stream all results for arbitrary SQL and canned queries 10
- docker image is duplicating db files somehow 10
- Handle spatialite geometry columns better 7
- base_url configuration setting 7
- Exceeding Cloud Run memory limits when deploying a 4.8G database 7
- create-index should run analyze after creating index 7
- Add new spatialite helper methods 7
- Add register_output_renderer hook 6
- Helper methods for working with SpatiaLite 6
- Plugin hook for dynamic metadata 6
- clean checkout & clean environment has test failures 6
- [WIP] Add publish to heroku support 5
- Scripted exports 5
- datasette publish lambda plugin 4
- Build Dockerfile with recent Sqlite + Spatialite 4
- Documentation with recommendations on running Datasette in production without using Docker 4
- bpylist.archiver.CircularReference: archive has a cycle with uid(13) 4
- Add insert --truncate option 4
- Make it easier to insert geometries, with documentation and maybe code 4
- Advanced class-based `conversions=` mechanism 4
- query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data 4
- Proposal: datasette query 4
- Writable canned queries fail with useless non-error against immutable databases 4
- Ability to merge databases and tables 4
- API to insert a single record into an existing table 4
- Exclude virtual tables from datasette inspect 4
- Call for birthday presents: if you're using Datasette, let us know how you're using it here 4
- …
reactions 7 ✖
- {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 589
- {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 20
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1} 2
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 2
- {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1} 1
- {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 1
- {"total_count": 3, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 3, "rocket": 0, "eyes": 0} 1
id | html_url ▲ | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1843975536 | https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843975536 | https://api.github.com/repos/simonw/sqlite-utils/issues/604 | IC_kwDOCGYnMM5t6NVw | tkhattra 16437338 | 2023-12-07T01:17:05Z | 2023-12-07T01:17:05Z | CONTRIBUTOR | Apologies - I pushed a fix that addresses the mypy failures. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add more STRICT table support 2001006157 | |
527211047 | https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527211047 | https://api.github.com/repos/simonw/sqlite-utils/issues/57 | MDEyOklzc3VlQ29tbWVudDUyNzIxMTA0Nw== | amjith 49260 | 2019-09-02T17:30:43Z | 2019-09-02T17:30:43Z | CONTRIBUTOR | I have merged the other PR (#56) into this one. I have incorporated your suggestions. Cheers! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add triggers while enabling FTS 487987958 | |
527209840 | https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527209840 | https://api.github.com/repos/simonw/sqlite-utils/issues/56 | MDEyOklzc3VlQ29tbWVudDUyNzIwOTg0MA== | amjith 49260 | 2019-09-02T17:23:21Z | 2019-09-02T17:23:21Z | CONTRIBUTOR | I have updated the other PR with the changes from this one and added tests. I have also changed the escaping from double quotes to brackets. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Escape the table name in populate_fts and search. 487847945 | |
1501017004 | https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1501017004 | https://api.github.com/repos/simonw/sqlite-utils/issues/531 | IC_kwDOCGYnMM5Zd7Os | eyeseast 25778 | 2023-04-09T01:49:43Z | 2023-04-09T01:49:43Z | CONTRIBUTOR | I'm going to close this in favor of #536. Will try a cleaner approach to custom paths once that one is merge. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add paths for homebrew on Apple silicon 1620164673 | |
1465315726 | https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465315726 | https://api.github.com/repos/simonw/sqlite-utils/issues/531 | IC_kwDOCGYnMM5XVvGO | eyeseast 25778 | 2023-03-12T22:21:56Z | 2023-03-12T22:21:56Z | CONTRIBUTOR | Exactly, that's what I was running into. On my M2 MacBook, SpatiaLite ends up in what is -- for the moment -- a non-standard location, so even when I passed in the location with What I learned doing this originally is that SQLite needs to load the extension for each connection, even if all the SpatiaLite stuff is already in the database. So that's why Here's the code where I hit the error: https://github.com/eyeseast/boston-parcels/blob/main/Makefile#L30 It works using this branch. I'm not attached to this solution if you can think of something better. And I'm not sure, TBH, my test would actually catch what I'm after here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add paths for homebrew on Apple silicon 1620164673 | |
1297788531 | https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297788531 | https://api.github.com/repos/simonw/sqlite-utils/issues/508 | IC_kwDOCGYnMM5NWq5z | chapmanjacobd 7908073 | 2022-10-31T22:54:33Z | 2022-11-17T15:11:16Z | CONTRIBUTOR | Maybe this is actually a problem in the python sqlite bindings. Given SQLITE's stance on this they should probably use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Allow surrogates in parameters 1430563092 | |
1292401308 | https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1292401308 | https://api.github.com/repos/simonw/sqlite-utils/issues/499 | IC_kwDOCGYnMM5NCHqc | chapmanjacobd 7908073 | 2022-10-26T17:54:26Z | 2022-10-26T17:54:51Z | CONTRIBUTOR | The problem with how it is currently is that the transformed fts table will return incorrect results (unless the table was only 1 row or something), even if create_triggers was enabled previously. Maybe the simplest solution is to disable fts on a transformed table rather than try to recreate it? Thoughts? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
feat: recreate fts triggers after table transform 1405196044 | |
1274153135 | https://github.com/simonw/sqlite-utils/pull/498#issuecomment-1274153135 | https://api.github.com/repos/simonw/sqlite-utils/issues/498 | IC_kwDOCGYnMM5L8giv | chapmanjacobd 7908073 | 2022-10-11T06:34:31Z | 2022-10-11T06:34:31Z | CONTRIBUTOR | nevermind it was because I was running |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fix: enable-fts permanently save triggers 1404013495 | |
1232356302 | https://github.com/simonw/sqlite-utils/pull/480#issuecomment-1232356302 | https://api.github.com/repos/simonw/sqlite-utils/issues/480 | IC_kwDOCGYnMM5JdEPO | chapmanjacobd 7908073 | 2022-08-31T01:51:49Z | 2022-08-31T01:51:49Z | CONTRIBUTOR | Thanks for pointing me to the right place |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
search_sql add include_rank option 1355433619 | |
1040998433 | https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040998433 | https://api.github.com/repos/simonw/sqlite-utils/issues/407 | IC_kwDOCGYnMM4-DGAh | eyeseast 25778 | 2022-02-16T01:29:39Z | 2022-02-16T01:29:39Z | CONTRIBUTOR | Happy to do it and have it in the library. Going to use it a bunch. This whole SpatiaLite toolchain become a huge part of my work in the past year. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add SpatiaLite helpers to CLI 1138948786 | |
1040580250 | https://github.com/simonw/sqlite-utils/pull/407#issuecomment-1040580250 | https://api.github.com/repos/simonw/sqlite-utils/issues/407 | IC_kwDOCGYnMM4-Bf6a | eyeseast 25778 | 2022-02-15T17:40:00Z | 2022-02-15T17:40:00Z | CONTRIBUTOR | @simonw I think this is ready for a look. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add SpatiaLite helpers to CLI 1138948786 | |
1030002502 | https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1030002502 | https://api.github.com/repos/simonw/sqlite-utils/issues/385 | IC_kwDOCGYnMM49ZJdG | eyeseast 25778 | 2022-02-04T13:50:19Z | 2022-02-04T13:50:19Z | CONTRIBUTOR | Awesome. Thanks for your help getting it in. Will now look at adding CLI versions of this. It's going to be super helpful on a bunch of my projects. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add new spatialite helper methods 1102899312 | |
1029370537 | https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029370537 | https://api.github.com/repos/simonw/sqlite-utils/issues/385 | IC_kwDOCGYnMM49WvKp | eyeseast 25778 | 2022-02-03T20:25:58Z | 2022-02-03T20:25:58Z | CONTRIBUTOR | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add new spatialite helper methods 1102899312 | ||
1029338360 | https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029338360 | https://api.github.com/repos/simonw/sqlite-utils/issues/385 | IC_kwDOCGYnMM49WnT4 | eyeseast 25778 | 2022-02-03T19:43:56Z | 2022-02-03T19:43:56Z | CONTRIBUTOR | Works for me. I was just looking at how the FTS extensions work and they're just methods, too. So this can be consistent with that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add new spatialite helper methods 1102899312 | |
1029326568 | https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029326568 | https://api.github.com/repos/simonw/sqlite-utils/issues/385 | IC_kwDOCGYnMM49Wkbo | eyeseast 25778 | 2022-02-03T19:28:26Z | 2022-02-03T19:28:26Z | CONTRIBUTOR |
This is fixed now. I had to take out the type annotations for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add new spatialite helper methods 1102899312 | |
1029306428 | https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029306428 | https://api.github.com/repos/simonw/sqlite-utils/issues/385 | IC_kwDOCGYnMM49Wfg8 | eyeseast 25778 | 2022-02-03T19:03:43Z | 2022-02-03T19:03:43Z | CONTRIBUTOR | I thought about adding these as methods on I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API. You could have |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add new spatialite helper methods 1102899312 | |
1029180984 | https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029180984 | https://api.github.com/repos/simonw/sqlite-utils/issues/385 | IC_kwDOCGYnMM49WA44 | eyeseast 25778 | 2022-02-03T16:42:04Z | 2022-02-03T16:42:04Z | CONTRIBUTOR | Fixed my spelling. That's a useful thing. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add new spatialite helper methods 1102899312 | |
1029175907 | https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029175907 | https://api.github.com/repos/simonw/sqlite-utils/issues/385 | IC_kwDOCGYnMM49V_pj | eyeseast 25778 | 2022-02-03T16:36:54Z | 2022-02-03T16:36:54Z | CONTRIBUTOR | @simonw Not sure if you've seen this, but any chance you can run the tests? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add new spatialite helper methods 1102899312 | |
916119657 | https://github.com/simonw/sqlite-utils/pull/326#issuecomment-916119657 | https://api.github.com/repos/simonw/sqlite-utils/issues/326 | IC_kwDOCGYnMM42muBp | meatcar 191622 | 2021-09-09T13:54:10Z | 2021-09-09T13:54:10Z | CONTRIBUTOR | dupe of #293? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Test against 3.10-dev 991237645 | |
1404070841 | https://github.com/simonw/sqlite-utils/pull/203#issuecomment-1404070841 | https://api.github.com/repos/simonw/sqlite-utils/issues/203 | IC_kwDOCGYnMM5TsGu5 | fgregg 536941 | 2023-01-25T18:47:18Z | 2023-01-25T18:47:18Z | CONTRIBUTOR | i'll adopt this PR to make the changes @simonw suggested https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932 |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
changes to allow for compound foreign keys 743384829 | |
717359145 | https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717359145 | https://api.github.com/repos/simonw/sqlite-utils/issues/189 | MDEyOklzc3VlQ29tbWVudDcxNzM1OTE0NQ== | adamwolf 35681 | 2020-10-27T16:20:32Z | 2020-10-27T16:20:32Z | CONTRIBUTOR | No problem. I added a test. Let me know if it looks sufficient or if you want me to to tweak something! If you don't mind, would you tag this PR as "hacktoberfest-accepted"? If you do mind, no problem and I'm sorry for asking :) My kiddos like the shirts. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Allow iterables other than Lists in m2m records 729818242 | |
688573964 | https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688573964 | https://api.github.com/repos/simonw/sqlite-utils/issues/146 | MDEyOklzc3VlQ29tbWVudDY4ODU3Mzk2NA== | simonwiles 96218 | 2020-09-08T01:55:07Z | 2020-09-08T01:55:07Z | CONTRIBUTOR | Okay, I've rewritten this PR to preserve the batching behaviour but still fix #145, and rebased the branch to account for the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle case where subsequent records (after first batch) include extra columns 688668680 | |
688481317 | https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688481317 | https://api.github.com/repos/simonw/sqlite-utils/issues/146 | MDEyOklzc3VlQ29tbWVudDY4ODQ4MTMxNw== | simonwiles 96218 | 2020-09-07T19:18:55Z | 2020-09-07T19:18:55Z | CONTRIBUTOR | Just force-pushed to update d042f9c with more formatting changes to satisfy |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle case where subsequent records (after first batch) include extra columns 688668680 | |
688479163 | https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688479163 | https://api.github.com/repos/simonw/sqlite-utils/issues/146 | MDEyOklzc3VlQ29tbWVudDY4ODQ3OTE2Mw== | simonwiles 96218 | 2020-09-07T19:10:33Z | 2020-09-07T19:11:57Z | CONTRIBUTOR | @simonw -- I've gone ahead updated the documentation to reflect the changes introduced in this PR. IMO it's ready to merge now. In writing the documentation changes, I begin to wonder about the value and role of Of course the documentation will need to change again too if/when something is done about #147. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle case where subsequent records (after first batch) include extra columns 688668680 | |
655643078 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA== | tsibley 79913 | 2020-07-08T17:05:59Z | 2020-07-08T17:05:59Z | CONTRIBUTOR |
Ah, yes, thanks for this reminder! I've repushed with doc bits added. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655239728 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA== | tsibley 79913 | 2020-07-08T02:16:42Z | 2020-07-08T02:16:42Z | CONTRIBUTOR | I fixed my original oops by moving the I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted): ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index d6b9ecf..4107ceb 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -1028,6 +1028,11 @@ class Table(Queryable): batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns)) self.last_rowid = None self.last_pk = None + with self.db.conn: + # Explicit BEGIN is necessary because Python's sqlite3 doesn't + # issue implicit BEGINs for DDL, only DML. We mix DDL and DML + # below and might execute DDL first, e.g. for table creation. + self.db.conn.execute("BEGIN") if truncate and self.exists(): self.db.conn.execute("DELETE FROM [{}];".format(self.name)) for chunk in chunks(itertools.chain([first_record], records), batch_size): @@ -1038,7 +1043,11 @@ class Table(Queryable): # Use the first batch to derive the table names column_types = suggest_column_types(chunk) column_types.update(columns or {}) - self.create( + # Not self.create() because that is wrapped in its own + # transaction and Python's sqlite3 doesn't support + # nested transactions. + self.db.create_table( + self.name, column_types, pk, foreign_keys, @@ -1139,7 +1148,6 @@ class Table(Queryable): flat_values = list(itertools.chain(*values)) queries_and_params = [(sql, flat_values)]
but that fails in tests because other methods call Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the There is also the caveat that for each transaction, an explicit |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655052451 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ== | tsibley 79913 | 2020-07-07T18:45:23Z | 2020-07-07T18:45:23Z | CONTRIBUTOR | Ah, I see the problem. The truncate is inside a loop I didn't realize was there. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
655018966 | https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655018966 | https://api.github.com/repos/simonw/sqlite-utils/issues/118 | MDEyOklzc3VlQ29tbWVudDY1NTAxODk2Ng== | tsibley 79913 | 2020-07-07T17:41:06Z | 2020-07-07T17:41:06Z | CONTRIBUTOR | Hmm, while tests pass, this may not work as intended on larger datasets. Looking into it. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add insert --truncate option 651844316 | |
1029317527 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM49WiOX | eyeseast 25778 | 2022-02-03T19:18:02Z | 2022-02-03T19:18:02Z | CONTRIBUTOR | Taking part of the conversation from #385 here.
Yes. And also |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Helper methods for working with SpatiaLite 557842245 | |
1013698557 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1013698557 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48a8_9 | eyeseast 25778 | 2022-01-15T15:15:22Z | 2022-01-15T15:15:22Z | CONTRIBUTOR | @simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Helper methods for working with SpatiaLite 557842245 | |
1012413729 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012413729 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48WDUh | eyeseast 25778 | 2022-01-13T18:50:00Z | 2022-01-13T18:50:00Z | CONTRIBUTOR | One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Helper methods for working with SpatiaLite 557842245 | |
1012253198 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012253198 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48VcIO | eyeseast 25778 | 2022-01-13T15:39:14Z | 2022-01-13T15:39:14Z | CONTRIBUTOR | Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file ( |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Helper methods for working with SpatiaLite 557842245 | |
1012230212 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012230212 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48VWhE | eyeseast 25778 | 2022-01-13T15:15:13Z | 2022-01-13T15:15:13Z | CONTRIBUTOR | Some proposals I'd add to sqlite-utils: Some version of this, from geojson-to-sqlite:
Also a function for creating a spatial index:
I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Helper methods for working with SpatiaLite 557842245 | |
1012158895 | https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1012158895 | https://api.github.com/repos/simonw/sqlite-utils/issues/79 | IC_kwDOCGYnMM48VFGv | eyeseast 25778 | 2022-01-13T13:55:59Z | 2022-01-13T13:55:59Z | CONTRIBUTOR | Came here to add this. I might pick it up. Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Helper methods for working with SpatiaLite 557842245 | |
573389669 | https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573389669 | https://api.github.com/repos/simonw/sqlite-utils/issues/74 | MDEyOklzc3VlQ29tbWVudDU3MzM4OTY2OQ== | jayvdb 15092 | 2020-01-12T07:21:17Z | 2020-01-12T07:21:17Z | CONTRIBUTOR | I guess there is some extra flag for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column 546073980 | |
573388052 | https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573388052 | https://api.github.com/repos/simonw/sqlite-utils/issues/74 | MDEyOklzc3VlQ29tbWVudDU3MzM4ODA1Mg== | jayvdb 15092 | 2020-01-12T06:51:30Z | 2020-01-12T06:51:30Z | CONTRIBUTOR | Thanks. That showed me that there was a click cli runner error, and setting |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column 546073980 | |
533818697 | https://github.com/simonw/sqlite-utils/issues/61#issuecomment-533818697 | https://api.github.com/repos/simonw/sqlite-utils/issues/61 | MDEyOklzc3VlQ29tbWVudDUzMzgxODY5Nw== | amjith 49260 | 2019-09-21T18:09:01Z | 2019-09-21T18:09:28Z | CONTRIBUTOR | @witeshadow The library version doesn't have helpers around CSV (at least not from what I can see in the code). But here's a snippet that makes it easy to insert from CSV using the library. ``` import csv from sqlite_utils import Database CSV Readercsv_file = open("filename.csv") # open the csv file. reader = csv.reader(csv_file) # Create a CSV reader headers = next(reader) # First line is the header docs = (dict(zip(headers, row)) for row in reader) Now you can use the
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
importing CSV to SQLite as library 491219910 | |
1668113177 | https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1668113177 | https://api.github.com/repos/simonw/sqlite-utils/issues/578 | IC_kwDOCGYnMM5jbWMZ | eyeseast 25778 | 2023-08-07T15:41:49Z | 2023-08-07T15:41:49Z | CONTRIBUTOR | I wonder if this should be two hooks: input and output. The current |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook for adding new output formats 1818838294 | |
1648339661 | https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648339661 | https://api.github.com/repos/simonw/sqlite-utils/issues/578 | IC_kwDOCGYnMM5iP6rN | eyeseast 25778 | 2023-07-24T17:44:30Z | 2023-07-24T17:44:30Z | CONTRIBUTOR |
This is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this: ```sh sqlite-utils insert data.db table file.geojson --format geojson ... explore and manipulate in Datasettesqlite-utils query data.db ... --format geojson > output.geojson ``` This would work especially well with datasette-query-files, since I already have the queries I need saved in standalone SQL files. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook for adding new output formats 1818838294 | |
1646656283 | https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646656283 | https://api.github.com/repos/simonw/sqlite-utils/issues/567 | IC_kwDOCGYnMM5iJfsb | eyeseast 25778 | 2023-07-22T19:32:24Z | 2023-07-22T19:32:24Z | CONTRIBUTOR | Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin system 1801394744 | |
1642808866 | https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1642808866 | https://api.github.com/repos/simonw/sqlite-utils/issues/567 | IC_kwDOCGYnMM5h60Yi | eyeseast 25778 | 2023-07-19T21:54:27Z | 2023-07-19T21:54:27Z | CONTRIBUTOR | Would this possibly make a bunch of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin system 1801394744 | |
1638910473 | https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473 | https://api.github.com/repos/simonw/sqlite-utils/issues/567 | IC_kwDOCGYnMM5hr8oJ | asg017 15178711 | 2023-07-17T21:27:41Z | 2023-07-17T21:27:41Z | CONTRIBUTOR | Another use-case: I want to make a
This could also be a datasette plugin (ex |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin system 1801394744 | |
1590531892 | https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892 | https://api.github.com/repos/simonw/sqlite-utils/issues/557 | IC_kwDOCGYnMM5ezZc0 | chapmanjacobd 7908073 | 2023-06-14T06:09:21Z | 2023-06-14T06:09:21Z | CONTRIBUTOR | I put together a simple script to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help ``` CREATE TABLE my_table ( id INTEGER PRIMARY KEY, column1 TEXT, column2 TEXT, column3 TEXT ); INSERT INTO my_table (column1, column2, column3) VALUES ('Value 1', 'Duplicate 1', 'Duplicate A'), ('Value 2', 'Duplicate 2', 'Duplicate B'), ('Value 3', 'Duplicate 2', 'Duplicate C'), ('Value 4', 'Duplicate 3', 'Duplicate D'), ('Value 5', 'Duplicate 3', 'Duplicate E'), ('Value 6', 'Duplicate 3', 'Duplicate F'); ```
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Aliased ROWID option for tables created from alter=True commands 1740150327 | |
1577355134 | https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134 | https://api.github.com/repos/simonw/sqlite-utils/issues/557 | IC_kwDOCGYnMM5eBId- | chapmanjacobd 7908073 | 2023-06-05T19:26:26Z | 2023-06-05T19:26:26Z | CONTRIBUTOR | this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Aliased ROWID option for tables created from alter=True commands 1740150327 | |
1575310378 | https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378 | https://api.github.com/repos/simonw/sqlite-utils/issues/556 | IC_kwDOCGYnMM5d5VQq | mcint 601708 | 2023-06-04T01:21:15Z | 2023-06-04T01:21:15Z | CONTRIBUTOR | I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support storing incrementally piped values 1740026046 | |
1592047502 | https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502 | https://api.github.com/repos/simonw/sqlite-utils/issues/555 | IC_kwDOCGYnMM5e5LeO | chapmanjacobd 7908073 | 2023-06-14T22:00:10Z | 2023-06-14T22:01:57Z | CONTRIBUTOR | You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python. That might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the You may also try chunking like this: ```py def chunks(lst, n) -> Generator: for i in range(0, len(lst), n): yield lst[i : i + n] SQLITE_PARAM_LIMIT = 32765 data = [] chunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT) for ids in chunked: data.expand( list( db.query( f"""SELECT * from videos WHERE id in (""" + ",".join(["?"] * len(ids)) + ")", (*ids,), ) ) ) ``` but that actually won't work with your Since you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Filter table by a large bunch of ids 1733198948 | |
1592052320 | https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320 | https://api.github.com/repos/simonw/sqlite-utils/issues/535 | IC_kwDOCGYnMM5e5Mpg | chapmanjacobd 7908073 | 2023-06-14T22:05:28Z | 2023-06-14T22:05:28Z | CONTRIBUTOR | piping to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
rows: --transpose or psql extended view-like functionality 1655860104 | |
1592110694 | https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694 | https://api.github.com/repos/simonw/sqlite-utils/issues/529 | IC_kwDOCGYnMM5e5a5m | chapmanjacobd 7908073 | 2023-06-14T23:11:47Z | 2023-06-14T23:12:12Z | CONTRIBUTOR | sorry i was wrong. ``` sqlite-utils --raw-lines :memory: "SELECT * FROM (VALUES ('test'), ('line2'))" | cat -A test$ line2$ sqlite-utils --csv --no-headers :memory: "SELECT * FROM (VALUES ('test'), ('line2'))" | cat -A test$ line2$ ``` I think this was fixed somewhat recently |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Microsoft line endings 1581090327 | |
1540900733 | https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1540900733 | https://api.github.com/repos/simonw/sqlite-utils/issues/527 | IC_kwDOCGYnMM5b2Ed9 | mcarpenter 167893 | 2023-05-09T21:15:05Z | 2023-05-09T21:15:05Z | CONTRIBUTOR | Sorry, I completely missed your first comment whilst on Easter break. This looks like a good practical compromise before v4. Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`Table.convert()` skips falsey values 1578790070 | |
1435318713 | https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1435318713 | https://api.github.com/repos/simonw/sqlite-utils/issues/525 | IC_kwDOCGYnMM5VjTm5 | mcarpenter 167893 | 2023-02-17T21:55:01Z | 2023-02-17T21:55:01Z | CONTRIBUTOR | Meanwhile, a cheap workaround is to invalidate the registered function cache:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Repeated calls to `Table.convert()` fail 1575131737 | |
1423387341 | https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1423387341 | https://api.github.com/repos/simonw/sqlite-utils/issues/525 | IC_kwDOCGYnMM5U1yrN | mcarpenter 167893 | 2023-02-08T23:48:52Z | 2023-02-09T00:17:30Z | CONTRIBUTOR | PR below |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Repeated calls to `Table.convert()` fail 1575131737 | |
1419357290 | https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419357290 | https://api.github.com/repos/simonw/sqlite-utils/issues/524 | IC_kwDOCGYnMM5Umaxq | eyeseast 25778 | 2023-02-06T16:21:44Z | 2023-02-06T16:21:44Z | CONTRIBUTOR | SQLite doesn't have a native |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Transformation type `--type DATETIME` 1572766460 | |
1407264466 | https://github.com/simonw/sqlite-utils/issues/523#issuecomment-1407264466 | https://api.github.com/repos/simonw/sqlite-utils/issues/523 | IC_kwDOCGYnMM5T4SbS | fgregg 536941 | 2023-01-28T02:41:14Z | 2023-01-28T02:41:14Z | CONTRIBUTOR | I also often then run another little script to cast all empty strings to null, but i save that for another issue if this gets accepted. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature request: trim all leading and trailing white space for all columns for all tables in a database 1560651350 | |
1421571810 | https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1421571810 | https://api.github.com/repos/simonw/sqlite-utils/issues/520 | IC_kwDOCGYnMM5Uu3bi | mcarpenter 167893 | 2023-02-07T22:43:09Z | 2023-02-07T22:43:09Z | CONTRIBUTOR | Hey, isn't this essentially the same issue as #448 ? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
rows_from_file() raises confusing error if file-like object is not in binary mode 1516644980 | |
1304320521 | https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304320521 | https://api.github.com/repos/simonw/sqlite-utils/issues/511 | IC_kwDOCGYnMM5NvloJ | chapmanjacobd 7908073 | 2022-11-04T22:54:09Z | 2022-11-04T22:59:54Z | CONTRIBUTOR | I ran |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
[insert_all, upsert_all] IntegrityError: constraint failed 1436539554 | |
1304078945 | https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304078945 | https://api.github.com/repos/simonw/sqlite-utils/issues/511 | IC_kwDOCGYnMM5Nuqph | chapmanjacobd 7908073 | 2022-11-04T19:38:36Z | 2022-11-04T20:13:17Z | CONTRIBUTOR | Even more bizarre, the source db only has one record and the target table has no conflicting record:
I've been able to use this code successfully several times before so not sure what's causing the issue. I guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
[insert_all, upsert_all] IntegrityError: constraint failed 1436539554 | |
1318777114 | https://github.com/simonw/sqlite-utils/issues/510#issuecomment-1318777114 | https://api.github.com/repos/simonw/sqlite-utils/issues/510 | IC_kwDOCGYnMM5OmvEa | chapmanjacobd 7908073 | 2022-11-17T15:09:47Z | 2022-11-17T15:09:47Z | CONTRIBUTOR | why close? is the only problem that the _config table that incorrectly says 4 for fts5? if so, that's still something that should be fixed |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Cannot enable FTS5 despite it being available 1434911255 | |
1297859539 | https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539 | https://api.github.com/repos/simonw/sqlite-utils/issues/507 | IC_kwDOCGYnMM5NW8PT | chapmanjacobd 7908073 | 2022-11-01T00:40:16Z | 2022-11-01T00:40:16Z | CONTRIBUTOR | Ideally people could fix their data if they run into this issue. If you are using filenames try convmv
maybe this script will also help: ```py import argparse, shutil from pathlib import Path import ftfy from xklb import utils from xklb.utils import log def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument("paths", nargs='*') parser.add_argument("--verbose", "-v", action="count", default=0) args = parser.parse_args()
def rename_invalid_paths() -> None: args = parse_args()
if name == "main": rename_invalid_paths() ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character 1430325103 | |
1303660293 | https://github.com/simonw/sqlite-utils/issues/50#issuecomment-1303660293 | https://api.github.com/repos/simonw/sqlite-utils/issues/50 | IC_kwDOCGYnMM5NtEcF | chapmanjacobd 7908073 | 2022-11-04T14:38:36Z | 2022-11-04T14:38:36Z | CONTRIBUTOR | where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"Too many SQL variables" on large inserts 473083260 | |
1264219650 | https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1264219650 | https://api.github.com/repos/simonw/sqlite-utils/issues/493 | IC_kwDOCGYnMM5LWnYC | chapmanjacobd 7908073 | 2022-10-01T03:22:50Z | 2022-10-01T03:23:58Z | CONTRIBUTOR | this is likely what you are looking for: https://stackoverflow.com/a/51076749/697964 but yeah I would say just disable smart quotes |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Tiny typographical error in install/uninstall docs 1386562662 | |
1264218914 | https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1264218914 | https://api.github.com/repos/simonw/sqlite-utils/issues/491 | IC_kwDOCGYnMM5LWnMi | chapmanjacobd 7908073 | 2022-10-01T03:18:36Z | 2023-06-14T22:14:24Z | CONTRIBUTOR |
I actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names. here is a full script: ``` import argparse from pathlib import Path from sqlite_utils import Database def connect(args, conn=None, kwargs) -> Database: db = Database(conn or args.database, kwargs) with db.conn: db.conn.execute("PRAGMA main.cache_size = 8000") return db def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument("database") parser.add_argument("dbs_folder") parser.add_argument("--db", "-db", help=argparse.SUPPRESS) parser.add_argument("--verbose", "-v", action="count", default=0) args = parser.parse_args()
def merge_db(args, source_db): source_db = str(Path(source_db).resolve())
def merge_directory(): args = parse_args() source_dbs = list(Path(args.dbs_folder).glob('*.db')) for s_db in source_dbs: merge_db(args, s_db) if name == 'main': merge_directory() ``` edit: I've made some improvements to this and put it on PyPI: ``` $ pip install xklb $ lb merge-db -h usage: library merge-dbs DEST_DB SOURCE_DB ... [--only-target-columns] [--only-new-rows] [--upsert] [--pk PK ...] [--table TABLE ...]
positional arguments: database source_dbs ``` Also if you want to dedupe a table based on a "business key" which isn't explicitly your primary key(s) you can run this: ``` $ lb dedupe-db -h usage: library dedupe-dbs DATABASE TABLE --bk BUSINESS_KEYS [--pk PRIMARY_KEYS] [--only-columns COLUMNS]
positional arguments: database table options: -h, --help show this help message and exit --skip-0 --only-columns ONLY_COLUMNS Comma separated column names to upsert --primary-keys PRIMARY_KEYS, --pk PRIMARY_KEYS Comma separated primary keys --business-keys BUSINESS_KEYS, --bk BUSINESS_KEYS Comma separated business keys ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to merge databases and tables 1383646615 | |
1258712931 | https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258712931 | https://api.github.com/repos/simonw/sqlite-utils/issues/491 | IC_kwDOCGYnMM5LBm9j | eyeseast 25778 | 2022-09-26T22:31:58Z | 2022-09-26T22:31:58Z | CONTRIBUTOR | Right. The backup command will copy tables completely, but in the case of conflicting table names, the destination gets overwritten silently. That might not be what you want here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to merge databases and tables 1383646615 | |
1258508215 | https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1258508215 | https://api.github.com/repos/simonw/sqlite-utils/issues/491 | IC_kwDOCGYnMM5LA0-3 | eyeseast 25778 | 2022-09-26T19:22:14Z | 2022-09-26T19:22:14Z | CONTRIBUTOR | This might be fairly straightforward using SQLite's backup utility: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to merge databases and tables 1383646615 | |
1256858763 | https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1256858763 | https://api.github.com/repos/simonw/sqlite-utils/issues/491 | IC_kwDOCGYnMM5K6iSL | chapmanjacobd 7908073 | 2022-09-24T04:50:59Z | 2022-09-24T04:52:08Z | CONTRIBUTOR | Instead of outputting binary data to stdout the interface might be better like this
similar to Actually I think this might already be possible within
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to merge databases and tables 1383646615 | |
1224382336 | https://github.com/simonw/sqlite-utils/issues/467#issuecomment-1224382336 | https://api.github.com/repos/simonw/sqlite-utils/issues/467 | IC_kwDOCGYnMM5I-peA | jefftriplett 50527 | 2022-08-23T17:16:13Z | 2022-08-23T17:16:13Z | CONTRIBUTOR |
I would lean towards not dropping them (or making a I do like that sqlite-utils mostly just works with existing tables but it's also nice to add to existing fields in a few cases. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for ensuring a table has all the columns 1348169997 | |
1190277829 | https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190277829 | https://api.github.com/repos/simonw/sqlite-utils/issues/456 | IC_kwDOCGYnMM5G8jLF | fgregg 536941 | 2022-07-20T13:19:15Z | 2022-07-20T13:19:15Z | CONTRIBUTOR | hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
feature request: pivot command 1310243385 | |
1190272780 | https://github.com/simonw/sqlite-utils/issues/456#issuecomment-1190272780 | https://api.github.com/repos/simonw/sqlite-utils/issues/456 | IC_kwDOCGYnMM5G8h8M | fgregg 536941 | 2022-07-20T13:14:54Z | 2022-07-20T13:14:54Z | CONTRIBUTOR | for example, i have data on votes that look like this: | ballot_id | option_id | choice | |-|-|-| | 1 | 1 | 0 | | 1 | 2 | 1 | | 1 | 3 | 0 | | 1 | 4 | 1 | | 2 | 1 | 1 | | 2 | 2 | 0 | | 2 | 3 | 1 | | 2 | 4 | 0 | and i want to reshape from this long form to this wide form: | ballot_id | option_id_1 | option_id_2 | option_id_3 | option_id_ 4| |-|-|-|-| -| | 1 | 0 | 1 | 0 | 1 | | 2 | 1 | 0 | 1| 0 | i could do such a think like this.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
feature request: pivot command 1310243385 | |
1179579878 | https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1179579878 | https://api.github.com/repos/simonw/sqlite-utils/issues/449 | IC_kwDOCGYnMM5GTvXm | davidleejy 1690072 | 2022-07-09T17:41:32Z | 2022-07-09T17:41:50Z | CONTRIBUTOR | Learnt that the types in Sqlite-utils differ somewhat from those in Sqlite. I've changed my test to account for this difference and the test has passed successfully. I will submit a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Utilities for duplicating tables and creating a table with the results of a query 1279863844 | |
1174027079 | https://github.com/simonw/sqlite-utils/issues/449#issuecomment-1174027079 | https://api.github.com/repos/simonw/sqlite-utils/issues/449 | IC_kwDOCGYnMM5F-jtH | davidleejy 1690072 | 2022-07-04T17:33:04Z | 2022-07-04T17:48:43Z | CONTRIBUTOR | I've written the code and test. Would you be able to advise how to compare table columns in a pytest function properly? Experiencing a challenge when comparing columns. Test:
Result: Failure is due to column types being named differently -- e.g. 'FLOAT' vs 'REAL', 'INTEGER' vs 'INT'. How should I go about comparing columns while accounting for equivalent types? Or did I miss out something in my duplication code correctly? Here's how I did it: in
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Utilities for duplicating tables and creating a table with the results of a query 1279863844 | |
1297703307 | https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1297703307 | https://api.github.com/repos/simonw/sqlite-utils/issues/448 | IC_kwDOCGYnMM5NWWGL | mcarpenter 167893 | 2022-10-31T21:23:51Z | 2022-10-31T21:27:32Z | CONTRIBUTOR | The Windows aspect is a red herring: OP's sample above produces the same error on Linux. (Though I don't know what's going on with the CI). The same error can also be obtained by passing an The fix for my case is easy: open the file in mode Minimal test case (derived from utils.py): ``` python import io from typing import cast fp = io.StringIO("id,name\n1,Cleo") # errorfp = io.BytesIO(bytes("id,name\n1,Cleo", encoding='utf-8')) # okay
reader = io.BufferedReader(cast(io.RawIOBase, fp))
reader.peek(1) # exception thrown here
Some thoughts on testing binary-ness of |
{ "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto' 1279144769 | |
1747231893 | https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1747231893 | https://api.github.com/repos/simonw/sqlite-utils/issues/433 | IC_kwDOCGYnMM5oJKSV | spookylukey 62745 | 2023-10-04T16:15:09Z | 2023-10-04T16:28:21Z | CONTRIBUTOR | I confirm the bug, as above, and that @jonafato 's patch fixes it for me. However, it's not the right fix. The problem is that ProgressBar is being used in the wrong way. This also results in two lines being printed instead of one, like this:
```json { "version": "0.2.0", "configurations": [ { "name": "Python: Module", "type": "python", "request": "launch", "module": "sqlite_utils", "justMyCode": false, "args": ["insert", "test.db", "test", "--csv", "tests/sniff/example1.csv"] } ] } ``` [edit - deleted my analysis of why the current code is wrong, which was confused and confusing] |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
CLI eats my cursor 1239034903 | |
1444474487 | https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1444474487 | https://api.github.com/repos/simonw/sqlite-utils/issues/433 | IC_kwDOCGYnMM5WGO53 | mcarpenter 167893 | 2023-02-24T20:57:43Z | 2023-02-24T22:22:18Z | CONTRIBUTOR | I think I see what is happening here, although I haven't quite work out a fix yet. Usually:
(See terminal escape sequences However the sqlite-utils
The yielded |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
CLI eats my cursor 1239034903 | |
1252898131 | https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1252898131 | https://api.github.com/repos/simonw/sqlite-utils/issues/433 | IC_kwDOCGYnMM5KrbVT | chapmanjacobd 7908073 | 2022-09-20T20:51:21Z | 2022-09-20T20:56:07Z | CONTRIBUTOR | When I run https://linux.die.net/man/1/reset
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
CLI eats my cursor 1239034903 | |
1189010812 | https://github.com/simonw/sqlite-utils/issues/423#issuecomment-1189010812 | https://api.github.com/repos/simonw/sqlite-utils/issues/423 | IC_kwDOCGYnMM5G3t18 | fgregg 536941 | 2022-07-19T12:47:39Z | 2022-07-19T12:47:39Z | CONTRIBUTOR | just ran into this! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.extract() doesn't set foreign key when extracted columns contain NULL value 1199158210 | |
1059647114 | https://github.com/simonw/sqlite-utils/issues/412#issuecomment-1059647114 | https://api.github.com/repos/simonw/sqlite-utils/issues/412 | IC_kwDOCGYnMM4_KO6K | eyeseast 25778 | 2022-03-05T01:54:24Z | 2022-03-05T01:54:24Z | CONTRIBUTOR | I haven't tried this, but it looks like Pandas has a method for this: https://pandas.pydata.org/docs/reference/api/pandas.read_sql_query.html |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Optional Pandas integration 1160182768 | |
1065477258 | https://github.com/simonw/sqlite-utils/issues/411#issuecomment-1065477258 | https://api.github.com/repos/simonw/sqlite-utils/issues/411 | IC_kwDOCGYnMM4_geSK | eyeseast 25778 | 2022-03-11T20:14:59Z | 2022-03-11T20:14:59Z | CONTRIBUTOR | Good call on adding this to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support for generated columns 1160034488 | |
1264223554 | https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223554 | https://api.github.com/repos/simonw/sqlite-utils/issues/409 | IC_kwDOCGYnMM5LWoVC | chapmanjacobd 7908073 | 2022-10-01T03:42:50Z | 2022-10-01T03:42:50Z | CONTRIBUTOR | oh weird. it inserts into db2 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`with db:` for transactions 1149661489 | |
1264223363 | https://github.com/simonw/sqlite-utils/issues/409#issuecomment-1264223363 | https://api.github.com/repos/simonw/sqlite-utils/issues/409 | IC_kwDOCGYnMM5LWoSD | chapmanjacobd 7908073 | 2022-10-01T03:41:45Z | 2022-10-01T03:41:45Z | CONTRIBUTOR | ``` pytest xklb/check.py --pdb xklb/check.py:11: in test_transaction assert list(db2["t"].rows) == [] E AssertionError: assert [{'foo': 1}] == [] E + where [{'foo': 1}] = list(<generator object Queryable.rows_where at 0x7f2d84d1f0d0>) E + where <generator object Queryable.rows_where at 0x7f2d84d1f0d0> = <Table t (foo)>.rows
It fails because it is already inserted. btw if you put these two lines in you pyproject.toml you can get
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`with db:` for transactions 1149661489 | |
1033332570 | https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1033332570 | https://api.github.com/repos/simonw/sqlite-utils/issues/403 | IC_kwDOCGYnMM49l2da | fgregg 536941 | 2022-02-09T04:22:43Z | 2022-02-09T04:22:43Z | CONTRIBUTOR | dddoooope |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Document how to add a primary key to a rowid table using `sqlite-utils transform --pk` 1126692066 | |
1032126353 | https://github.com/simonw/sqlite-utils/issues/403#issuecomment-1032126353 | https://api.github.com/repos/simonw/sqlite-utils/issues/403 | IC_kwDOCGYnMM49hP-R | fgregg 536941 | 2022-02-08T01:45:15Z | 2022-02-08T01:45:31Z | CONTRIBUTOR | you can hack something like this to achieve this result:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Document how to add a primary key to a rowid table using `sqlite-utils transform --pk` 1126692066 | |
1035057014 | https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1035057014 | https://api.github.com/repos/simonw/sqlite-utils/issues/402 | IC_kwDOCGYnMM49sbd2 | eyeseast 25778 | 2022-02-10T15:30:28Z | 2022-02-10T15:30:40Z | CONTRIBUTOR | Yeah, the CLI experience is probably where any kind of multi-column, configured setup is going to fall apart. Sticking with GIS examples, one way I might think about this is using the fiona CLI: ```sh assuming a database is already created and has SpatiaLitefio cat boundary.shp | sqlite-utils insert boundaries --conversion geometry GeometryGeoJSON - ``` Anyway, very interested to see where you land here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Advanced class-based `conversions=` mechanism 1125297737 | |
1032732242 | https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1032732242 | https://api.github.com/repos/simonw/sqlite-utils/issues/402 | IC_kwDOCGYnMM49jj5S | eyeseast 25778 | 2022-02-08T15:26:59Z | 2022-02-08T15:26:59Z | CONTRIBUTOR | What if you did something like this: ```python class Conversion: def init(self, args, *kwargs): "Put whatever settings you need here"
``` This way, you're always passing an instance, which has methods that do the conversion. (Or you're passing a SQL string, as you would now.) The You'd then use it like this: ```python subclass might be unneeded here, if methods are presentclass LngLatConversion(Conversion): def init(self, x="longitude", y="latitude"): self.x = x self.y = y
table.insert_all(rows, conversions={"point": LngLatConversion("lng", "lat"))} ``` I haven't thought through all the implementation details here, and it'll probably break in ways I haven't foreseen, but wanted to get this idea out of my head. Hope it helps. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Advanced class-based `conversions=` mechanism 1125297737 | |
1031791783 | https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031791783 | https://api.github.com/repos/simonw/sqlite-utils/issues/402 | IC_kwDOCGYnMM49f-Sn | eyeseast 25778 | 2022-02-07T18:37:40Z | 2022-02-07T18:37:40Z | CONTRIBUTOR | I've never used it either, but it's interesting, right? Feel like I should try it for something. I'm trying to get my head around how this conversions feature might work, because I really like the idea of it. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Advanced class-based `conversions=` mechanism 1125297737 | |
1031779460 | https://github.com/simonw/sqlite-utils/issues/402#issuecomment-1031779460 | https://api.github.com/repos/simonw/sqlite-utils/issues/402 | IC_kwDOCGYnMM49f7SE | eyeseast 25778 | 2022-02-07T18:24:56Z | 2022-02-07T18:24:56Z | CONTRIBUTOR | I wonder if there's any overlap with the goals here and the I'm not sure that's exactly what we're talking about here, but it might be a parallel with some useful ideas to borrow. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Advanced class-based `conversions=` mechanism 1125297737 | |
1077671779 | https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1077671779 | https://api.github.com/repos/simonw/sqlite-utils/issues/399 | IC_kwDOCGYnMM5AO_dj | eyeseast 25778 | 2022-03-24T14:11:33Z | 2022-03-24T14:11:43Z | CONTRIBUTOR | Coming back to this. I was about to add a utility function to datasette-geojson to convert lat/lng columns to geometries. Thankfully I googled first. There's a SpatiaLite function for this: MakePoint.
I'm not sure if that would work with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make it easier to insert geometries, with documentation and maybe code 1124731464 | |
1030741289 | https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030741289 | https://api.github.com/repos/simonw/sqlite-utils/issues/399 | IC_kwDOCGYnMM49b90p | eyeseast 25778 | 2022-02-06T03:03:43Z | 2022-02-06T03:03:43Z | CONTRIBUTOR |
Off the top of my head:
Some of this is easy enough with SQL functions, some is easier in Python. Maybe that's where having pre-built classes gets really handy, because it saves you from thinking about which way it's implemented. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make it easier to insert geometries, with documentation and maybe code 1124731464 | |
1030740826 | https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740826 | https://api.github.com/repos/simonw/sqlite-utils/issues/399 | IC_kwDOCGYnMM49b9ta | eyeseast 25778 | 2022-02-06T02:59:10Z | 2022-02-06T02:59:10Z | CONTRIBUTOR | All this said, I don't think it's unreasonable to point people to dedicated tools like |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make it easier to insert geometries, with documentation and maybe code 1124731464 | |
1030740653 | https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1030740653 | https://api.github.com/repos/simonw/sqlite-utils/issues/399 | IC_kwDOCGYnMM49b9qt | eyeseast 25778 | 2022-02-06T02:57:17Z | 2022-02-06T02:57:17Z | CONTRIBUTOR | I like the idea of having stock conversions you could import. I'd actually move them to a dedicated module (call it ```python from sqlite_utils.conversions import LongitudeLatitude db["places"].insert( { "name": "London", "lng": -0.118092, "lat": 51.509865, }, conversions={"point": LongitudeLatitude("lng", "lat")}, ) ``` I would definitely use that for every CSV I get with lat/lng columns where I actually need GeoJSON. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make it easier to insert geometries, with documentation and maybe code 1124731464 | |
1038336591 | https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1038336591 | https://api.github.com/repos/simonw/sqlite-utils/issues/398 | IC_kwDOCGYnMM4948JP | eyeseast 25778 | 2022-02-13T18:48:21Z | 2022-02-13T18:49:49Z | CONTRIBUTOR | Been chipping away at this between other things and realized I think it's probably worth keeping a SpatiaLite flag on |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add SpatiaLite helpers to CLI 1124237013 | |
1030629879 | https://github.com/simonw/sqlite-utils/issues/398#issuecomment-1030629879 | https://api.github.com/repos/simonw/sqlite-utils/issues/398 | IC_kwDOCGYnMM49bin3 | eyeseast 25778 | 2022-02-05T13:57:33Z | 2022-02-05T19:49:38Z | CONTRIBUTOR | I'm mostly using geojson-to-sqlite at the moment. Even with shapefiles, I'm usually converting to GeoJSON and projecting to EPSG:4326 (with ogr2ogr) first. I think an open question here is how much you want to leave to external libraries and how much you want here. My thinking has been that adding Spatialite helpers here would make external stuff easier, but it would be nice to have some standard way to insert geometries. I'm in the middle of adding GeoJSON and Spatialite support to geocode-sqlite, and that will probably use WKT. Since that's all points, I think I can just make the string inline. But for polygons, I'd generally use Shapely, which probably isn't a dependency you want to add to sqlite-utils. I've also been trying to get some of the approaches here to work, but haven't had any success so far. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add SpatiaLite helpers to CLI 1124237013 | |
1009548580 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1009548580 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48LH0k | fgregg 536941 | 2022-01-11T02:43:34Z | 2022-01-11T02:43:34Z | CONTRIBUTOR | thanks so much! always a pleasure to see how you work through these things |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
create-index should run analyze after creating index 1096558279 | |
1008275546 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008275546 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48GRBa | fgregg 536941 | 2022-01-09T11:01:15Z | 2022-01-09T13:37:51Z | CONTRIBUTOR | i don’t want to be such a partisan for analyze, but the query planner deciding not to use an index based on information collected by analyze is not necessarily a bug, but could be the correct choice. <s>the original poster in that stack overflow doesn’t say there’s a performance regression </s> |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
create-index should run analyze after creating index 1096558279 | |
1008166084 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008166084 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48F2TE | fgregg 536941 | 2022-01-08T22:32:47Z | 2022-01-08T22:32:47Z | CONTRIBUTOR | or using “ pragma optimize” |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
create-index should run analyze after creating index 1096558279 | |
1008164786 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164786 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48F1-y | fgregg 536941 | 2022-01-08T22:24:19Z | 2022-01-08T22:24:19Z | CONTRIBUTOR | the out-of-date scenario you describe could be addressed by automatically adding an analyze to the insert or convert commands if they implicate an index |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
create-index should run analyze after creating index 1096558279 | |
1008164116 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008164116 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48F10U | fgregg 536941 | 2022-01-08T22:18:57Z | 2022-01-08T22:18:57Z | CONTRIBUTOR | the table with the query ran so bad was about 50k. i think the scenario should not be worse than no stats. i also did not know that sqlite was so different from postgres and needed an explicit analyze call. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
create-index should run analyze after creating index 1096558279 | |
1008161965 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1008161965 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48F1St | fgregg 536941 | 2022-01-08T22:02:56Z | 2022-01-08T22:02:56Z | CONTRIBUTOR | for options 2 and 3, i would worry about discoverablity. in other db’s it is not necessary to explicitly call analyze for most indices. ie for postgres
i suppose i would propose raising a warning if the stats table is created that explains what is going on and informs users about a —no-analyze argument. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
create-index should run analyze after creating index 1096558279 | |
1007636709 | https://github.com/simonw/sqlite-utils/issues/365#issuecomment-1007636709 | https://api.github.com/repos/simonw/sqlite-utils/issues/365 | IC_kwDOCGYnMM48D1Dl | fgregg 536941 | 2022-01-07T18:28:33Z | 2022-01-07T18:29:43Z | CONTRIBUTOR | i added an index to one table with sqlite-utils, and then a query that used to take about 1 second started taking hundreds of seconds. running analyze got me back to sub second speed. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
create-index should run analyze after creating index 1096558279 | |
991405755 | https://github.com/simonw/sqlite-utils/issues/353#issuecomment-991405755 | https://api.github.com/repos/simonw/sqlite-utils/issues/353 | IC_kwDOCGYnMM47F6a7 | fgregg 536941 | 2021-12-11T01:38:29Z | 2021-12-11T01:38:29Z | CONTRIBUTOR | wow! that's awesome! thanks so much, @simonw! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Allow passing a file of code to "sqlite-utils convert" 1077102934 | |
983155079 | https://github.com/simonw/sqlite-utils/issues/348#issuecomment-983155079 | https://api.github.com/repos/simonw/sqlite-utils/issues/348 | IC_kwDOCGYnMM46mcGH | eyeseast 25778 | 2021-12-01T00:28:40Z | 2021-12-01T00:28:40Z | CONTRIBUTOR | I'd use this. Right now, I tend to do |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for creating an empty database 1067771698 | |
1815825863 | https://github.com/simonw/sqlite-utils/issues/344#issuecomment-1815825863 | https://api.github.com/repos/simonw/sqlite-utils/issues/344 | IC_kwDOCGYnMM5sO03H | tkhattra 16437338 | 2023-11-17T06:44:49Z | 2023-11-17T06:44:49Z | CONTRIBUTOR | hello Simon, I've added more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776 in changeset https://github.com/simonw/sqlite-utils/commit/e4b9b582cdb4e48430865f8739f341bc8017c1e4. It also fixes table.transform() to preserve STRICT mode. Please pull if you deem appropriate. Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support STRICT tables 1066474200 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30