issue_comments
8,883 rows where user = 9599 sorted by updated_at descending
This data as json, CSV (advanced)
user 1
- simonw · 5,150 ✖
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
866184260 | https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866184260 | https://api.github.com/repos/simonw/sqlite-utils/issues/267 | MDEyOklzc3VlQ29tbWVudDg2NjE4NDI2MA== | simonw 9599 | 2021-06-22T17:26:18Z | 2021-06-22T17:27:27Z | OWNER | If an |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
row.update() or row.pk 915421499 | |
866182655 | https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866182655 | https://api.github.com/repos/simonw/sqlite-utils/issues/267 | MDEyOklzc3VlQ29tbWVudDg2NjE4MjY1NQ== | simonw 9599 | 2021-06-22T17:24:03Z | 2021-06-22T17:24:03Z | OWNER | I'm re-opening this as a research task because it may be possible to cleanly implement this using a Since this would just be for adding methods (and maybe a property for returning the primary keys for a row) the usual disadvantages of subclassing One catch: dictionaries already have a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
row.update() or row.pk 915421499 | |
865510796 | https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865510796 | https://api.github.com/repos/simonw/sqlite-utils/issues/290 | MDEyOklzc3VlQ29tbWVudDg2NTUxMDc5Ng== | simonw 9599 | 2021-06-22T04:04:40Z | 2021-06-22T04:04:48Z | OWNER | Still needs documentation, which will involve rewriting the whole Executing queries section. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`db.query()` method (renamed `db.execute_returning_dicts()`) 926777310 | |
865497846 | https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865497846 | https://api.github.com/repos/simonw/sqlite-utils/issues/290 | MDEyOklzc3VlQ29tbWVudDg2NTQ5Nzg0Ng== | simonw 9599 | 2021-06-22T03:21:38Z | 2021-06-22T03:21:38Z | OWNER | The Python docs say: https://docs.python.org/3/library/sqlite3.html
Looking at the C source code, both So it looks like the most efficient way to iterate over a cursor may well be |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`db.query()` method (renamed `db.execute_returning_dicts()`) 926777310 | |
865495370 | https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865495370 | https://api.github.com/repos/simonw/sqlite-utils/issues/290 | MDEyOklzc3VlQ29tbWVudDg2NTQ5NTM3MA== | simonw 9599 | 2021-06-22T03:14:30Z | 2021-06-22T03:14:30Z | OWNER | One small problem with the existing method: https://github.com/simonw/sqlite-utils/blob/8cedc6a8b29180e68326f6b76f249d5e39e4b591/sqlite_utils/db.py#L362-L365 It returns a full list, but what if the user would rather have a generator they can iterate over without loading the results into memory in one go? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`db.query()` method (renamed `db.execute_returning_dicts()`) 926777310 | |
865491922 | https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865491922 | https://api.github.com/repos/simonw/sqlite-utils/issues/290 | MDEyOklzc3VlQ29tbWVudDg2NTQ5MTkyMg== | simonw 9599 | 2021-06-22T03:05:35Z | 2021-06-22T03:05:35Z | OWNER | Potential names:
I think |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`db.query()` method (renamed `db.execute_returning_dicts()`) 926777310 | |
865160132 | https://github.com/simonw/datasette/pull/1368#issuecomment-865160132 | https://api.github.com/repos/simonw/datasette/issues/1368 | MDEyOklzc3VlQ29tbWVudDg2NTE2MDEzMg== | simonw 9599 | 2021-06-21T16:07:06Z | 2021-06-21T16:08:48Z | OWNER | A few tests failed - Black, the test that checks the docs mention the new hook - the most interesting failing test looks like this one: ``` updated_metadata["databases"]["fixtures"]["queries"]["magic_parameters"][ "allow" ] = (allow if "query" in permissions else deny)
This test is directly manipulating |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
DRAFT: A new plugin hook for dynamic metadata 913865304 | |
864609271 | https://github.com/simonw/sqlite-utils/issues/289#issuecomment-864609271 | https://api.github.com/repos/simonw/sqlite-utils/issues/289 | MDEyOklzc3VlQ29tbWVudDg2NDYwOTI3MQ== | simonw 9599 | 2021-06-20T20:42:07Z | 2021-06-20T20:42:07Z | OWNER | Wow, thank you! I didn't know about |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mypy fixes for rows_from_file() 925677191 | |
864594956 | https://github.com/simonw/sqlite-utils/issues/286#issuecomment-864594956 | https://api.github.com/repos/simonw/sqlite-utils/issues/286 | MDEyOklzc3VlQ29tbWVudDg2NDU5NDk1Ng== | simonw 9599 | 2021-06-20T18:38:05Z | 2021-06-20T18:38:05Z | OWNER | 3.10 is out in Homebrew now (they turn that around so fast): https://formulae.brew.sh/formula/sqlite-utils |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add installation instructions 925487946 | |
864480051 | https://github.com/simonw/datasette/issues/1382#issuecomment-864480051 | https://api.github.com/repos/simonw/datasette/issues/1382 | MDEyOklzc3VlQ29tbWVudDg2NDQ4MDA1MQ== | simonw 9599 | 2021-06-20T00:20:06Z | 2021-06-20T00:21:02Z | OWNER | Yes you can - thanks for pointing this out, I've added a comment to the
So if you edit that file in your own project and change the line to this:
It should fix this for you. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding? 925406964 | |
864476167 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864476167 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2NDQ3NjE2Nw== | simonw 9599 | 2021-06-19T23:36:48Z | 2021-06-19T23:36:48Z | OWNER | Wrote this up on my blog here: https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/ - with a video demo here: https://www.youtube.com/watch?v=OUjd0rkc678 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
864419283 | https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864419283 | https://api.github.com/repos/simonw/sqlite-utils/issues/284 | MDEyOklzc3VlQ29tbWVudDg2NDQxOTI4Mw== | simonw 9599 | 2021-06-19T15:15:34Z | 2021-06-19T15:15:34Z | OWNER | I think this code is at fault: https://github.com/simonw/sqlite-utils/blob/5b257949d996fe43dc5d218d4308b88796a90740/sqlite_utils/db.py#L1017-L1023 It's using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.transform(types=) turns rowid into a concrete column 925320167 | |
864418795 | https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418795 | https://api.github.com/repos/simonw/sqlite-utils/issues/285 | MDEyOklzc3VlQ29tbWVudDg2NDQxODc5NQ== | simonw 9599 | 2021-06-19T15:11:05Z | 2021-06-19T15:11:14Z | OWNER | Actually I'm going to go with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introspection property for telling if a table is a rowid table 925410305 | |
864418188 | https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418188 | https://api.github.com/repos/simonw/sqlite-utils/issues/285 | MDEyOklzc3VlQ29tbWVudDg2NDQxODE4OA== | simonw 9599 | 2021-06-19T15:05:53Z | 2021-06-19T15:05:53Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introspection property for telling if a table is a rowid table 925410305 | |
864417808 | https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417808 | https://api.github.com/repos/simonw/sqlite-utils/issues/285 | MDEyOklzc3VlQ29tbWVudDg2NDQxNzgwOA== | simonw 9599 | 2021-06-19T15:03:00Z | 2021-06-19T15:03:00Z | OWNER | I think I like |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introspection property for telling if a table is a rowid table 925410305 | |
864417765 | https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417765 | https://api.github.com/repos/simonw/sqlite-utils/issues/285 | MDEyOklzc3VlQ29tbWVudDg2NDQxNzc2NQ== | simonw 9599 | 2021-06-19T15:02:42Z | 2021-06-19T15:02:42Z | OWNER | Some options:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introspection property for telling if a table is a rowid table 925410305 | |
864417493 | https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417493 | https://api.github.com/repos/simonw/sqlite-utils/issues/285 | MDEyOklzc3VlQ29tbWVudDg2NDQxNzQ5Mw== | simonw 9599 | 2021-06-19T15:00:43Z | 2021-06-19T15:00:43Z | OWNER | I have to be careful about the language I use here. Here's the official definition: https://www.sqlite.org/rowidtable.html
So it's not correct to call a table a "rowid table" only if it is missing its own primary keys. Maybe |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introspection property for telling if a table is a rowid table 925410305 | |
864417133 | https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417133 | https://api.github.com/repos/simonw/sqlite-utils/issues/285 | MDEyOklzc3VlQ29tbWVudDg2NDQxNzEzMw== | simonw 9599 | 2021-06-19T14:57:36Z | 2021-06-19T14:57:36Z | OWNER | So the logic is:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introspection property for telling if a table is a rowid table 925410305 | |
864417031 | https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417031 | https://api.github.com/repos/simonw/sqlite-utils/issues/285 | MDEyOklzc3VlQ29tbWVudDg2NDQxNzAzMQ== | simonw 9599 | 2021-06-19T14:56:45Z | 2021-06-19T14:56:45Z | OWNER | ```pycon <Table rowid_table (name)> >>> db["regular_table"].insert({"id": 1, "name": "Cleo"}, pk="id") <Table regular_table (id, name)> >>> db["rowid_table"].pks ['rowid'] >>> db["regular_table"].pks ['id'] ``` But that's because the `.pks` property hides the difference: https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Introspection property for telling if a table is a rowid table 925410305 | |
864416911 | https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416911 | https://api.github.com/repos/simonw/sqlite-utils/issues/284 | MDEyOklzc3VlQ29tbWVudDg2NDQxNjkxMQ== | simonw 9599 | 2021-06-19T14:55:45Z | 2021-06-19T14:55:45Z | OWNER | So I can indeed detect a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.transform(types=) turns rowid into a concrete column 925320167 | |
864416785 | https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785 | https://api.github.com/repos/simonw/sqlite-utils/issues/284 | MDEyOklzc3VlQ29tbWVudDg2NDQxNjc4NQ== | simonw 9599 | 2021-06-19T14:54:41Z | 2021-06-19T14:54:41Z | OWNER | ```pycon <Table rowid_table (name)> >>> db["regular_table"].insert({"id": 1, "name": "Cleo"}, pk="id") <Table regular_table (id, name)> >>> db["rowid_table"].pks ['rowid'] >>> db["regular_table"].pks ['id'] ``` I think I need an introspection property for working out if a table is a `rowid` table or not. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.transform(types=) turns rowid into a concrete column 925320167 | |
864416086 | https://github.com/simonw/sqlite-utils/issues/283#issuecomment-864416086 | https://api.github.com/repos/simonw/sqlite-utils/issues/283 | MDEyOklzc3VlQ29tbWVudDg2NDQxNjA4Ng== | simonw 9599 | 2021-06-19T14:49:06Z | 2021-06-19T14:49:13Z | OWNER | Once again, this is difficult because of the use of a generator here - I'm going to change |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
memory: Shouldn't detect types for JSON 925319214 | |
864358951 | https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358951 | https://api.github.com/repos/simonw/sqlite-utils/issues/284 | MDEyOklzc3VlQ29tbWVudDg2NDM1ODk1MQ== | simonw 9599 | 2021-06-19T05:30:00Z | 2021-06-19T05:30:00Z | OWNER | If this can be fixed it will be in the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.transform(types=) turns rowid into a concrete column 925320167 | |
864358680 | https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358680 | https://api.github.com/repos/simonw/sqlite-utils/issues/284 | MDEyOklzc3VlQ29tbWVudDg2NDM1ODY4MA== | simonw 9599 | 2021-06-19T05:27:13Z | 2021-06-19T05:27:13Z | OWNER | How easy is it to detect a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.transform(types=) turns rowid into a concrete column 925320167 | |
864354627 | https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864354627 | https://api.github.com/repos/simonw/sqlite-utils/issues/282 | MDEyOklzc3VlQ29tbWVudDg2NDM1NDYyNw== | simonw 9599 | 2021-06-19T04:42:03Z | 2021-06-19T04:42:03Z | OWNER | Demo:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Automatic type detection for CSV data 925305186 | |
864350407 | https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864350407 | https://api.github.com/repos/simonw/sqlite-utils/issues/282 | MDEyOklzc3VlQ29tbWVudDg2NDM1MDQwNw== | simonw 9599 | 2021-06-19T03:52:20Z | 2021-06-19T03:52:20Z | OWNER | I'll have an environment variable for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Automatic type detection for CSV data 925305186 | |
864349123 | https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864349123 | https://api.github.com/repos/simonw/sqlite-utils/issues/282 | MDEyOklzc3VlQ29tbWVudDg2NDM0OTEyMw== | simonw 9599 | 2021-06-19T03:36:54Z | 2021-06-19T03:36:54Z | OWNER | I may change the default for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Automatic type detection for CSV data 925305186 | |
864349066 | https://github.com/simonw/sqlite-utils/issues/179#issuecomment-864349066 | https://api.github.com/repos/simonw/sqlite-utils/issues/179 | MDEyOklzc3VlQ29tbWVudDg2NDM0OTA2Ng== | simonw 9599 | 2021-06-19T03:36:04Z | 2021-06-19T03:36:04Z | OWNER | This work is going to happen in #282. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils transform/insert --detect-types 709577625 | |
864348954 | https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864348954 | https://api.github.com/repos/simonw/sqlite-utils/issues/282 | MDEyOklzc3VlQ29tbWVudDg2NDM0ODk1NA== | simonw 9599 | 2021-06-19T03:34:42Z | 2021-06-19T03:35:46Z | OWNER | I built some prototype code here for something which looks at every row in a CSV import and records the likely types: https://gist.github.com/simonw/465f9356f175d1cf86957947dff501d4 This could be used by the command-line tools to figure out what This is a different approach to the pure SQL version I tried building in https://github.com/simonw/sqlite-utils/issues/179 - I think this is a better approach though, it's less prone to weird idiosyncrasies of SQLite types, and it's also easy for us to add on to the existing CSV import code in a way that won't require scanning the data twice. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Automatic type detection for CSV data 925305186 | |
864330508 | https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864330508 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | MDEyOklzc3VlQ29tbWVudDg2NDMzMDUwOA== | simonw 9599 | 2021-06-19T00:34:24Z | 2021-06-19T00:34:24Z | OWNER | Got this working:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory should handle TSV and JSON in addition to CSV 924990677 | |
864328927 | https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | MDEyOklzc3VlQ29tbWVudDg2NDMyODkyNw== | simonw 9599 | 2021-06-19T00:25:08Z | 2021-06-19T00:25:17Z | OWNER | I tried writing this function with type hints, but eventually gave up:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory should handle TSV and JSON in addition to CSV 924990677 | |
864323438 | https://github.com/simonw/sqlite-utils/issues/281#issuecomment-864323438 | https://api.github.com/repos/simonw/sqlite-utils/issues/281 | MDEyOklzc3VlQ29tbWVudDg2NDMyMzQzOA== | simonw 9599 | 2021-06-18T23:55:06Z | 2021-06-18T23:55:06Z | OWNER | The I'm going to do |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory 924992318 | |
864208476 | https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng== | simonw 9599 | 2021-06-18T18:30:08Z | 2021-06-18T23:30:19Z | OWNER | So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory should handle TSV and JSON in addition to CSV 924990677 | |
864207841 | https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ== | simonw 9599 | 2021-06-18T18:28:40Z | 2021-06-18T18:28:46Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory should handle TSV and JSON in addition to CSV 924990677 | |
864206308 | https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA== | simonw 9599 | 2021-06-18T18:25:04Z | 2021-06-18T18:25:04Z | OWNER | Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it. I still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too? This does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway. So I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory should handle TSV and JSON in addition to CSV 924990677 | |
864129273 | https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw== | simonw 9599 | 2021-06-18T15:47:47Z | 2021-06-18T15:47:47Z | OWNER | Detecting valid JSON is tricky - just because a stream starts with One way to solve this would be with a custom state machine. Another would be to use the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory should handle TSV and JSON in addition to CSV 924990677 | |
864128489 | https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864128489 | https://api.github.com/repos/simonw/sqlite-utils/issues/278 | MDEyOklzc3VlQ29tbWVudDg2NDEyODQ4OQ== | simonw 9599 | 2021-06-18T15:46:24Z | 2021-06-18T15:46:24Z | OWNER | A workaround could be to define a bash or zsh alias of some sort. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support db as first parameter before subcommand, or as environment variable 923697888 | |
864126781 | https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864126781 | https://api.github.com/repos/simonw/sqlite-utils/issues/278 | MDEyOklzc3VlQ29tbWVudDg2NDEyNjc4MQ== | simonw 9599 | 2021-06-18T15:43:19Z | 2021-06-18T15:43:19Z | OWNER | I don't think it's possible to do this without breaking backwards compatibility, unfortunately. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support db as first parameter before subcommand, or as environment variable 923697888 | |
864103005 | https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005 | https://api.github.com/repos/simonw/sqlite-utils/issues/279 | MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ== | simonw 9599 | 2021-06-18T15:04:15Z | 2021-06-18T15:04:15Z | OWNER | To detect JSON, check to see if the stream starts with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory should handle TSV and JSON in addition to CSV 924990677 | |
864101267 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864101267 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2NDEwMTI2Nw== | simonw 9599 | 2021-06-18T15:01:41Z | 2021-06-18T15:01:41Z | OWNER | I'll split the remaining work out into separate issues. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
864099764 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA== | simonw 9599 | 2021-06-18T14:59:27Z | 2021-06-18T14:59:27Z | OWNER | I'm going to merge this as-is and work on the JSON/TSV support in a separate issue. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
864092515 | https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515 | https://api.github.com/repos/simonw/sqlite-utils/issues/277 | MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ== | simonw 9599 | 2021-06-18T14:47:57Z | 2021-06-18T14:47:57Z | OWNER | This is a neat improvement. |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
add -h support closes #276 923612361 | |
862617165 | https://github.com/simonw/sqlite-utils/issues/275#issuecomment-862617165 | https://api.github.com/repos/simonw/sqlite-utils/issues/275 | MDEyOklzc3VlQ29tbWVudDg2MjYxNzE2NQ== | simonw 9599 | 2021-06-16T18:34:51Z | 2021-06-16T18:34:51Z | OWNER | Also use this: https://github.com/simonw/datasette/blob/83e9c8bc7585dcc62f200e37c2daefcd669ee05e/codecov.yml And add a badge, as seen on https://github.com/simonw/asgi-csrf |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Enable code coverage 922955697 | |
862605436 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg== | simonw 9599 | 2021-06-16T18:19:05Z | 2021-06-16T18:19:05Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory command for directly querying CSV/JSON data 922099793 | ||
862494864 | https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862494864 | https://api.github.com/repos/simonw/sqlite-utils/issues/267 | MDEyOklzc3VlQ29tbWVudDg2MjQ5NDg2NA== | simonw 9599 | 2021-06-16T15:51:28Z | 2021-06-16T16:26:15Z | OWNER | I did add a slightly clumsy mechanism recently to help a bit here though: the More details in the issue for that feature: #240 The idea here is that if you want to call update you need the primary key for the row - so you can do this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
row.update() or row.pk 915421499 | |
862495803 | https://github.com/simonw/sqlite-utils/issues/131#issuecomment-862495803 | https://api.github.com/repos/simonw/sqlite-utils/issues/131 | MDEyOklzc3VlQ29tbWVudDg2MjQ5NTgwMw== | simonw 9599 | 2021-06-16T15:52:33Z | 2021-06-16T15:52:33Z | OWNER | I like |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils insert: options for column types 675753042 | |
862493179 | https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862493179 | https://api.github.com/repos/simonw/sqlite-utils/issues/267 | MDEyOklzc3VlQ29tbWVudDg2MjQ5MzE3OQ== | simonw 9599 | 2021-06-16T15:49:13Z | 2021-06-16T15:49:13Z | OWNER | The big challenge here is that the rows returned by this library aren't objects, they are Python dictionaries - so adding methods to them isn't possible without changing the type that is returned by these methods. Part of the philosophy of the library is that it should make it as easy as possible to round-trip between Python dictionaries and SQLite table data, so I don't think adding methods like this is going to fit. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
row.update() or row.pk 915421499 | |
862491721 | https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862491721 | https://api.github.com/repos/simonw/sqlite-utils/issues/270 | MDEyOklzc3VlQ29tbWVudDg2MjQ5MTcyMQ== | simonw 9599 | 2021-06-16T15:47:06Z | 2021-06-16T15:47:06Z | OWNER | SQLite doesn't have a JSON column type - it has JSON processing functions, but they operate against TEXT columns - so there's nothing I can do here unfortunately. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Cannot set type JSON 919314806 | |
862491016 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862491016 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjQ5MTAxNg== | simonw 9599 | 2021-06-16T15:46:13Z | 2021-06-16T15:46:13Z | OWNER | Columns from data imported from CSV in this way is currently treated as |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
862485408 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862485408 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjQ4NTQwOA== | simonw 9599 | 2021-06-16T15:38:58Z | 2021-06-16T15:39:28Z | OWNER | Also And it helps emphasize that the file you are querying will be loaded into memory, so probably don't try this against a 1GB CSV file. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
862484557 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862484557 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjQ4NDU1Nw== | simonw 9599 | 2021-06-16T15:37:51Z | 2021-06-16T15:38:34Z | OWNER | I wonder if there's a better name for this than
I think |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
862479704 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862479704 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjQ3OTcwNA== | simonw 9599 | 2021-06-16T15:31:31Z | 2021-06-16T15:31:31Z | OWNER | Plus, could I make this change to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
862478881 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862478881 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjQ3ODg4MQ== | simonw 9599 | 2021-06-16T15:30:24Z | 2021-06-16T15:30:24Z | OWNER | But... |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
862475685 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862475685 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjQ3NTY4NQ== | simonw 9599 | 2021-06-16T15:26:19Z | 2021-06-16T15:29:38Z | OWNER | Here's a radical idea: what if I combined The trick here would be to detect if the arguments passed on the command-line refer to SQLite databases or if they refer to CSV/JSON data that should be imported into temporary tables. Detecting a SQLite database file is actually really easy - they all start with the same binary string: ```pycon
So then what would the semantics of
The complexity here is definitely in the handling of a combination of SQLite database files and CSV filenames. Also, I'm not 100% sold on this as being better than having a separate |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
862046009 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ== | simonw 9599 | 2021-06-16T05:15:38Z | 2021-06-16T05:15:38Z | OWNER | I'm going to add a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862045639 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ== | simonw 9599 | 2021-06-16T05:14:38Z | 2021-06-16T05:14:38Z | OWNER | Can't share much code though since a bunch of that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862045438 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA== | simonw 9599 | 2021-06-16T05:14:00Z | 2021-06-16T05:14:00Z | OWNER | I should probably refactor the CSV/JSON/loading stuff into a function in |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862043974 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA== | simonw 9599 | 2021-06-16T05:10:12Z | 2021-06-16T05:10:12Z | OWNER | I can stop promoting |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862042110 | https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110 | https://api.github.com/repos/simonw/sqlite-utils/issues/273 | MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA== | simonw 9599 | 2021-06-16T05:05:51Z | 2021-06-16T05:06:11Z | OWNER | Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data It only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils memory command for directly querying CSV/JSON data 922099793 | |
862040971 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040971 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjA0MDk3MQ== | simonw 9599 | 2021-06-16T05:02:56Z | 2021-06-16T05:02:56Z | OWNER | Moving this to a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
862040906 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040906 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjA0MDkwNg== | simonw 9599 | 2021-06-16T05:02:47Z | 2021-06-16T05:02:47Z | OWNER | Got a prototype working!
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
862018937 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MjAxODkzNw== | simonw 9599 | 2021-06-16T03:59:28Z | 2021-06-16T04:00:05Z | OWNER | Mainly for debugging purposes it would be useful to be able to save the created in-memory database back to a file again later. This could be done with:
Can use Maybe instead (or as-well-as) offer |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861989987 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861989987 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTk4OTk4Nw== | simonw 9599 | 2021-06-16T02:34:21Z | 2021-06-16T02:34:21Z | OWNER | The documentation already covers this
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861987651 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861987651 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTk4NzY1MQ== | simonw 9599 | 2021-06-16T02:27:20Z | 2021-06-16T02:27:20Z | OWNER | Solution: |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861985944 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTk4NTk0NA== | simonw 9599 | 2021-06-16T02:22:52Z | 2021-06-16T02:22:52Z | OWNER | Another option: allow an optional
One catch: how to treat
That's fine for CSV, but what about TSV or JSON or nl-JSON? Maybe this:
Bit weird though. The alternative would be to support this:
But that's verbose compared to the version without the long |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861984707 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861984707 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTk4NDcwNw== | simonw 9599 | 2021-06-16T02:19:48Z | 2021-06-16T02:19:48Z | OWNER | This is going to need to be a separate command, for relatively non-obvious reasons.
Is equivalent to this, because
But... this means that making the filename optional doesn't actually work - because then this is ambiguous:
So instead, I'm going to add a new sub-command. I'm currently thinking
I still think I need to use Another option: allow multiple arguments which are filenames, and use the extension (or sniff the content) to decide what to do with them:
This would require the last positional argument to always be a SQL query, and would treat all other positional arguments as files that should be imported into memory. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861891835 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891835 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTg5MTgzNQ== | simonw 9599 | 2021-06-15T23:09:31Z | 2021-06-15T23:09:31Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861891693 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891693 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTg5MTY5Mw== | simonw 9599 | 2021-06-15T23:09:08Z | 2021-06-15T23:09:08Z | OWNER | Problem: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861891272 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891272 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTg5MTI3Mg== | simonw 9599 | 2021-06-15T23:08:02Z | 2021-06-15T23:08:02Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861891110 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891110 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTg5MTExMA== | simonw 9599 | 2021-06-15T23:07:38Z | 2021-06-15T23:07:38Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861890689 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861890689 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTg5MDY4OQ== | simonw 9599 | 2021-06-15T23:06:37Z | 2021-06-15T23:06:37Z | OWNER | How about And what happens if you provide a filename too? I'm tempted to say that the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861889437 | https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861889437 | https://api.github.com/repos/simonw/sqlite-utils/issues/272 | MDEyOklzc3VlQ29tbWVudDg2MTg4OTQzNw== | simonw 9599 | 2021-06-15T23:03:26Z | 2021-06-15T23:03:26Z | OWNER | Maybe also support |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Idea: import CSV to memory, run SQL, export in a single command 921878733 | |
861103967 | https://github.com/simonw/sqlite-utils/issues/269#issuecomment-861103967 | https://api.github.com/repos/simonw/sqlite-utils/issues/269 | MDEyOklzc3VlQ29tbWVudDg2MTEwMzk2Nw== | simonw 9599 | 2021-06-15T01:34:10Z | 2021-06-15T01:34:10Z | OWNER | SQLite doesn't have the concept of a boolean column, so there's not much I can do here: https://www.sqlite.org/datatype3.html#boolean_datatype |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
bool type not supported 919250621 | |
861103684 | https://github.com/simonw/sqlite-utils/issues/266#issuecomment-861103684 | https://api.github.com/repos/simonw/sqlite-utils/issues/266 | MDEyOklzc3VlQ29tbWVudDg2MTEwMzY4NA== | simonw 9599 | 2021-06-15T01:33:13Z | 2021-06-15T01:33:13Z | OWNER | Dupe of #37 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add some types, enforce with mypy 913135723 | |
861089794 | https://github.com/simonw/datasette/issues/1377#issuecomment-861089794 | https://api.github.com/repos/simonw/datasette/issues/1377 | MDEyOklzc3VlQ29tbWVudDg2MTA4OTc5NA== | simonw 9599 | 2021-06-15T00:53:29Z | 2021-06-15T00:53:29Z | OWNER | Potential hook names:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for plugins to exclude certain paths from CSRF checks 920884085 | |
861087949 | https://github.com/simonw/datasette/issues/1377#issuecomment-861087949 | https://api.github.com/repos/simonw/datasette/issues/1377 | MDEyOklzc3VlQ29tbWVudDg2MTA4Nzk0OQ== | simonw 9599 | 2021-06-15T00:49:19Z | 2021-06-15T00:49:19Z | OWNER | The new Now I need to design a plugin hook that allows plugins to have an opinion on whether a specific |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for plugins to exclude certain paths from CSRF checks 920884085 | |
861042050 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861042050 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MTA0MjA1MA== | simonw 9599 | 2021-06-14T22:45:42Z | 2021-06-14T22:45:42Z | MEMBER | I'm definitely interested in supporting events in this tool - see #14. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
feature: support "events" 920636216 | |
861041597 | https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861041597 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64 | MDEyOklzc3VlQ29tbWVudDg2MTA0MTU5Nw== | simonw 9599 | 2021-06-14T22:44:54Z | 2021-06-14T22:44:54Z | MEMBER | Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
feature: support "events" 920636216 | |
860230663 | https://github.com/simonw/datasette/issues/1376#issuecomment-860230663 | https://api.github.com/repos/simonw/datasette/issues/1376 | MDEyOklzc3VlQ29tbWVudDg2MDIzMDY2Mw== | simonw 9599 | 2021-06-13T15:39:37Z | 2021-06-13T15:39:37Z | OWNER | Actually it looks like there is a PR open already that addresses this: #1296 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns) 919822817 | |
860230385 | https://github.com/simonw/datasette/issues/1375#issuecomment-860230385 | https://api.github.com/repos/simonw/datasette/issues/1375 | MDEyOklzc3VlQ29tbWVudDg2MDIzMDM4NQ== | simonw 9599 | 2021-06-13T15:37:49Z | 2021-06-13T15:37:49Z | OWNER | There is a feature for this at the moment, but it's a little bit hidden: you can use I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
JSON export dumps JSON fields as TEXT 919508498 | |
860229397 | https://github.com/simonw/datasette/issues/1376#issuecomment-860229397 | https://api.github.com/repos/simonw/datasette/issues/1376 | MDEyOklzc3VlQ29tbWVudDg2MDIyOTM5Nw== | simonw 9599 | 2021-06-13T15:31:02Z | 2021-06-13T15:31:02Z | OWNER | Alternative fix would be to update that section of the documentation - if the container upgrade proves tricky I can fall back on that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns) 919822817 | |
860229226 | https://github.com/simonw/datasette/issues/1376#issuecomment-860229226 | https://api.github.com/repos/simonw/datasette/issues/1376 | MDEyOklzc3VlQ29tbWVudDg2MDIyOTIyNg== | simonw 9599 | 2021-06-13T15:29:45Z | 2021-06-13T15:29:45Z | OWNER | Oh good catch - this is a SQLite version issue. The https://latest.datasette.io/-/versions But... it looks like the packaged Datasette Docker container doesn't have that SQLite version! I should fix that. I'm renaming this issue. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns) 919822817 | |
860142489 | https://github.com/simonw/sqlite-utils/issues/271#issuecomment-860142489 | https://api.github.com/repos/simonw/sqlite-utils/issues/271 | MDEyOklzc3VlQ29tbWVudDg2MDE0MjQ4OQ== | simonw 9599 | 2021-06-13T02:53:06Z | 2021-06-13T02:53:06Z | OWNER | Looks like this is the problem: https://github.com/simonw/sqlite-utils/blob/b0f9d1e494c9891ce407e27b0f5c6deeea361d30/sqlite_utils/db.py#L1724-L1742 Note how |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
table.upsert_all() fails if input has a single column that should be a primary key 919702451 | |
859986489 | https://github.com/simonw/sqlite-utils/issues/270#issuecomment-859986489 | https://api.github.com/repos/simonw/sqlite-utils/issues/270 | MDEyOklzc3VlQ29tbWVudDg1OTk4NjQ4OQ== | simonw 9599 | 2021-06-12T02:47:12Z | 2021-06-12T02:47:12Z | OWNER | Can you expand on what you'd like to change here? The library and CLI tool already allow JSON data to be stored in columns: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Cannot set type JSON 919314806 | |
859898736 | https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859898736 | https://api.github.com/repos/simonw/sqlite-utils/issues/268 | MDEyOklzc3VlQ29tbWVudDg1OTg5ODczNg== | simonw 9599 | 2021-06-11T20:37:44Z | 2021-06-11T20:37:44Z | OWNER | From the prototype:
I don't think this matters. I see this tool as more for introspection than for recreating table structures. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
db.schema property and sqlite-utils schema command 919181559 | |
859895540 | https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859895540 | https://api.github.com/repos/simonw/sqlite-utils/issues/268 | MDEyOklzc3VlQ29tbWVudDg1OTg5NTU0MA== | simonw 9599 | 2021-06-11T20:30:34Z | 2021-06-11T20:30:34Z | OWNER | You can currently see the
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
db.schema property and sqlite-utils schema command 919181559 | |
859894105 | https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859894105 | https://api.github.com/repos/simonw/sqlite-utils/issues/268 | MDEyOklzc3VlQ29tbWVudDg1OTg5NDEwNQ== | simonw 9599 | 2021-06-11T20:28:52Z | 2021-06-11T20:28:52Z | OWNER | Out of interest, here are the rows from that table where
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
db.schema property and sqlite-utils schema command 919181559 | |
859888469 | https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859888469 | https://api.github.com/repos/simonw/sqlite-utils/issues/268 | MDEyOklzc3VlQ29tbWVudDg1OTg4ODQ2OQ== | simonw 9599 | 2021-06-11T20:26:20Z | 2021-06-11T20:26:20Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
db.schema property and sqlite-utils schema command 919181559 | |
858099514 | https://github.com/simonw/datasette/issues/1371#issuecomment-858099514 | https://api.github.com/repos/simonw/datasette/issues/1371 | MDEyOklzc3VlQ29tbWVudDg1ODA5OTUxNA== | simonw 9599 | 2021-06-09T21:03:49Z | 2021-06-09T21:03:49Z | OWNER | I'll release these as an alpha straight away - it makes sense to have plugin hook changes available for people to test as alpha dependencies ASAP. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Menu plugin hooks should include the request 915455228 | |
857139881 | https://github.com/simonw/datasette/pull/1370#issuecomment-857139881 | https://api.github.com/repos/simonw/datasette/issues/1370 | MDEyOklzc3VlQ29tbWVudDg1NzEzOTg4MQ== | simonw 9599 | 2021-06-08T20:58:41Z | 2021-06-08T20:58:41Z | OWNER | We can remove a bunch of unnecessary |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ensure db.path is a string before trying to insert into internal database 914130834 | |
856231119 | https://github.com/simonw/sqlite-utils/issues/266#issuecomment-856231119 | https://api.github.com/repos/simonw/sqlite-utils/issues/266 | MDEyOklzc3VlQ29tbWVudDg1NjIzMTExOQ== | simonw 9599 | 2021-06-07T20:26:05Z | 2021-06-07T20:26:05Z | OWNER | In Python 3.6 or higher can do this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add some types, enforce with mypy 913135723 | |
856212136 | https://github.com/simonw/datasette/issues/1365#issuecomment-856212136 | https://api.github.com/repos/simonw/datasette/issues/1365 | MDEyOklzc3VlQ29tbWVudDg1NjIxMjEzNg== | simonw 9599 | 2021-06-07T19:54:04Z | 2021-06-07T19:54:04Z | OWNER | I've hit this one too. I agree, fixing this in Datasette itself is better than fixing it in the tests across multiple other projects. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
pathlib.Path breaks internal schema 913017577 | |
856208637 | https://github.com/simonw/datasette/issues/1369#issuecomment-856208637 | https://api.github.com/repos/simonw/datasette/issues/1369 | MDEyOklzc3VlQ29tbWVudDg1NjIwODYzNw== | simonw 9599 | 2021-06-07T19:47:23Z | 2021-06-07T19:47:23Z | OWNER | No point in showing the IDs twice if the blue label doesn't differ from the gray ID |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Don't show foreign key IDs twice if no label 913900374 | |
856160770 | https://github.com/simonw/datasette/issues/1367#issuecomment-856160770 | https://api.github.com/repos/simonw/datasette/issues/1367 | MDEyOklzc3VlQ29tbWVudDg1NjE2MDc3MA== | simonw 9599 | 2021-06-07T18:22:33Z | 2021-06-07T18:22:33Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Navigation menu display bug 913823889 | ||
856147969 | https://github.com/simonw/datasette/issues/1366#issuecomment-856147969 | https://api.github.com/repos/simonw/datasette/issues/1366 | MDEyOklzc3VlQ29tbWVudDg1NjE0Nzk2OQ== | simonw 9599 | 2021-06-07T18:03:03Z | 2021-06-07T18:03:03Z | OWNER | Here's an example of a test that uses it. It's necessary because sometimes fixtures that create temporary directories break in unexpected ways: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Get rid of this `restore_working_directory` hack entirely 913809802 | |
856147450 | https://github.com/simonw/datasette/issues/1366#issuecomment-856147450 | https://api.github.com/repos/simonw/datasette/issues/1366 | MDEyOklzc3VlQ29tbWVudDg1NjE0NzQ1MA== | simonw 9599 | 2021-06-07T18:02:13Z | 2021-06-07T18:02:13Z | OWNER | The hack in question is this fixture, which I've been using in an ad-hoc manner to work around errors while running the tests: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/tests/conftest.py#L62-L75 I don't understand the underlying issue well enough to know how to get rid of it. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Get rid of this `restore_working_directory` hack entirely 913809802 | |
855611939 | https://github.com/simonw/sqlite-utils/issues/266#issuecomment-855611939 | https://api.github.com/repos/simonw/sqlite-utils/issues/266 | MDEyOklzc3VlQ29tbWVudDg1NTYxMTkzOQ== | simonw 9599 | 2021-06-07T06:07:41Z | 2021-06-07T06:07:41Z | OWNER | Looks like this is the way to do this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add some types, enforce with mypy 913135723 | |
855430317 | https://github.com/simonw/datasette/issues/1362#issuecomment-855430317 | https://api.github.com/repos/simonw/datasette/issues/1362 | MDEyOklzc3VlQ29tbWVudDg1NTQzMDMxNw== | simonw 9599 | 2021-06-06T17:07:48Z | 2021-06-06T17:07:48Z | OWNER | I guess I can offer a I like the idea of very actively encouraging CSP across all Datasette projects, but I'm nervous about making the software unusable for certain edge cases. Maybe require CSP and wait for someone to complain? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Consider using CSP to protect against future XSS 912864936 | |
855429111 | https://github.com/simonw/datasette/issues/1362#issuecomment-855429111 | https://api.github.com/repos/simonw/datasette/issues/1362 | MDEyOklzc3VlQ29tbWVudDg1NTQyOTExMQ== | simonw 9599 | 2021-06-06T16:59:05Z | 2021-06-06T17:00:15Z | OWNER | Twitter conversation: https://twitter.com/simonw/status/1401565566045806594 @dracos provided some really useful code examples there:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Consider using CSP to protect against future XSS 912864936 | |
855428601 | https://github.com/simonw/datasette/issues/1362#issuecomment-855428601 | https://api.github.com/repos/simonw/datasette/issues/1362 | MDEyOklzc3VlQ29tbWVudDg1NTQyODYwMQ== | simonw 9599 | 2021-06-06T16:55:33Z | 2021-06-06T16:55:33Z | OWNER |
Hah, of course! Thanks for the correction. So the nonce mechanism would actually be pretty great here, especially for the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Consider using CSP to protect against future XSS 912864936 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
issue >30