issue_comments
10,495 rows sorted by updated_at descending
This data as json, CSV (advanced)
updated_at (date) >30 ✖
- 2020-09-22 53
- 2020-10-15 52
- 2020-10-30 49
- 2020-06-09 43
- 2020-10-20 40
- 2020-06-18 39
- 2020-12-18 39
- 2020-05-27 38
- 2020-12-30 38
- 2020-10-09 37
- 2020-06-01 33
- 2020-06-08 33
- 2020-09-15 33
- 2021-01-04 33
- 2019-06-24 31
- 2020-09-21 31
- 2019-06-23 30
- 2018-04-16 29
- 2020-06-06 29
- 2020-06-16 29
- 2020-10-16 29
- 2020-05-30 28
- 2020-11-24 28
- 2020-12-31 28
- 2021-01-03 28
- 2017-11-13 27
- 2017-11-14 27
- 2020-02-14 27
- 2020-03-23 27
- 2020-08-18 26
- …
issue >30
- JavaScript plugin hooks mechanism similar to pluggy 47
- Redesign default .json format 45
- Port Datasette to ASGI 42
- Authentication (and permissions) as a core concept 40
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Ability to sort (and paginate) by column 31
- link_or_copy_directory() error - Invalid cross-device link 28
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- Proof of concept for Datasette on AWS Lambda with EFS 25
- Support cross-database joins 24
- Redesign register_output_renderer callback 24
- "datasette insert" command and plugin hook 23
- Datasette Plugins 22
- .json and .csv exports fail to apply base_url 22
- table.extract(...) method and "sqlite-utils extract" command 21
- Handle spatialite geometry columns better 20
- "flash messages" mechanism 20
- Move CI to GitHub Issues 20
- load_template hook doesn't work for include/extends 20
- ?sort=colname~numeric to sort by by column cast to real 19
- Better way of representing binary data in .csv output 19
- Introspect if table is FTS4 or FTS5 19
- Ability to ship alpha and beta releases 18
- Magic parameters for canned queries 18
- datasette.client internal requests mechanism 18
- …
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
783265830 | https://github.com/simonw/datasette/issues/782#issuecomment-783265830 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MzI2NTgzMA== | frankieroberto 30665 | 2021-02-22T10:21:14Z | 2021-02-22T10:21:14Z | NONE | @simonw:
Interesting! Although I don't think it matters too much what the underlying implementation is - I more meant that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782789598 | https://github.com/simonw/datasette/issues/782#issuecomment-782789598 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc4OTU5OA== | simonw 9599 | 2021-02-21T03:30:02Z | 2021-02-21T03:30:02Z | OWNER | Another benefit to default:object - I could include a key that shows a list of available extras. I could then use that to power an interactive API explorer. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782765665 | https://github.com/simonw/datasette/issues/782#issuecomment-782765665 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc2NTY2NQ== | simonw 9599 | 2021-02-20T23:34:41Z | 2021-02-20T23:34:41Z | OWNER | OK, I'm back to the "top level object as the default" side of things now - it's pretty much unanimous at this point, and it's certainly true that it's not a decision you'll even regret. |
{ "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782756398 | https://github.com/simonw/datasette/issues/782#issuecomment-782756398 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc1NjM5OA== | simonrjones 601316 | 2021-02-20T22:05:48Z | 2021-02-20T22:05:48Z | NONE |
I agree it is more predictable if the top level item is an object with a rows or data object that contains an array of data, which then allows for other top-level meta data. I can see the argument for removing this and just using an array for convenience - but I think that's OK as an option (as you have now). Rather than have lots of top-level keys you could have a "meta" object to contain non-data stuff. You could use something like "links" for API endpoint URLs (or use a standard like HAL). Which would then leave the top level a bit cleaner - if that's what you what. Have you had much feedback from users who use the Datasette API a lot? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782748501 | https://github.com/simonw/datasette/issues/782#issuecomment-782748501 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0ODUwMQ== | simonw 9599 | 2021-02-20T20:58:18Z | 2021-02-20T20:58:18Z | OWNER | Yet another option: support a
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782748093 | https://github.com/simonw/datasette/issues/782#issuecomment-782748093 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0ODA5Mw== | simonw 9599 | 2021-02-20T20:54:52Z | 2021-02-20T20:54:52Z | OWNER |
Adding a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782747878 | https://github.com/simonw/datasette/issues/782#issuecomment-782747878 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzg3OA== | simonw 9599 | 2021-02-20T20:53:11Z | 2021-02-20T20:53:11Z | OWNER | ... though thinking about this further, I could re-implement the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782747743 | https://github.com/simonw/datasette/issues/782#issuecomment-782747743 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzc0Mw== | simonw 9599 | 2021-02-20T20:52:10Z | 2021-02-20T20:52:10Z | OWNER |
The problem there is that https://latest-with-plugins.datasette.io/github?sql=select+*+from+commits - Datasette returns 1,000 results, and shows a "Custom SQL query returning more than 1,000 rows" message at the top. That's the I find myself using This is only really a concern for arbitrary SQL queries though - for table pages such as https://latest-with-plugins.datasette.io/github/commits?_size=10 adding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782747164 | https://github.com/simonw/datasette/issues/782#issuecomment-782747164 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0NzE2NA== | simonw 9599 | 2021-02-20T20:47:16Z | 2021-02-20T20:47:16Z | OWNER | (I started a thread on Twitter about this: https://twitter.com/simonw/status/1363220355318358016) |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782746755 | https://github.com/simonw/datasette/issues/782#issuecomment-782746755 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0Njc1NQ== | frankieroberto 30665 | 2021-02-20T20:44:05Z | 2021-02-20T20:44:05Z | NONE | Minor suggestion: rename I like the idea of specifying a limit of 0 if you don’t want any rows data - and returning an empty array under the Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782746633 | https://github.com/simonw/datasette/issues/782#issuecomment-782746633 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0NjYzMw== | simonw 9599 | 2021-02-20T20:43:07Z | 2021-02-20T20:43:07Z | OWNER | Another option:
I had something similar to this in Datasette a few years ago - a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782745199 | https://github.com/simonw/datasette/issues/782#issuecomment-782745199 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0NTE5OQ== | frankieroberto 30665 | 2021-02-20T20:32:03Z | 2021-02-20T20:32:03Z | NONE | I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default. Mainly because it allows you to add extra keys in a backwards-compatible way. Also just seems more expected somehow. The API design guidance for the UK government also recommends this: https://www.gov.uk/guidance/gds-api-technical-and-data-standards#use-json I also strongly dislike having versioned APIs (eg with a |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1 } |
Redesign default .json format 627794879 | |
782742233 | https://github.com/simonw/datasette/issues/782#issuecomment-782742233 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MjIzMw== | simonw 9599 | 2021-02-20T20:09:16Z | 2021-02-20T20:09:16Z | OWNER | I just noticed that https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0&_trace=1 executes 35 SQL queries at the moment! A great reminder that a big improvement from this change will be a reduction in queries through not calculating things like suggested facets unless they are explicitly requested. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782741719 | https://github.com/simonw/datasette/issues/782#issuecomment-782741719 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MTcxOQ== | simonw 9599 | 2021-02-20T20:05:04Z | 2021-02-20T20:05:04Z | OWNER |
This is something I've not thought very hard about. If there's an error, I need to return a top-level object, not a top-level array, so I can provide details of the error. But this means that client code will have to handle this difference - it will have to know that the returned data can be array-shaped if nothing went wrong, and object-shaped if there's an error. The HTTP status code helps here - calling client code can know that a 200 status code means there will be an array, but an error status code means an object. If developers really hate that the shape could be different, they can always use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782741107 | https://github.com/simonw/datasette/issues/782#issuecomment-782741107 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MTEwNw== | simonw 9599 | 2021-02-20T20:00:22Z | 2021-02-20T20:00:22Z | OWNER | A really exciting opportunity this opens up is for parallel execution - the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782740985 | https://github.com/simonw/datasette/issues/782#issuecomment-782740985 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MDk4NQ== | simonw 9599 | 2021-02-20T19:59:21Z | 2021-02-20T19:59:21Z | OWNER | This design should be influenced by how it's implemented. One implementation that could be nice is that each of the keys that can be requested - This raises more questions: Both I think I need to flesh out the full default collection of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782740604 | https://github.com/simonw/datasette/issues/782#issuecomment-782740604 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MDYwNA== | simonw 9599 | 2021-02-20T19:56:21Z | 2021-02-20T19:56:33Z | OWNER | I think I want to support |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782740488 | https://github.com/simonw/datasette/issues/782#issuecomment-782740488 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4Mjc0MDQ4OA== | simonw 9599 | 2021-02-20T19:55:23Z | 2021-02-20T19:55:23Z | OWNER | Am I saying you won't get back a key in the response unless you explicitly request it, either by name or by specifying a bundle of extras (e.g. The Also: having bundles like Maybe bundles could be prefixed with something, perhaps an underscore? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782739926 | https://github.com/simonw/datasette/issues/782#issuecomment-782739926 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MjczOTkyNg== | simonw 9599 | 2021-02-20T19:51:30Z | 2021-02-20T19:52:19Z | OWNER | Demos:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782709425 | https://github.com/simonw/datasette/issues/782#issuecomment-782709425 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MjcwOTQyNQ== | simonw 9599 | 2021-02-20T16:24:54Z | 2021-02-20T16:24:54Z | OWNER | Having shortcuts means I could support |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782709270 | https://github.com/simonw/datasette/issues/782#issuecomment-782709270 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MjcwOTI3MA== | simonw 9599 | 2021-02-20T16:23:51Z | 2021-02-20T16:24:11Z | OWNER | Also how would you opt out of returning the Some options:
I quite like that last one with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782708938 | https://github.com/simonw/datasette/issues/782#issuecomment-782708938 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDc4MjcwODkzOA== | simonw 9599 | 2021-02-20T16:22:14Z | 2021-02-20T16:22:14Z | OWNER | I'm leaning back in the direction of a flat JSON array of objects as the default - this:
Would return:
Opting in to the But... having to add |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
782464306 | https://github.com/simonw/datasette/issues/1236#issuecomment-782464306 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ2NDMwNg== | simonw 9599 | 2021-02-19T23:57:32Z | 2021-02-19T23:57:32Z | OWNER | Need to test this on mobile. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to increase size of the SQL editor window 812228314 | |
782464215 | https://github.com/simonw/datasette/issues/1236#issuecomment-782464215 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ2NDIxNQ== | simonw 9599 | 2021-02-19T23:57:13Z | 2021-02-19T23:57:13Z | OWNER | Now live on https://latest.datasette.io/_memory |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to increase size of the SQL editor window 812228314 | |
782462049 | https://github.com/simonw/datasette/issues/1236#issuecomment-782462049 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ2MjA0OQ== | simonw 9599 | 2021-02-19T23:51:12Z | 2021-02-19T23:51:12Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to increase size of the SQL editor window 812228314 | ||
782459550 | https://github.com/simonw/datasette/issues/1236#issuecomment-782459550 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ1OTU1MA== | simonw 9599 | 2021-02-19T23:45:30Z | 2021-02-19T23:45:30Z | OWNER | Encoded using https://meyerweb.com/eric/tools/dencoder/
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to increase size of the SQL editor window 812228314 | |
782459405 | https://github.com/simonw/datasette/issues/1236#issuecomment-782459405 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ1OTQwNQ== | simonw 9599 | 2021-02-19T23:45:02Z | 2021-02-19T23:45:02Z | OWNER | I'm going to use a variant of the Datasette menu icon. Here it is in
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to increase size of the SQL editor window 812228314 | |
782458983 | https://github.com/simonw/datasette/issues/1236#issuecomment-782458983 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ1ODk4Mw== | simonw 9599 | 2021-02-19T23:43:34Z | 2021-02-19T23:43:34Z | OWNER | I only want it to resize up and down, not left to right - so I'm not keen on the default resize handle: https://rawgit.com/Sphinxxxx/cm-resize/master/demo/index.html |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to increase size of the SQL editor window 812228314 | |
782458744 | https://github.com/simonw/datasette/issues/1236#issuecomment-782458744 | https://api.github.com/repos/simonw/datasette/issues/1236 | MDEyOklzc3VlQ29tbWVudDc4MjQ1ODc0NA== | simonw 9599 | 2021-02-19T23:42:42Z | 2021-02-19T23:42:42Z | OWNER | I can use https://github.com/Sphinxxxx/cm-resize for this |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ability to increase size of the SQL editor window 812228314 | |
782430028 | https://github.com/simonw/datasette/issues/1212#issuecomment-782430028 | https://api.github.com/repos/simonw/datasette/issues/1212 | MDEyOklzc3VlQ29tbWVudDc4MjQzMDAyOA== | kbaikov 4488943 | 2021-02-19T22:54:13Z | 2021-02-19T22:54:13Z | CONTRIBUTOR | I will close this issue since it appears only in my particular setup. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Tests are very slow. 797651831 | |
782246111 | https://github.com/simonw/datasette/issues/619#issuecomment-782246111 | https://api.github.com/repos/simonw/datasette/issues/619 | MDEyOklzc3VlQ29tbWVudDc4MjI0NjExMQ== | simonw 9599 | 2021-02-19T18:11:22Z | 2021-02-19T18:11:22Z | OWNER | Big usability improvement, see also #1236 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"Invalid SQL" page should let you edit the SQL 520655983 | |
782053455 | https://github.com/simonw/datasette/pull/1229#issuecomment-782053455 | https://api.github.com/repos/simonw/datasette/issues/1229 | MDEyOklzc3VlQ29tbWVudDc4MjA1MzQ1NQ== | camallen 295329 | 2021-02-19T12:47:19Z | 2021-02-19T12:47:19Z | CONTRIBUTOR | I believe this pr and #1031 are related and fix the same issue. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
ensure immutable databses when starting in configuration directory mode with 810507413 | |
781825726 | https://github.com/simonw/sqlite-utils/issues/236#issuecomment-781825726 | https://api.github.com/repos/simonw/sqlite-utils/issues/236 | MDEyOklzc3VlQ29tbWVudDc4MTgyNTcyNg== | simonw 9599 | 2021-02-19T05:10:41Z | 2021-02-19T05:10:41Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--attach command line option for attaching extra databases 811680502 | ||
781825187 | https://github.com/simonw/sqlite-utils/issues/113#issuecomment-781825187 | https://api.github.com/repos/simonw/sqlite-utils/issues/113 | MDEyOklzc3VlQ29tbWVudDc4MTgyNTE4Nw== | simonw 9599 | 2021-02-19T05:09:12Z | 2021-02-19T05:09:12Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Syntactic sugar for ATTACH DATABASE 621286870 | ||
781764561 | https://github.com/simonw/datasette/issues/283#issuecomment-781764561 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTc2NDU2MQ== | simonw 9599 | 2021-02-19T02:10:21Z | 2021-02-19T02:10:21Z | OWNER | This feature is now released! https://docs.datasette.io/en/stable/changelog.html#v0-55 |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0 } |
Support cross-database joins 325958506 | |
781736855 | https://github.com/simonw/datasette/issues/1235#issuecomment-781736855 | https://api.github.com/repos/simonw/datasette/issues/1235 | MDEyOklzc3VlQ29tbWVudDc4MTczNjg1NQ== | simonw 9599 | 2021-02-19T00:52:47Z | 2021-02-19T01:47:53Z | OWNER | I bumped the two lines in the
Then I ran it with:
http://0.0.0.0:8001/-/versions confirmed that it was now running Python 3.7.10 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Upgrade Python version used by official Datasette Docker image 811589344 | |
781735887 | https://github.com/simonw/datasette/issues/1235#issuecomment-781735887 | https://api.github.com/repos/simonw/datasette/issues/1235 | MDEyOklzc3VlQ29tbWVudDc4MTczNTg4Nw== | simonw 9599 | 2021-02-19T00:50:21Z | 2021-02-19T00:50:55Z | OWNER | I'll bump to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Upgrade Python version used by official Datasette Docker image 811589344 | |
781670827 | https://github.com/simonw/datasette/issues/283#issuecomment-781670827 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTY3MDgyNw== | simonw 9599 | 2021-02-18T22:16:46Z | 2021-02-18T22:16:46Z | OWNER | Demo is now live here: https://latest.datasette.io/_memory The documentation is at https://docs.datasette.io/en/latest/sql_queries.html#cross-database-queries - it links to this example query: https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support cross-database joins 325958506 | |
781599929 | https://github.com/simonw/datasette/pull/1232#issuecomment-781599929 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTU5OTkyOQ== | codecov[bot] 22429695 | 2021-02-18T19:59:54Z | 2021-02-18T22:06:42Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1232 +/-==========================================
+ Coverage 91.42% 91.46% +0.03% | Impacted Files | Coverage Δ | |
|---|---|---|
| datasette/app.py | Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--crossdb option for joining across databases 811407131 | |
781665560 | https://github.com/simonw/datasette/issues/283#issuecomment-781665560 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTY2NTU2MA== | simonw 9599 | 2021-02-18T22:06:14Z | 2021-02-18T22:06:14Z | OWNER | The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see Datasette Library) this won't be particularly useful for you. So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting Also worth noting: plugins that implement the prepare_connection() hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support cross-database joins 325958506 | |
781651283 | https://github.com/simonw/datasette/pull/1232#issuecomment-781651283 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTY1MTI4Mw== | simonw 9599 | 2021-02-18T21:37:55Z | 2021-02-18T21:37:55Z | OWNER | UI listing the attached tables: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--crossdb option for joining across databases 811407131 | |
781641728 | https://github.com/simonw/datasette/pull/1232#issuecomment-781641728 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTY0MTcyOA== | simonw 9599 | 2021-02-18T21:19:34Z | 2021-02-18T21:19:34Z | OWNER | I tested the demo deployment like this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--crossdb option for joining across databases 811407131 | |
781637292 | https://github.com/simonw/datasette/pull/1232#issuecomment-781637292 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTYzNzI5Mg== | simonw 9599 | 2021-02-18T21:11:31Z | 2021-02-18T21:11:31Z | OWNER | Due to bug #1233 I'm going to publish the additional database as |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--crossdb option for joining across databases 811407131 | |
781636590 | https://github.com/simonw/datasette/issues/1233#issuecomment-781636590 | https://api.github.com/repos/simonw/datasette/issues/1233 | MDEyOklzc3VlQ29tbWVudDc4MTYzNjU5MA== | simonw 9599 | 2021-02-18T21:10:08Z | 2021-02-18T21:10:08Z | OWNER | I think the bug is here: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/init.py#L349-L373 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"datasette publish cloudrun" cannot publish files with spaces in their name 811458446 | |
781634819 | https://github.com/simonw/datasette/pull/1232#issuecomment-781634819 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTYzNDgxOQ== | simonw 9599 | 2021-02-18T21:06:43Z | 2021-02-18T21:06:43Z | OWNER | I'll document this option on https://docs.datasette.io/en/stable/sql_queries.html |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--crossdb option for joining across databases 811407131 | |
781629841 | https://github.com/simonw/datasette/pull/1232#issuecomment-781629841 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTYyOTg0MQ== | simonw 9599 | 2021-02-18T20:57:23Z | 2021-02-18T20:57:23Z | OWNER | The new warning looks like this: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--crossdb option for joining across databases 811407131 | |
781598585 | https://github.com/simonw/datasette/pull/1232#issuecomment-781598585 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTU5ODU4NQ== | simonw 9599 | 2021-02-18T19:57:30Z | 2021-02-18T19:57:30Z | OWNER | It would also be neat if https://latest.datasette.io/ had multiple databases attached in order to demonstrate this feature. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--crossdb option for joining across databases 811407131 | |
781594632 | https://github.com/simonw/datasette/pull/1232#issuecomment-781594632 | https://api.github.com/repos/simonw/datasette/issues/1232 | MDEyOklzc3VlQ29tbWVudDc4MTU5NDYzMg== | simonw 9599 | 2021-02-18T19:50:21Z | 2021-02-18T19:50:21Z | OWNER | It would be neat if the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--crossdb option for joining across databases 811407131 | |
781593169 | https://github.com/simonw/datasette/issues/283#issuecomment-781593169 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTU5MzE2OQ== | simonw 9599 | 2021-02-18T19:47:34Z | 2021-02-18T19:47:34Z | OWNER | I have a working version now, moving development to a pull request. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support cross-database joins 325958506 | |
781591015 | https://github.com/simonw/datasette/issues/283#issuecomment-781591015 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTU5MTAxNQ== | simonw 9599 | 2021-02-18T19:44:02Z | 2021-02-18T19:44:02Z | OWNER | For the moment I'm going to hard-code a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support cross-database joins 325958506 | |
781574786 | https://github.com/simonw/datasette/issues/283#issuecomment-781574786 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTU3NDc4Ng== | simonw 9599 | 2021-02-18T19:15:37Z | 2021-02-18T19:15:37Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support cross-database joins 325958506 | |
781573676 | https://github.com/simonw/datasette/issues/283#issuecomment-781573676 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTU3MzY3Ng== | simonw 9599 | 2021-02-18T19:13:30Z | 2021-02-18T19:13:30Z | OWNER | It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support cross-database joins 325958506 | |
781560989 | https://github.com/simonw/datasette/issues/1231#issuecomment-781560989 | https://api.github.com/repos/simonw/datasette/issues/1231 | MDEyOklzc3VlQ29tbWVudDc4MTU2MDk4OQ== | simonw 9599 | 2021-02-18T18:50:53Z | 2021-02-18T18:50:53Z | OWNER | Ideally I'd figure out a way to replicate this error in a concurrent unit test. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Race condition errors in new refresh_schemas() mechanism 811367257 | |
781560865 | https://github.com/simonw/datasette/issues/1231#issuecomment-781560865 | https://api.github.com/repos/simonw/datasette/issues/1231 | MDEyOklzc3VlQ29tbWVudDc4MTU2MDg2NQ== | simonw 9599 | 2021-02-18T18:50:38Z | 2021-02-18T18:50:38Z | OWNER | I started trying to use locks to resolve this but I've not figured out the right way to do that yet - here's my first experiment: ```diff diff --git a/datasette/app.py b/datasette/app.py index 9e15a16..1681c9d 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -217,6 +217,7 @@ class Datasette: self.inspect_data = inspect_data self.immutables = set(immutables or []) self.databases = collections.OrderedDict() + self._refresh_schemas_lock = threading.Lock() if memory or not self.files: self.add_database(Database(self, is_memory=True), name="_memory") # memory_name is a random string so that each Datasette instance gets its own @@ -324,6 +325,13 @@ class Datasette: self.client = DatasetteClient(self)
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Race condition errors in new refresh_schemas() mechanism 811367257 | |
781546512 | https://github.com/simonw/datasette/issues/1226#issuecomment-781546512 | https://api.github.com/repos/simonw/datasette/issues/1226 | MDEyOklzc3VlQ29tbWVudDc4MTU0NjUxMg== | simonw 9599 | 2021-02-18T18:26:19Z | 2021-02-18T18:26:19Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--port option should validate port is between 0 and 65535 808843401 | ||
781530157 | https://github.com/simonw/datasette/issues/1226#issuecomment-781530157 | https://api.github.com/repos/simonw/datasette/issues/1226 | MDEyOklzc3VlQ29tbWVudDc4MTUzMDE1Nw== | simonw 9599 | 2021-02-18T18:00:15Z | 2021-02-18T18:00:15Z | OWNER | I can use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--port option should validate port is between 0 and 65535 808843401 | |
781451701 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-781451701 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc4MTQ1MTcwMQ== | Btibert3 203343 | 2021-02-18T16:06:21Z | 2021-02-18T16:06:21Z | NONE | Awesome! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature Request: Gmail 778380836 | |
781330466 | https://github.com/simonw/datasette/issues/1230#issuecomment-781330466 | https://api.github.com/repos/simonw/datasette/issues/1230 | MDEyOklzc3VlQ29tbWVudDc4MTMzMDQ2Ng== | Kabouik 7107523 | 2021-02-18T13:06:22Z | 2021-02-18T15:22:15Z | NONE | [Edit] Oh, I just saw the "Load all" button under the cluster map as well as the setting to alter the max number or results. So I guess this issue only is about the Vega charts.
Note that datasette-cluster-map also seems to be limited to 998 displayed points:
![ss-2021-02-18_140548](https://user-images.githubusercontent.com/7107523/108361225-15fb2a80-71ea-11eb-9a19-d885e8513f55.png)
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Vega charts are plotted only for rows on the visible page, cluster maps only for rows in the remaining pages 811054000 | |
781077127 | https://github.com/simonw/datasette/issues/283#issuecomment-781077127 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw== | simonw 9599 | 2021-02-18T05:56:30Z | 2021-02-18T05:57:34Z | OWNER | I'm going to to try prototyping the |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0 } |
Support cross-database joins 325958506 | |
780991910 | https://github.com/simonw/datasette/issues/283#issuecomment-780991910 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4MDk5MTkxMA== | rayvoelker 9308268 | 2021-02-18T02:13:56Z | 2021-02-18T02:13:56Z | NONE | I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining? I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots https://ilsweb.cincinnatilibrary.org/collection-analysis/ and thanks again for such an amazing tool! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support cross-database joins 325958506 | |
780817596 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-780817596 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc4MDgxNzU5Ng== | UtahDave 306240 | 2021-02-17T20:01:35Z | 2021-02-17T20:01:35Z | NONE | I've got this almost working. Just needs some polish |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature Request: Gmail 778380836 | |
779785638 | https://github.com/simonw/sqlite-utils/issues/227#issuecomment-779785638 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | MDEyOklzc3VlQ29tbWVudDc3OTc4NTYzOA== | camallen 295329 | 2021-02-16T11:48:03Z | 2021-02-16T11:48:03Z | NONE | Thank you @simonw |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error reading csv files with large column data 807174161 | |
779467451 | https://github.com/simonw/datasette/issues/1226#issuecomment-779467451 | https://api.github.com/repos/simonw/datasette/issues/1226 | MDEyOklzc3VlQ29tbWVudDc3OTQ2NzQ1MQ== | simonw 9599 | 2021-02-15T22:02:46Z | 2021-02-15T22:02:46Z | OWNER | I'm OK with the current error message shown if you try to use too low a port:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--port option should validate port is between 0 and 65535 808843401 | |
779467160 | https://github.com/simonw/datasette/issues/1226#issuecomment-779467160 | https://api.github.com/repos/simonw/datasette/issues/1226 | MDEyOklzc3VlQ29tbWVudDc3OTQ2NzE2MA== | simonw 9599 | 2021-02-15T22:01:53Z | 2021-02-15T22:01:53Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--port option should validate port is between 0 and 65535 808843401 | ||
779416619 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779416619 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQxNjYxOQ== | simonw 9599 | 2021-02-15T19:40:57Z | 2021-02-15T21:27:55Z | OWNER | Tried this experiment (not proper binary search, it only searches downwards): ```python import sqlite3 db = sqlite3.connect(":memory:") def tryit(n): sql = "select 1 where 1 in ({})".format(", ".join("?" for i in range(n))) db.execute(sql, [0 for i in range(n)]) def find_limit(min=0, max=5_000_000):
value = max
while True:
print('Trying', value)
try:
tryit(value)
return value
except:
value = value // 2
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779448912 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779448912 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQ0ODkxMg== | simonw 9599 | 2021-02-15T21:09:50Z | 2021-02-15T21:09:50Z | OWNER | I fiddled around and replaced that line with
43s is definitely better than 56s, but it's still not as big as the ~26.5s to ~3.5s improvement described by @simonwiles at the top of this issue. I wonder what I'm missing here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779446652 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779446652 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQ0NjY1Mg== | simonw 9599 | 2021-02-15T21:04:19Z | 2021-02-15T21:04:19Z | OWNER | ... but it looks like And |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779445423 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779445423 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQ0NTQyMw== | simonw 9599 | 2021-02-15T21:00:44Z | 2021-02-15T21:01:09Z | OWNER | I tried changing the hard-coded value from 999 to 156_250 and running Increased the setting here(sqlite-utils) sqlite-utils % time sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers [###################################-] 99% 00:00:00sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv 39.40s user 5.15s system 96% cpu 46.320 total ``` Not as big a difference as I was expecting. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779417723 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779417723 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQxNzcyMw== | simonw 9599 | 2021-02-15T19:44:02Z | 2021-02-15T19:47:00Z | OWNER |
All of these are still slow enough that I'm not comfortable running this search for every time the library is imported. Allowing users to opt-in to this as a performance enhancement might be better. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
779409770 | https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779409770 | https://api.github.com/repos/simonw/sqlite-utils/issues/147 | MDEyOklzc3VlQ29tbWVudDc3OTQwOTc3MA== | simonw 9599 | 2021-02-15T19:23:11Z | 2021-02-15T19:23:11Z | OWNER | On my Mac right now I'm seeing a limit of 500,000:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
SQLITE_MAX_VARS maybe hard-coded too low 688670158 | |
778854808 | https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778854808 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | MDEyOklzc3VlQ29tbWVudDc3ODg1NDgwOA== | simonw 9599 | 2021-02-14T22:46:54Z | 2021-02-14T22:46:54Z | OWNER | Fix is released in 3.5. |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error reading csv files with large column data 807174161 | |
778851721 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778851721 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODg1MTcyMQ== | simonw 9599 | 2021-02-14T22:23:46Z | 2021-02-14T22:23:46Z | OWNER | I called this |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--no-headers option for CSV and TSV 807437089 | |
778849394 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778849394 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODg0OTM5NA== | simonw 9599 | 2021-02-14T22:06:53Z | 2021-02-14T22:06:53Z | OWNER | For the moment I think just adding Users can import with that option, then use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--no-headers option for CSV and TSV 807437089 | |
778844016 | https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778844016 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | MDEyOklzc3VlQ29tbWVudDc3ODg0NDAxNg== | simonw 9599 | 2021-02-14T21:22:45Z | 2021-02-14T21:22:45Z | OWNER | I'm going to use this pattern from https://stackoverflow.com/a/15063941 ```python import sys import csv maxInt = sys.maxsize while True: # decrease the maxInt value by factor 10 # as long as the OverflowError occurs.
``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Hitting `_csv.Error: field larger than field limit (131072)` 807817197 | |
778843503 | https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843503 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzUwMw== | simonw 9599 | 2021-02-14T21:18:51Z | 2021-02-14T21:18:51Z | OWNER | I want to set this to the maximum allowed limit, which seems to be surprisingly hard! That StackOverflow thread is full of ideas for that, many of them involving |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Hitting `_csv.Error: field larger than field limit (131072)` 807817197 | |
778843362 | https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843362 | https://api.github.com/repos/simonw/sqlite-utils/issues/229 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzM2Mg== | simonw 9599 | 2021-02-14T21:17:53Z | 2021-02-14T21:17:53Z | OWNER | Same issue as #227. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Hitting `_csv.Error: field larger than field limit (131072)` 807817197 | |
778811746 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811746 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODgxMTc0Ng== | simonw 9599 | 2021-02-14T17:39:30Z | 2021-02-14T21:16:54Z | OWNER | I'm going to detach this from the #131 column types idea. The three things I need to handle here are:
Here's a potential design that covers the first two:
It doesn't cover the "give me unknown column names" case though. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--no-headers option for CSV and TSV 807437089 | |
778843086 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778843086 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODg0MzA4Ng== | simonw 9599 | 2021-02-14T21:15:43Z | 2021-02-14T21:15:43Z | OWNER | I'm not convinced the
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--no-headers option for CSV and TSV 807437089 | |
778842982 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778842982 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODg0Mjk4Mg== | simonw 9599 | 2021-02-14T21:15:11Z | 2021-02-14T21:15:11Z | OWNER | Implementation tip: I have code that reads the first row and uses it as headers here: https://github.com/simonw/sqlite-utils/blob/8f042ae1fd323995d966a94e8e6df85cc843b938/sqlite_utils/cli.py#L689-L691 So If I want to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--no-headers option for CSV and TSV 807437089 | |
778841704 | https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778841704 | https://api.github.com/repos/simonw/sqlite-utils/issues/227 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTcwNA== | simonw 9599 | 2021-02-14T21:05:20Z | 2021-02-14T21:05:20Z | OWNER | This has also been reported in #229. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error reading csv files with large column data 807174161 | |
778841547 | https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778841547 | https://api.github.com/repos/simonw/sqlite-utils/issues/225 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTU0Nw== | simonw 9599 | 2021-02-14T21:04:13Z | 2021-02-14T21:04:13Z | OWNER | I added a test and fixed this in #234 - thanks for the fix. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fix for problem in Table.insert_all on search for columns per chunk of rows 797159961 | |
778841278 | https://github.com/simonw/sqlite-utils/issues/234#issuecomment-778841278 | https://api.github.com/repos/simonw/sqlite-utils/issues/234 | MDEyOklzc3VlQ29tbWVudDc3ODg0MTI3OA== | simonw 9599 | 2021-02-14T21:02:11Z | 2021-02-14T21:02:11Z | OWNER | I managed to replicate this in a test:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.insert_all() fails if subsequent chunks contain additional columns 808046597 | |
778834504 | https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778834504 | https://api.github.com/repos/simonw/sqlite-utils/issues/225 | MDEyOklzc3VlQ29tbWVudDc3ODgzNDUwNA== | simonw 9599 | 2021-02-14T20:09:30Z | 2021-02-14T20:09:30Z | OWNER | Thanks for this. I'm going to try and get the test suite to run in Windows on GitHub Actions. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fix for problem in Table.insert_all on search for columns per chunk of rows 797159961 | |
778829456 | https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778829456 | https://api.github.com/repos/simonw/sqlite-utils/issues/231 | MDEyOklzc3VlQ29tbWVudDc3ODgyOTQ1Ng== | simonw 9599 | 2021-02-14T19:37:52Z | 2021-02-14T19:37:52Z | OWNER | I'm going to add
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
limit=X, offset=Y parameters for more Python methods 808028757 | |
778828758 | https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778828758 | https://api.github.com/repos/simonw/sqlite-utils/issues/231 | MDEyOklzc3VlQ29tbWVudDc3ODgyODc1OA== | simonw 9599 | 2021-02-14T19:33:14Z | 2021-02-14T19:33:14Z | OWNER | The |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
limit=X, offset=Y parameters for more Python methods 808028757 | |
778828495 | https://github.com/simonw/sqlite-utils/pull/224#issuecomment-778828495 | https://api.github.com/repos/simonw/sqlite-utils/issues/224 | MDEyOklzc3VlQ29tbWVudDc3ODgyODQ5NQ== | simonw 9599 | 2021-02-14T19:31:06Z | 2021-02-14T19:31:06Z | OWNER | I'm going to add a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add fts offset docs. 792297010 | |
778827570 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778827570 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgyNzU3MA== | simonw 9599 | 2021-02-14T19:24:20Z | 2021-02-14T19:24:20Z | OWNER | Here's the implementation in Python: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L204-L225 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--sniff option for sniffing delimiters 808008305 | |
778824361 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778824361 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgyNDM2MQ== | simonw 9599 | 2021-02-14T18:59:22Z | 2021-02-14T18:59:22Z | OWNER | I think I've got it. I can use
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--sniff option for sniffing delimiters 808008305 | |
778821403 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778821403 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgyMTQwMw== | simonw 9599 | 2021-02-14T18:38:16Z | 2021-02-14T18:38:16Z | OWNER | There are two code paths here that matter:
I'm a bit stuck on the second one. Ideally I could use something like |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--sniff option for sniffing delimiters 808008305 | |
778818639 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778818639 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgxODYzOQ== | simonw 9599 | 2021-02-14T18:22:38Z | 2021-02-14T18:22:38Z | OWNER | Maybe I shouldn't be using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--sniff option for sniffing delimiters 808008305 | |
778817494 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778817494 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgxNzQ5NA== | simonw 9599 | 2021-02-14T18:16:06Z | 2021-02-14T18:16:06Z | OWNER | Types involved:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--sniff option for sniffing delimiters 808008305 | |
778816333 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778816333 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgxNjMzMw== | simonw 9599 | 2021-02-14T18:08:44Z | 2021-02-14T18:08:44Z | OWNER | No, you can't |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--sniff option for sniffing delimiters 808008305 | |
778815740 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778815740 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgxNTc0MA== | simonw 9599 | 2021-02-14T18:05:03Z | 2021-02-14T18:05:03Z | OWNER | The challenge here is how to read the first 2048 bytes and then reset the incoming file. The Python docs example looks like this:
The challenge is going to be having the If |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--sniff option for sniffing delimiters 808008305 | |
778812684 | https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778812684 | https://api.github.com/repos/simonw/sqlite-utils/issues/230 | MDEyOklzc3VlQ29tbWVudDc3ODgxMjY4NA== | simonw 9599 | 2021-02-14T17:45:16Z | 2021-02-14T17:45:16Z | OWNER | Running this could take any CSV (or TSV) file and automatically detect the delimiter. If no header row is detected it could add
(Using This could be called |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--sniff option for sniffing delimiters 808008305 | |
778812050 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODgxMjA1MA== | simonw 9599 | 2021-02-14T17:41:30Z | 2021-02-14T17:41:30Z | OWNER | I just spotted that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--no-headers option for CSV and TSV 807437089 | |
778811934 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811934 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODgxMTkzNA== | simonw 9599 | 2021-02-14T17:40:48Z | 2021-02-14T17:40:48Z | OWNER | Another pattern that might be useful is to generate a header that is just "unknown1,unknown2,unknown3" for each of the columns in the rest of the file. This makes it easy to e.g. facet-explore within Datasette to figure out the correct names, then use I needed to do that for the https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c example. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--no-headers option for CSV and TSV 807437089 | |
778511347 | https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778511347 | https://api.github.com/repos/simonw/sqlite-utils/issues/228 | MDEyOklzc3VlQ29tbWVudDc3ODUxMTM0Nw== | simonw 9599 | 2021-02-12T23:27:50Z | 2021-02-12T23:27:50Z | OWNER | For the moment, a workaround can be to
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--no-headers option for CSV and TSV 807437089 | |
778510528 | https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778510528 | https://api.github.com/repos/simonw/sqlite-utils/issues/131 | MDEyOklzc3VlQ29tbWVudDc3ODUxMDUyOA== | simonw 9599 | 2021-02-12T23:25:06Z | 2021-02-12T23:25:06Z | OWNER | If |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils insert: options for column types 675753042 | |
778508887 | https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778508887 | https://api.github.com/repos/simonw/sqlite-utils/issues/131 | MDEyOklzc3VlQ29tbWVudDc3ODUwODg4Nw== | simonw 9599 | 2021-02-12T23:20:11Z | 2021-02-12T23:20:11Z | OWNER | Annoyingly Particularly annoying because I attempted to remove the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
sqlite-utils insert: options for column types 675753042 | |
778467759 | https://github.com/simonw/datasette/issues/1220#issuecomment-778467759 | https://api.github.com/repos/simonw/datasette/issues/1220 | MDEyOklzc3VlQ29tbWVudDc3ODQ2Nzc1OQ== | aborruso 30607 | 2021-02-12T21:35:17Z | 2021-02-12T21:35:17Z | NONE | Thank you |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Installing datasette via docker: Path 'fixtures.db' does not exist 806743116 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30