issue_comments
10,495 rows sorted by updated_at descending
This data as json, CSV (advanced)
issue >30
- JavaScript plugin hooks mechanism similar to pluggy 47
- Redesign default .json format 46
- Port Datasette to ASGI 42
- Authentication (and permissions) as a core concept 40
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Ability to sort (and paginate) by column 31
- link_or_copy_directory() error - Invalid cross-device link 28
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- Support cross-database joins 25
- Proof of concept for Datasette on AWS Lambda with EFS 25
- Redesign register_output_renderer callback 24
- "datasette insert" command and plugin hook 23
- Datasette Plugins 22
- .json and .csv exports fail to apply base_url 22
- table.extract(...) method and "sqlite-utils extract" command 21
- Handle spatialite geometry columns better 20
- "flash messages" mechanism 20
- Move CI to GitHub Issues 20
- load_template hook doesn't work for include/extends 20
- ?sort=colname~numeric to sort by by column cast to real 19
- Better way of representing binary data in .csv output 19
- Introspect if table is FTS4 or FTS5 19
- WIP: Add Gmail takeout mbox import 19
- Ability to ship alpha and beta releases 18
- Magic parameters for canned queries 18
- …
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
803697546 | https://github.com/simonw/datasette/issues/1249#issuecomment-803697546 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng== | simonw 9599 | 2021-03-22T00:59:47Z | 2021-03-22T00:59:47Z | OWNER | To debug I'm running:
This gets me a shell I can use. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803697211 | https://github.com/simonw/datasette/issues/1249#issuecomment-803697211 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ== | simonw 9599 | 2021-03-22T00:58:01Z | 2021-03-22T00:58:01Z | OWNER | I'm messing around with the
And then:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803694661 | https://github.com/simonw/datasette/issues/1249#issuecomment-803694661 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ== | simonw 9599 | 2021-03-22T00:46:49Z | 2021-03-22T00:46:49Z | OWNER | Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803694436 | https://github.com/simonw/datasette/issues/1249#issuecomment-803694436 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NDQzNg== | simonw 9599 | 2021-03-22T00:46:00Z | 2021-03-22T00:46:00Z | OWNER | So I'm going to try |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803694359 | https://github.com/simonw/datasette/issues/1249#issuecomment-803694359 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ== | simonw 9599 | 2021-03-22T00:45:47Z | 2021-03-22T00:45:47Z | OWNER | https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803693181 | https://github.com/simonw/datasette/issues/1249#issuecomment-803693181 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5MzE4MQ== | simonw 9599 | 2021-03-22T00:41:02Z | 2021-03-22T00:41:02Z | OWNER | Debian sid has it too: https://packages.debian.org/sid/libspatialite-dev |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803692673 | https://github.com/simonw/datasette/issues/1249#issuecomment-803692673 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw== | simonw 9599 | 2021-03-22T00:38:42Z | 2021-03-22T00:38:42Z | OWNER | Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803691236 | https://github.com/simonw/datasette/issues/1249#issuecomment-803691236 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg== | simonw 9599 | 2021-03-22T00:32:03Z | 2021-03-22T00:32:03Z | OWNER | Here's something odd: when I run But when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages. This is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803674728 | https://github.com/simonw/datasette/issues/1259#issuecomment-803674728 | https://api.github.com/repos/simonw/datasette/issues/1259 | MDEyOklzc3VlQ29tbWVudDgwMzY3NDcyOA== | simonw 9599 | 2021-03-21T22:55:31Z | 2021-03-21T22:55:31Z | OWNER | CTEs were added in 2014-02-03 SQLite 3.8.3 - so I think it's OK to depend on them for Datasette. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research using CTEs for faster facet counts 830567275 | |
803673225 | https://github.com/simonw/datasette/issues/647#issuecomment-803673225 | https://api.github.com/repos/simonw/datasette/issues/647 | MDEyOklzc3VlQ29tbWVudDgwMzY3MzIyNQ== | simonw 9599 | 2021-03-21T22:44:19Z | 2021-03-21T22:44:19Z | OWNER | Now that I'm looking at refactoring how views work in #878 it's clear that the gnarliest, most convoluted code I need to deal with relates to this old feature. I'm going to remove it entirely. Any performance enhancement or provides can be achieved just as well by using regular URLs and a caching proxy. I may provide a 404 handling plugin that attempts to rewrite old URLs that used this mechanism, but I won't do any more than that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Move hashed URL mode out to a plugin 531755959 | |
803631102 | https://github.com/simonw/datasette/issues/942#issuecomment-803631102 | https://api.github.com/repos/simonw/datasette/issues/942 | MDEyOklzc3VlQ29tbWVudDgwMzYzMTEwMg== | mroswell 192568 | 2021-03-21T17:48:42Z | 2021-03-21T17:48:42Z | CONTRIBUTOR | I like this idea. Though it might be nice to have some kind of automated system from database to file, so that developers could easily track diffs. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support column descriptions in metadata.json 681334912 | |
803502424 | https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803502424 | https://api.github.com/repos/simonw/sqlite-utils/issues/249 | MDEyOklzc3VlQ29tbWVudDgwMzUwMjQyNA== | prabhur 36287 | 2021-03-21T02:43:32Z | 2021-03-21T02:43:32Z | NONE |
Wow. Wasn't expecting a response this quick, especially during a weekend. :-) Sincerely appreciate it.
I tried the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Full text search possibly broken? 836963850 | |
803501756 | https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803501756 | https://api.github.com/repos/simonw/sqlite-utils/issues/249 | MDEyOklzc3VlQ29tbWVudDgwMzUwMTc1Ng== | simonw 9599 | 2021-03-21T02:33:45Z | 2021-03-21T02:33:45Z | OWNER | Did you run If so you'll need to run A better solution may be to add |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Full text search possibly broken? 836963850 | |
803499509 | https://github.com/simonw/datasette/issues/1261#issuecomment-803499509 | https://api.github.com/repos/simonw/datasette/issues/1261 | MDEyOklzc3VlQ29tbWVudDgwMzQ5OTUwOQ== | brimstone 812795 | 2021-03-21T02:06:43Z | 2021-03-21T02:06:43Z | NONE | I can confirm 0.9.2 fixes the problem. Thanks for the fast response! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Some links aren't properly URL encoded. 832092321 | |
803473015 | https://github.com/simonw/datasette/issues/878#issuecomment-803473015 | https://api.github.com/repos/simonw/datasette/issues/878 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ== | simonw 9599 | 2021-03-20T22:33:05Z | 2021-03-20T22:33:05Z | OWNER | Things this mechanism needs to be able to support:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
803472595 | https://github.com/simonw/datasette/issues/878#issuecomment-803472595 | https://api.github.com/repos/simonw/datasette/issues/878 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ== | simonw 9599 | 2021-03-20T22:28:12Z | 2021-03-20T22:28:12Z | OWNER | Another idea I had: a view is a class that takes the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
803472278 | https://github.com/simonw/datasette/issues/878#issuecomment-803472278 | https://api.github.com/repos/simonw/datasette/issues/878 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MjI3OA== | simonw 9599 | 2021-03-20T22:25:04Z | 2021-03-20T22:25:04Z | OWNER | I came up with a slightly wild idea for this that would involve pytest-style dependency injection. Prototype here: https://gist.github.com/simonw/496b24fdad44f6f8b7237fe394a0ced7 Copying from my private notes:
|
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
803471917 | https://github.com/simonw/datasette/issues/878#issuecomment-803471917 | https://api.github.com/repos/simonw/datasette/issues/878 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw== | simonw 9599 | 2021-03-20T22:21:33Z | 2021-03-20T22:21:33Z | OWNER | This has been blocking things for too long. If this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
803471702 | https://github.com/simonw/datasette/issues/1258#issuecomment-803471702 | https://api.github.com/repos/simonw/datasette/issues/1258 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MTcwMg== | simonw 9599 | 2021-03-20T22:19:39Z | 2021-03-20T22:19:39Z | OWNER | This is a good idea. I avoided this initially because it should be possible to run a canned query with a parameter set to the empty string, but that view could definitely be smart enough to differentiate between |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Allow canned query params to specify default values 828858421 | |
803469623 | https://github.com/simonw/datasette/issues/782#issuecomment-803469623 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDgwMzQ2OTYyMw== | simonw 9599 | 2021-03-20T22:01:23Z | 2021-03-20T22:01:23Z | OWNER | I'm going to keep |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
803468314 | https://github.com/simonw/datasette/issues/1261#issuecomment-803468314 | https://api.github.com/repos/simonw/datasette/issues/1261 | MDEyOklzc3VlQ29tbWVudDgwMzQ2ODMxNA== | simonw 9599 | 2021-03-20T21:48:48Z | 2021-03-20T21:48:48Z | OWNER | That's fixed in this release of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Some links aren't properly URL encoded. 832092321 | |
803466868 | https://github.com/simonw/datasette/issues/1261#issuecomment-803466868 | https://api.github.com/repos/simonw/datasette/issues/1261 | MDEyOklzc3VlQ29tbWVudDgwMzQ2Njg2OA== | simonw 9599 | 2021-03-20T21:36:06Z | 2021-03-20T21:36:06Z | OWNER | This isn't a Datasette bug - it's a Vercel bug: https://github.com/simonw/datasette-publish-vercel/issues/28 I'm looking at a fix for that now, so watch that issue for updates. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Some links aren't properly URL encoded. 832092321 | |
803466730 | https://github.com/simonw/datasette/issues/1266#issuecomment-803466730 | https://api.github.com/repos/simonw/datasette/issues/1266 | MDEyOklzc3VlQ29tbWVudDgwMzQ2NjczMA== | simonw 9599 | 2021-03-20T21:35:00Z | 2021-03-20T21:35:00Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documentation for Response.asgi_send(send) method 836273891 | ||
803160804 | https://github.com/simonw/datasette/issues/1265#issuecomment-803160804 | https://api.github.com/repos/simonw/datasette/issues/1265 | MDEyOklzc3VlQ29tbWVudDgwMzE2MDgwNA== | yunzheng 468612 | 2021-03-19T22:05:12Z | 2021-03-19T22:05:12Z | NONE | Wow that was fast! Thanks for this very cool project and quick update! 👍 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support for HTTP Basic Authentication 836123030 | |
803130332 | https://github.com/simonw/datasette/issues/1265#issuecomment-803130332 | https://api.github.com/repos/simonw/datasette/issues/1265 | MDEyOklzc3VlQ29tbWVudDgwMzEzMDMzMg== | simonw 9599 | 2021-03-19T21:03:09Z | 2021-03-19T21:03:09Z | OWNER | This is now available in |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0 } |
Support for HTTP Basic Authentication 836123030 | |
802923254 | https://github.com/simonw/datasette/issues/1265#issuecomment-802923254 | https://api.github.com/repos/simonw/datasette/issues/1265 | MDEyOklzc3VlQ29tbWVudDgwMjkyMzI1NA== | bobwhitelock 7476523 | 2021-03-19T15:39:15Z | 2021-03-19T15:39:15Z | CONTRIBUTOR | It doesn't use basic auth, but you can put a whole datasette instance, or parts of this, behind a username/password prompt using https://github.com/simonw/datasette-auth-passwords |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support for HTTP Basic Authentication 836123030 | |
802164134 | https://github.com/simonw/datasette/issues/1262#issuecomment-802164134 | https://api.github.com/repos/simonw/datasette/issues/1262 | MDEyOklzc3VlQ29tbWVudDgwMjE2NDEzNA== | henry501 19328961 | 2021-03-18T17:55:00Z | 2021-03-18T17:55:00Z | NONE | Thanks for the comments. I'll take a look at the documentation to familiarize myself, as I haven't tried to write any plugins yet. With some luck I might be ready to write it when the hook is implemented. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook that could support 'order by random()' for table view 834602299 | |
802099264 | https://github.com/simonw/datasette/issues/1262#issuecomment-802099264 | https://api.github.com/repos/simonw/datasette/issues/1262 | MDEyOklzc3VlQ29tbWVudDgwMjA5OTI2NA== | simonw 9599 | 2021-03-18T16:43:09Z | 2021-03-18T16:43:09Z | OWNER | I often find myself wanting this too, when I'm exploring a new dataset. i agree with Bob that this is a good candidate for a plugin. The plugin system isn't quite setup for this yet though - there isn't an obvious mechanism for adding extra sort orders or other interface elements that manipulate the query used by the table view in some way. I'm going to promote this issue to status of a plugin hook feature request - I have a hunch that a plugin hook that enables |
{ "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook that could support 'order by random()' for table view 834602299 | |
802095132 | https://github.com/simonw/datasette/issues/1262#issuecomment-802095132 | https://api.github.com/repos/simonw/datasette/issues/1262 | MDEyOklzc3VlQ29tbWVudDgwMjA5NTEzMg== | bobwhitelock 7476523 | 2021-03-18T16:37:45Z | 2021-03-18T16:37:45Z | CONTRIBUTOR | This sounds like a good use case for a plugin, since this will only be useful for a subset of Datasette users. It shouldn't be too difficult to add a button to do this with the available plugin hooks - have you taken a look at https://docs.datasette.io/en/latest/writing_plugins.html? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook that could support 'order by random()' for table view 834602299 | |
802032152 | https://github.com/simonw/sqlite-utils/issues/159#issuecomment-802032152 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | MDEyOklzc3VlQ29tbWVudDgwMjAzMjE1Mg== | limar 1025224 | 2021-03-18T15:42:52Z | 2021-03-18T15:42:52Z | NONE | I confirm the bug. Happens for me in version 3.6. I use the call to delete all the records:
I see that |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.delete_where() does not auto-commit (unlike .insert() or .upsert()) 702386948 | |
801816980 | https://github.com/simonw/sqlite-utils/issues/246#issuecomment-801816980 | https://api.github.com/repos/simonw/sqlite-utils/issues/246 | MDEyOklzc3VlQ29tbWVudDgwMTgxNjk4MA== | polyrand 37962604 | 2021-03-18T10:40:32Z | 2021-03-18T10:43:04Z | NONE | I have found a similar problem, but I only when using that type of query (with
I thought I could build a query like the first one using this function:
And then I use the output of that function as the query parameter for the standard However, my use case is different because I'm the one "deciding" when to use a prefix search, not the end user. I also haven't done many tests, but maybe you found that useful. One thing I could think of is checking if the query has an This is just for prefix queries, I think having the escaping function is still useful for other use cases. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Escaping FTS search strings 831751367 | |
799479175 | https://github.com/simonw/sqlite-utils/issues/246#issuecomment-799479175 | https://api.github.com/repos/simonw/sqlite-utils/issues/246 | MDEyOklzc3VlQ29tbWVudDc5OTQ3OTE3NQ== | simonw 9599 | 2021-03-15T14:47:31Z | 2021-03-15T14:47:31Z | OWNER | This is a smart feature. I have something that does this in Datasette, extracting it out to https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/init.py#L818-L829 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Escaping FTS search strings 831751367 | |
799066252 | https://github.com/simonw/datasette/issues/236#issuecomment-799066252 | https://api.github.com/repos/simonw/datasette/issues/236 | MDEyOklzc3VlQ29tbWVudDc5OTA2NjI1Mg== | simonw 9599 | 2021-03-15T03:34:52Z | 2021-03-15T03:34:52Z | OWNER | Yeah the Lambda Docker stuff is pretty odd - you still don't get to speak HTTP, you have to speak their custom event protocol instead. https://github.com/glassechidna/serverlessish looks interesting here - it adds a proxy inside the container which allows your existing HTTP Docker image to run within Docker-on-Lambda. I've not tried it out yet though. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette publish lambda plugin 317001500 | |
799003172 | https://github.com/simonw/datasette/issues/236#issuecomment-799003172 | https://api.github.com/repos/simonw/datasette/issues/236 | MDEyOklzc3VlQ29tbWVudDc5OTAwMzE3Mg== | jacobian 21148 | 2021-03-14T23:42:57Z | 2021-03-14T23:42:57Z | CONTRIBUTOR | Oh, and the container image can be up to 10GB, so the EFS step might not be needed except for pretty big stuff. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette publish lambda plugin 317001500 | |
799002993 | https://github.com/simonw/datasette/issues/236#issuecomment-799002993 | https://api.github.com/repos/simonw/datasette/issues/236 | MDEyOklzc3VlQ29tbWVudDc5OTAwMjk5Mw== | jacobian 21148 | 2021-03-14T23:41:51Z | 2021-03-14T23:41:51Z | CONTRIBUTOR | Now that Lambda supports Docker, this probably is a bit easier and may be able to build on top of the existing package command. There are weirdnesses in how the command actually gets invoked; the aws-lambda-python image shows a bit of that. So Datasette would probably need some sort of Lambda-specific entry point to make this work. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette publish lambda plugin 317001500 | |
798913090 | https://github.com/simonw/datasette/pull/1260#issuecomment-798913090 | https://api.github.com/repos/simonw/datasette/issues/1260 | MDEyOklzc3VlQ29tbWVudDc5ODkxMzA5MA== | codecov[bot] 22429695 | 2021-03-14T14:01:30Z | 2021-03-14T14:01:30Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1260 +/-=======================================
Coverage 91.51% 91.51% | Impacted Files | Coverage Δ | |
|---|---|---|
| datasette/inspect.py | Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fix: code quality issues 831163537 | |
798468572 | https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798468572 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDc5ODQ2ODU3Mg== | n8henrie 1234956 | 2021-03-13T14:47:31Z | 2021-03-13T14:47:31Z | NONE | Ok, new PR works. I'm not I still end up with a lot of activities that are missing an
I also end up with some unhappy characters (in the skipped events), such as: But it's successfully making it through the file, and the resulting db opens in datasette, so I'd call that progress. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UNIQUE constraint failed: workouts.id 771608692 | |
798436026 | https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798436026 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDc5ODQzNjAyNg== | n8henrie 1234956 | 2021-03-13T14:23:16Z | 2021-03-13T14:23:16Z | NONE | This PR allows my import to succeed. It looks like some events don't have an If a record has neither of these, I changed it to just print the record (for debugging) and For some odd reason this ran fine at first, and now (after removing the generated db and trying again) I'm getting a different error (duplicate column name). Looks like it may have run when I had two successive runs without remembering to delete the db in between. Will try to refactor. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UNIQUE constraint failed: workouts.id 771608692 | |
797827038 | https://github.com/simonw/datasette/issues/1259#issuecomment-797827038 | https://api.github.com/repos/simonw/datasette/issues/1259 | MDEyOklzc3VlQ29tbWVudDc5NzgyNzAzOA== | simonw 9599 | 2021-03-13T00:15:40Z | 2021-03-13T00:15:40Z | OWNER | If all of the facets were being calculated in a single query, I'd be willing to bump the facet time limit up to something a lot higher, maybe even a full second. There's a chance that could work amazingly well with a materialized CTE. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research using CTEs for faster facet counts 830567275 | |
797804869 | https://github.com/simonw/datasette/issues/1259#issuecomment-797804869 | https://api.github.com/repos/simonw/datasette/issues/1259 | MDEyOklzc3VlQ29tbWVudDc5NzgwNDg2OQ== | simonw 9599 | 2021-03-12T23:05:05Z | 2021-03-12T23:05:05Z | OWNER | I wonder if I could optimize facet suggestion in the same way? One challenge: the query time limit will apply to the full CTE query, not to the individual columns. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research using CTEs for faster facet counts 830567275 | |
797801075 | https://github.com/simonw/datasette/issues/1259#issuecomment-797801075 | https://api.github.com/repos/simonw/datasette/issues/1259 | MDEyOklzc3VlQ29tbWVudDc5NzgwMTA3NQ== | simonw 9599 | 2021-03-12T22:53:56Z | 2021-03-12T22:55:16Z | OWNER | OK, a better comparison: https://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aowner+as+%28%0D%0Aselect%0D%0A++%27owner%27+as+col%2C+owner+as+value%2C+count%28%29+as+c+from+data+group+by+owner%0D%0A++order+by+c+desc+limit+31%0D%0A%29%0D%0Aselect++from+primary_fuel+union+select++from+country_long%0D%0Aunion+select++from+owner+order+by+col%2C+c+desc calculates facets against three columns. It takes 78.5ms* (and 34.5ms when I refreshed it, presumably after warming some SQLite caches of some sort). https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet=country_long&_facet=primary_fuel&_trace=1&_size=0 shows those facets with size=0 on the SQL query - and shows a SQL trace at the bottom of the page. The country_long facet query takes 45.36ms, owner takes 38.45ms, primary_fuel takes 49.04ms - so a total of 132.85ms That's against https://global-power-plants.datasettes.com/-/versions says SQLite 3.27.3 - so even on a SQLite version that doesn't materialize the CTEs there's a significant performance boost to doing all three facets in a single CTE query. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research using CTEs for faster facet counts 830567275 | |
797790017 | https://github.com/simonw/datasette/issues/1259#issuecomment-797790017 | https://api.github.com/repos/simonw/datasette/issues/1259 | MDEyOklzc3VlQ29tbWVudDc5Nzc5MDAxNw== | simonw 9599 | 2021-03-12T22:22:12Z | 2021-03-12T22:22:12Z | OWNER | https://sqlite.org/lang_with.html
It looks like this optimization is completely unavailable on SQLite prior to 3.35.0 (released 12th March 2021). But I could still rewrite the faceting to work in this way, using the exact same SQL - it would just be significantly faster on 3.35.0+ (assuming it's actually faster in practice - would need to benchmark). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research using CTEs for faster facet counts 830567275 | |
797159434 | https://github.com/simonw/datasette/issues/1193#issuecomment-797159434 | https://api.github.com/repos/simonw/datasette/issues/1193 | MDEyOklzc3VlQ29tbWVudDc5NzE1OTQzNA== | simonw 9599 | 2021-03-12T01:01:54Z | 2021-03-12T01:01:54Z | OWNER | DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research plugin hook for alternative database backends 787173276 | |
797159221 | https://github.com/simonw/datasette/issues/1250#issuecomment-797159221 | https://api.github.com/repos/simonw/datasette/issues/1250 | MDEyOklzc3VlQ29tbWVudDc5NzE1OTIyMQ== | simonw 9599 | 2021-03-12T01:01:17Z | 2021-03-12T01:01:17Z | OWNER | This is a duplicate of #1193. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: Plugin hook for alternative database connections 824067604 | |
797158641 | https://github.com/simonw/datasette/issues/670#issuecomment-797158641 | https://api.github.com/repos/simonw/datasette/issues/670 | MDEyOklzc3VlQ29tbWVudDc5NzE1ODY0MQ== | simonw 9599 | 2021-03-12T00:59:49Z | 2021-03-12T00:59:49Z | OWNER |
It looks like the answer to this is yes - I'll need users to setup read-only credentials. Here's a TIL about that: https://til.simonwillison.net/postgresql/read-only-postgresql-user |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
Prototoype for Datasette on PostgreSQL 564833696 | |
796854370 | https://github.com/simonw/datasette/pull/1211#issuecomment-796854370 | https://api.github.com/repos/simonw/datasette/issues/1211 | MDEyOklzc3VlQ29tbWVudDc5Njg1NDM3MA== | simonw 9599 | 2021-03-11T16:15:29Z | 2021-03-11T16:15:29Z | OWNER | Thanks very much for this - it's really comprehensive. I need to bake some of these patterns into my coding habits better! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use context manager instead of plain open 797649915 | |
795950636 | https://github.com/simonw/datasette/issues/838#issuecomment-795950636 | https://api.github.com/repos/simonw/datasette/issues/838 | MDEyOklzc3VlQ29tbWVudDc5NTk1MDYzNg== | tsibley 79913 | 2021-03-10T19:24:13Z | 2021-03-10T19:24:13Z | NONE | I think this could be solved by one of:
Option 1 seems like the easiest to me, if you can get away with never having to generate an absolute URL. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Incorrect URLs when served behind a proxy with base_url set 637395097 | |
795939998 | https://github.com/simonw/datasette/issues/838#issuecomment-795939998 | https://api.github.com/repos/simonw/datasette/issues/838 | MDEyOklzc3VlQ29tbWVudDc5NTkzOTk5OA== | tsibley 79913 | 2021-03-10T19:16:55Z | 2021-03-10T19:16:55Z | NONE | Nod. The problem with the tests is that they're ignoring the origin (hostname, port) of links. In a reverse proxy situation, the frontend request origin is different than the backend request origin. The problem is Datasette generates links with the backend request origin. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Incorrect URLs when served behind a proxy with base_url set 637395097 | |
795918377 | https://github.com/simonw/datasette/issues/838#issuecomment-795918377 | https://api.github.com/repos/simonw/datasette/issues/838 | MDEyOklzc3VlQ29tbWVudDc5NTkxODM3Nw== | simonw 9599 | 2021-03-10T19:01:48Z | 2021-03-10T19:01:48Z | OWNER | The biggest challenge here I think is to replicate the exact situation here this happens in a Python unit test. The fix should be easy once we have a test in place. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Incorrect URLs when served behind a proxy with base_url set 637395097 | |
795895436 | https://github.com/simonw/datasette/issues/838#issuecomment-795895436 | https://api.github.com/repos/simonw/datasette/issues/838 | MDEyOklzc3VlQ29tbWVudDc5NTg5NTQzNg== | simonw 9599 | 2021-03-10T18:44:46Z | 2021-03-10T18:44:57Z | OWNER | Let's reopen this. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Incorrect URLs when served behind a proxy with base_url set 637395097 | |
795893813 | https://github.com/simonw/datasette/issues/838#issuecomment-795893813 | https://api.github.com/repos/simonw/datasette/issues/838 | MDEyOklzc3VlQ29tbWVudDc5NTg5MzgxMw== | tsibley 79913 | 2021-03-10T18:43:39Z | 2021-03-10T18:43:39Z | NONE | @simonw Unfortunately this issue as I reported it is not actually solved in version 0.55. Every link which is returned by the What I wrote originally still stands:
Would you prefer to re-open this issue or have me create a new one? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Incorrect URLs when served behind a proxy with base_url set 637395097 | |
795870524 | https://github.com/simonw/datasette/pull/1254#issuecomment-795870524 | https://api.github.com/repos/simonw/datasette/issues/1254 | MDEyOklzc3VlQ29tbWVudDc5NTg3MDUyNA== | simonw 9599 | 2021-03-10T18:27:45Z | 2021-03-10T18:27:45Z | OWNER | What other breaks did you spot? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions 826613352 | |
795869144 | https://github.com/simonw/datasette/pull/1256#issuecomment-795869144 | https://api.github.com/repos/simonw/datasette/issues/1256 | MDEyOklzc3VlQ29tbWVudDc5NTg2OTE0NA== | simonw 9599 | 2021-03-10T18:26:46Z | 2021-03-10T18:26:46Z | OWNER | Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Minor type in IP adress 827341657 | |
795112935 | https://github.com/simonw/datasette/pull/1256#issuecomment-795112935 | https://api.github.com/repos/simonw/datasette/issues/1256 | MDEyOklzc3VlQ29tbWVudDc5NTExMjkzNQ== | JBPressac 6371750 | 2021-03-10T08:59:45Z | 2021-03-10T08:59:45Z | CONTRIBUTOR | Sorry, I meant "minor typo" not "minor type". |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Minor type in IP adress 827341657 | |
795085921 | https://github.com/simonw/datasette/pull/1256#issuecomment-795085921 | https://api.github.com/repos/simonw/datasette/issues/1256 | MDEyOklzc3VlQ29tbWVudDc5NTA4NTkyMQ== | codecov[bot] 22429695 | 2021-03-10T08:35:17Z | 2021-03-10T08:35:17Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1256 +/-=======================================
Coverage 91.56% 91.56% Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Minor type in IP adress 827341657 | |
794518438 | https://github.com/simonw/datasette/pull/1254#issuecomment-794518438 | https://api.github.com/repos/simonw/datasette/issues/1254 | MDEyOklzc3VlQ29tbWVudDc5NDUxODQzOA== | durkie 3200608 | 2021-03-09T22:04:23Z | 2021-03-09T22:04:23Z | NONE | Dang, you're absolutely right. Spatialite 5.0 had been working fine for a plugin I was developing, but it apparently is broken in several other ways. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions 826613352 | |
794441034 | https://github.com/simonw/datasette/pull/1254#issuecomment-794441034 | https://api.github.com/repos/simonw/datasette/issues/1254 | MDEyOklzc3VlQ29tbWVudDc5NDQ0MTAzNA== | codecov[bot] 22429695 | 2021-03-09T20:54:18Z | 2021-03-09T21:12:15Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1254 +/-==========================================
- Coverage 91.56% 91.51% -0.05% | Impacted Files | Coverage Δ | |
|---|---|---|
| datasette/database.py | Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions 826613352 | |
794443710 | https://github.com/simonw/datasette/pull/1254#issuecomment-794443710 | https://api.github.com/repos/simonw/datasette/issues/1254 | MDEyOklzc3VlQ29tbWVudDc5NDQ0MzcxMA== | durkie 3200608 | 2021-03-09T20:56:45Z | 2021-03-09T20:56:45Z | NONE | Oh wow I didn't even see that you had opened an issue about this so recently. I'll check on |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions 826613352 | |
794439632 | https://github.com/simonw/datasette/pull/1254#issuecomment-794439632 | https://api.github.com/repos/simonw/datasette/issues/1254 | MDEyOklzc3VlQ29tbWVudDc5NDQzOTYzMg== | simonw 9599 | 2021-03-09T20:53:02Z | 2021-03-09T20:53:02Z | OWNER | Thanks for catching that documentation update! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions 826613352 | |
794437715 | https://github.com/simonw/datasette/pull/1254#issuecomment-794437715 | https://api.github.com/repos/simonw/datasette/issues/1254 | MDEyOklzc3VlQ29tbWVudDc5NDQzNzcxNQ== | simonw 9599 | 2021-03-09T20:51:19Z | 2021-03-09T20:51:19Z | OWNER | Did you see my note on https://github.com/simonw/datasette/issues/1249#issuecomment-792384382 about a weird issue I was having with the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions 826613352 | |
793308483 | https://github.com/simonw/datasette/pull/1252#issuecomment-793308483 | https://api.github.com/repos/simonw/datasette/issues/1252 | MDEyOklzc3VlQ29tbWVudDc5MzMwODQ4Mw== | codecov[bot] 22429695 | 2021-03-09T03:06:10Z | 2021-03-09T03:06:10Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1252 +/-==========================================
- Coverage 91.56% 91.51% -0.05% | Impacted Files | Coverage Δ | |
|---|---|---|
| datasette/database.py | Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add back styling to lists within table cells (fixes #1141) 825217564 | |
792386484 | https://github.com/simonw/datasette/issues/1250#issuecomment-792386484 | https://api.github.com/repos/simonw/datasette/issues/1250 | MDEyOklzc3VlQ29tbWVudDc5MjM4NjQ4NA== | simonw 9599 | 2021-03-08T00:29:06Z | 2021-03-08T00:29:06Z | OWNER | DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research: Plugin hook for alternative database connections 824067604 | |
792385274 | https://github.com/simonw/datasette/issues/1248#issuecomment-792385274 | https://api.github.com/repos/simonw/datasette/issues/1248 | MDEyOklzc3VlQ29tbWVudDc5MjM4NTI3NA== | simonw 9599 | 2021-03-08T00:25:10Z | 2021-03-08T00:25:10Z | OWNER | It's not possible yet, unfortunately. This came up on the forums recently: https://github.com/simonw/datasette/discussions/968 I'm leaning further towards making the database connection layer itself work via a plugin hook, which would open up the possibility of supporting DuckDB and other databases as well. I've not committed to doing this yet though. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
duckdb database (very low performance in SQLite) 823035080 | |
792384854 | https://github.com/simonw/datasette/issues/1249#issuecomment-792384854 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDc5MjM4NDg1NA== | simonw 9599 | 2021-03-08T00:23:38Z | 2021-03-08T00:23:38Z | OWNER | One reason to prioritize this issue: Homebrew upgraded to SpatiaLite 5.0 recently https://formulae.brew.sh/formula/spatialite-tools and as a result SpatiaLite database created on my laptop don't appear to be compatible with Datasette when published using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
792384382 | https://github.com/simonw/datasette/issues/1249#issuecomment-792384382 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDc5MjM4NDM4Mg== | simonw 9599 | 2021-03-08T00:22:02Z | 2021-03-08T00:22:02Z | OWNER | I tried this patch against # Setup build dependencies RUN apt update \ -&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \ - && apt clean + && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \ + libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \ + && apt clean - -RUN wget "https://www.sqlite.org/2020/sqlite-autoconf-3310100.tar.gz" && tar xzf sqlite-autoconf-3310100.tar.gz \ - && cd sqlite-autoconf-3310100 && ./configure --disable-static --enable-fts5 --enable-json1 CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \ +RUN wget "https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz" && tar xzf sqlite-autoconf-3340100.tar.gz \ + && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \ + CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \ && make && make install -RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-sources/freexl-1.0.5.tar.gz" && tar zxf freexl-1.0.5.tar.gz \ - && cd freexl-1.0.5 && ./configure && make && make install +RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz" && tar zxf freexl-1.0.6.tar.gz \ + && cd freexl-1.0.6 && ./configure && make && make install -RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-4.4.0-RC0.tar.gz" && tar zxf libspatialite-4.4.0-RC0.tar.gz \ - && cd libspatialite-4.4.0-RC0 && ./configure && make && make install +RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz" && tar zxf libspatialite-5.0.1.tar.gz \ + && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install RUN wget "http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install -RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/spatialite-tools-4.4.0-RC0.tar.gz" && tar zxf spatialite-tools-4.4.0-RC0.tar.gz \ - && cd spatialite-tools-4.4.0-RC0 && ./configure && make && make install +RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz" && tar zxf spatialite-tools-5.0.0.tar.gz \ + && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install # Add local code to the image instead of fetching from pypi. @@ -27,7 +28,7 @@ COPY . /datasette RUN pip install /datasette -FROM python:3.7.10-slim-stretch +FROM python:3.9.2-slim-buster # Copy python dependencies and spatialite libraries COPY --from=build /usr/local/lib/ /usr/local/lib/ ``` I had to use This works, sort of... I'm getting a weird issue where the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
792383956 | https://github.com/simonw/datasette/issues/1249#issuecomment-792383956 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDc5MjM4Mzk1Ng== | simonw 9599 | 2021-03-08T00:20:09Z | 2021-03-08T00:20:09Z | OWNER | Worth noting that the Docker image used by https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/init.py#L349-L353 Where the apt extras for SpatiaLite are: https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/init.py#L344-L345
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
792308036 | https://github.com/simonw/datasette/issues/858#issuecomment-792308036 | https://api.github.com/repos/simonw/datasette/issues/858 | MDEyOklzc3VlQ29tbWVudDc5MjMwODAzNg== | robroc 1219001 | 2021-03-07T16:41:54Z | 2021-03-07T16:41:54Z | NONE | Apologies if I sound dense but I don't see where you would pass 'shell=True'. I'm using the CLI installed via pip. On Sun., Mar. 7, 2021, 2:15 a.m. David Smith, notifications@github.com wrote:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
publish heroku does not work on Windows 10 642388564 | |
792233255 | https://github.com/simonw/datasette/pull/1223#issuecomment-792233255 | https://api.github.com/repos/simonw/datasette/issues/1223 | MDEyOklzc3VlQ29tbWVudDc5MjIzMzI1NQ== | simonw 9599 | 2021-03-07T07:41:01Z | 2021-03-07T07:41:01Z | OWNER | This is fantastic, thanks so much for tracking this down. |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0 } |
Add compile option to Dockerfile to fix failing test (fixes #696) 806918878 | |
792230560 | https://github.com/simonw/datasette/issues/858#issuecomment-792230560 | https://api.github.com/repos/simonw/datasette/issues/858 | MDEyOklzc3VlQ29tbWVudDc5MjIzMDU2MA== | smithdc1 39445562 | 2021-03-07T07:14:58Z | 2021-03-07T07:14:58Z | NONE | To get it to work I had to:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
publish heroku does not work on Windows 10 642388564 | |
792129022 | https://github.com/simonw/datasette/issues/858#issuecomment-792129022 | https://api.github.com/repos/simonw/datasette/issues/858 | MDEyOklzc3VlQ29tbWVudDc5MjEyOTAyMg== | robroc 1219001 | 2021-03-07T00:23:34Z | 2021-03-07T00:23:34Z | NONE | @smithdc1 Can you tell us what you did to get it to publish in Windows? What commands did you pass? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
publish heroku does not work on Windows 10 642388564 | |
791509910 | https://github.com/simonw/datasette/issues/766#issuecomment-791509910 | https://api.github.com/repos/simonw/datasette/issues/766 | MDEyOklzc3VlQ29tbWVudDc5MTUwOTkxMA== | JBPressac 6371750 | 2021-03-05T15:57:35Z | 2021-03-05T16:35:21Z | CONTRIBUTOR | Hello, I have the same wildcards search problems with an instance of Datasette. http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwerz&_sort=rowid is OK but http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwe* is not (FTS is activated on "Reference" "IntituleAnalyse" "NomDuProducteur" "PresentationDuContenu" "Notes"). Notice that a SQL query as below launched directly from SQLite in the server's shell, retrieves results.
Thanks, |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Enable wildcard-searches by default 617323873 | |
791530093 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791530093 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MTUzMDA5Mw== | UtahDave 306240 | 2021-03-05T16:28:07Z | 2021-03-05T16:28:07Z | NONE |
@maxhawkins a limitation of the python mbox module is it loads the entire mbox into memory. I did find another approach to this problem that didn't use the builtin python mbox module and created a generator so that it didn't have to load the whole mbox into memory. I was hoping to use standard library modules, but this might be a good reason to investigate that approach a bit more. My worry is making sure a custom processor handles all the ins and outs of the mbox format correctly. Hm. As I'm writing this, I thought of something. I think I can parse each message one at a time, and then use an mbox function to load each message using the python mbox module. That way the mbox module can still deal with the specifics of the mbox format, but I can use a generator. I'll give that a try. Thanks for the feedback @maxhawkins and @simonw. I'll give that a try. @simonw can we hold off on merging this until I can test this new approach? |
{ "total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
791089881 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791089881 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MTA4OTg4MQ== | maxhawkins 28565 | 2021-03-05T02:03:19Z | 2021-03-05T02:03:19Z | NONE | I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout. Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
791053721 | https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-791053721 | https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32 | MDEyOklzc3VlQ29tbWVudDc5MTA1MzcyMQ== | dsisnero 6213 | 2021-03-05T00:31:27Z | 2021-03-05T00:31:27Z | NONE | I am getting the same thing for US West (N. California) us-west-1 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
KeyError: 'Contents' on running upload 803333769 | |
790934616 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790934616 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc5MDkzNDYxNg== | Btibert3 203343 | 2021-03-04T20:54:44Z | 2021-03-04T20:54:44Z | NONE | Sorry for the delay, I got sidetracked after class last night. I am getting the following error: ``` /content# google-takeout-to-sqlite mbox takeout.db Takeout/Mail/gmail.mbox Usage: google-takeout-to-sqlite [OPTIONS] COMMAND [ARGS]...Try 'google-takeout-to-sqlite --help' for help. Error: No such command 'mbox'. ``` On the box, I installed with pip after cloning: https://github.com/UtahDave/google-takeout-to-sqlite.git |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature Request: Gmail 778380836 | |
790857004 | https://github.com/simonw/datasette/issues/1238#issuecomment-790857004 | https://api.github.com/repos/simonw/datasette/issues/1238 | MDEyOklzc3VlQ29tbWVudDc5MDg1NzAwNA== | tsibley 79913 | 2021-03-04T19:06:55Z | 2021-03-04T19:06:55Z | NONE | @rgieseke Ah, that's super helpful. Thank you for the workaround for now! |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Custom pages don't work with base_url setting 813899472 | |
790695126 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790695126 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDY5NTEyNg== | simonw 9599 | 2021-03-04T15:20:42Z | 2021-03-04T15:20:42Z | MEMBER | I'm not sure why but my most recent import, when displayed in Datasette, looks like this: Sorting by |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790693674 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790693674 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDY5MzY3NA== | simonw 9599 | 2021-03-04T15:18:36Z | 2021-03-04T15:18:36Z | MEMBER | I imported my 10GB mbox with 750,000 emails in it, ran this tool (with a hacked fix for the blob column problem) - and now a search that returns 92 results takes 25.37ms! This is fantastic. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790669767 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790669767 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDY2OTc2Nw== | simonw 9599 | 2021-03-04T14:46:06Z | 2021-03-04T14:46:06Z | MEMBER | Solution could be to pre-process that string by splitting on |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790668263 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790668263 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDY2ODI2Mw== | simonw 9599 | 2021-03-04T14:43:58Z | 2021-03-04T14:43:58Z | MEMBER | I added this code to output a message ID on errors:
This was for the following error:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790391711 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790391711 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM5MTcxMQ== | UtahDave 306240 | 2021-03-04T07:36:24Z | 2021-03-04T07:36:24Z | NONE |
Ah, that's good to know. I think explicitly creating the tables will be a great improvement. I'll add that. Also, I noticed after I opened this PR that the Thanks for the feedback. I should have time tomorrow to put together some improvements. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790389335 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790389335 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM4OTMzNQ== | UtahDave 306240 | 2021-03-04T07:32:04Z | 2021-03-04T07:32:04Z | NONE |
The wait is from python loading the mbox file. This happens regardless if you're getting the length of the mbox. The mbox module is on the slow side. It is possible to do one's own parsing of the mbox, but I kind of wanted to avoid doing that. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790384087 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/6#issuecomment-790384087 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDc5MDM4NDA4Nw== | simonw 9599 | 2021-03-04T07:22:51Z | 2021-03-04T07:22:51Z | MEMBER | 3 also mentions the conflicting version with other tools. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Upgrade to latest sqlite-utils 821841046 | |
790380839 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ== | simonw 9599 | 2021-03-04T07:17:05Z | 2021-03-04T07:17:05Z | MEMBER | Looks like you're doing this:
I imagine the reason the column is a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790379629 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790379629 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3OTYyOQ== | simonw 9599 | 2021-03-04T07:14:41Z | 2021-03-04T07:14:41Z | MEMBER | Confirmed: removing the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790378658 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790378658 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3ODY1OA== | simonw 9599 | 2021-03-04T07:12:48Z | 2021-03-04T07:12:48Z | MEMBER | It looks like the If I It would be great if we could store the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790373024 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790373024 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3MzAyNA== | simonw 9599 | 2021-03-04T07:01:58Z | 2021-03-04T07:04:06Z | MEMBER | I got 9 warnings that look like this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790372621 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790372621 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3MjYyMQ== | simonw 9599 | 2021-03-04T07:01:18Z | 2021-03-04T07:01:18Z | MEMBER | I'm not sure if it would work, but there is an alternative pattern for showing a progress bar against a really large file that I've used in https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/cli.py#L24-L57 and https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/utils.py#L4-L19 (the It can be a bit of a convoluted pattern, and I'm not at all sure it would work for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790370485 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ== | simonw 9599 | 2021-03-04T06:57:25Z | 2021-03-04T06:57:48Z | MEMBER | The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count: I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790369076 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790369076 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDM2OTA3Ng== | simonw 9599 | 2021-03-04T06:54:46Z | 2021-03-04T06:54:46Z | MEMBER | The Rich-powered progress bar is pretty: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790312268 | https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790312268 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 | MDEyOklzc3VlQ29tbWVudDc5MDMxMjI2OA== | simonw 9599 | 2021-03-04T05:48:16Z | 2021-03-04T05:48:16Z | MEMBER | Wow, my mbox is a 10.35 GB download! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
WIP: Add Gmail takeout mbox import 813880401 | |
790311215 | https://github.com/simonw/datasette/pull/1243#issuecomment-790311215 | https://api.github.com/repos/simonw/datasette/issues/1243 | MDEyOklzc3VlQ29tbWVudDc5MDMxMTIxNQ== | simonw 9599 | 2021-03-04T05:45:57Z | 2021-03-04T05:45:57Z | OWNER | Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fix small typo 815955014 | |
790257263 | https://github.com/simonw/datasette/issues/268#issuecomment-790257263 | https://api.github.com/repos/simonw/datasette/issues/268 | MDEyOklzc3VlQ29tbWVudDc5MDI1NzI2Mw== | mhalle 649467 | 2021-03-04T03:20:23Z | 2021-03-04T03:20:23Z | NONE | It's kind of an ugly hack, but you can try out what using the fts5 table as an actual datasette-accessible table looks like without changing any datasette code by creating yet another view on top of the fts5 table:
That's now visible from datasette, just like any other view, but you can use This is only good as a proof of concept because you're inefficiently going from view -> fts5 external content table -> view -> data table. However, it does show it works. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for ranking results from SQLite full-text search 323718842 | |
790198930 | https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790198930 | https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDc5MDE5ODkzMA== | Btibert3 203343 | 2021-03-04T00:58:40Z | 2021-03-04T00:58:40Z | NONE | I am just seeing this sorry, yes! I will kick the tires later on tonight. My apologies for the delay. |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature Request: Gmail 778380836 | |
789680230 | https://github.com/simonw/datasette/issues/283#issuecomment-789680230 | https://api.github.com/repos/simonw/datasette/issues/283 | MDEyOklzc3VlQ29tbWVudDc4OTY4MDIzMA== | justinpinkney 605492 | 2021-03-03T12:28:42Z | 2021-03-03T12:28:42Z | NONE | One note on using this pragma I got an error on starting datasette I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support cross-database joins 325958506 | |
789409126 | https://github.com/simonw/datasette/issues/268#issuecomment-789409126 | https://api.github.com/repos/simonw/datasette/issues/268 | MDEyOklzc3VlQ29tbWVudDc4OTQwOTEyNg== | mhalle 649467 | 2021-03-03T03:57:15Z | 2021-03-03T03:58:40Z | NONE | In FTS5, I think doing an FTS search is actually much easier than doing a join against the main table like datasette does now. In fact, FTS5 external content tables provide a transparent interface back to the original table or view. Here's what I'm currently doing:
* build a view that joins whatever tables I want and rename the columns to non-joiny names (e.g, Unfortunately, datasette doesn't currently seem happy being coerced into doing a real query on an fts5 table. This works:
But this doesn't work in the datasette SQL query interface:
For what datasette is doing right now, I think you could just use contentless fts5 tables ( I guess if you want to follow this suggestion, you'd need a somewhat different code path for fts5. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Mechanism for ranking results from SQLite full-text search 323718842 | |
789186458 | https://github.com/simonw/datasette/issues/1238#issuecomment-789186458 | https://api.github.com/repos/simonw/datasette/issues/1238 | MDEyOklzc3VlQ29tbWVudDc4OTE4NjQ1OA== | rgieseke 198537 | 2021-03-02T20:19:30Z | 2021-03-02T20:19:30Z | CONTRIBUTOR | A custom |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Custom pages don't work with base_url setting 813899472 | |
787616446 | https://github.com/simonw/datasette/issues/1247#issuecomment-787616446 | https://api.github.com/repos/simonw/datasette/issues/1247 | MDEyOklzc3VlQ29tbWVudDc4NzYxNjQ0Ng== | simonw 9599 | 2021-03-01T03:50:37Z | 2021-03-01T03:50:37Z | OWNER | I like the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette.add_memory_database() method 818430405 | |
787616158 | https://github.com/simonw/datasette/issues/1247#issuecomment-787616158 | https://api.github.com/repos/simonw/datasette/issues/1247 | MDEyOklzc3VlQ29tbWVudDc4NzYxNjE1OA== | simonw 9599 | 2021-03-01T03:49:27Z | 2021-03-01T03:49:27Z | OWNER | A couple of options: ```python datasette.add_memory_database("test_json_array") or make that first argument to add_database() optional and support:datasette.add_database(memory_name="test_json_array") ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette.add_memory_database() method 818430405 | |
787611153 | https://github.com/simonw/datasette/issues/1246#issuecomment-787611153 | https://api.github.com/repos/simonw/datasette/issues/1246 | MDEyOklzc3VlQ29tbWVudDc4NzYxMTE1Mw== | simonw 9599 | 2021-03-01T03:30:57Z | 2021-03-01T03:30:57Z | OWNER | I'm going to try a new pattern for testing this, enabled by #1151 - the test will create a new named in-memory database, write some records to it and then run some test facets against that. This will save me from having to add yet another fixtures table for this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Suggest for ArrayFacet possibly confused by blank values 817597268 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30