{"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-803697546", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 803697546, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T00:59:47Z", "updated_at": "2021-03-22T00:59:47Z", "author_association": "OWNER", "body": "To debug I'm running:\r\n\r\n docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash\r\n\r\nThis gets me a shell I can use.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-803697211", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 803697211, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T00:58:01Z", "updated_at": "2021-03-22T00:58:01Z", "author_association": "OWNER", "body": "I'm messing around with the `Dockerfile` and after each change I'm running:\r\n\r\n docker build . -t datasette-spatialite\r\n\r\nAnd then:\r\n\r\n docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-803694661", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 803694661, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T00:46:49Z", "updated_at": "2021-03-22T00:46:49Z", "author_association": "OWNER", "body": "Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-803694436", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 803694436, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY5NDQzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T00:46:00Z", "updated_at": "2021-03-22T00:46:00Z", "author_association": "OWNER", "body": "So I'm going to try `20.10` and see where that gets me.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-803694359", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 803694359, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T00:45:47Z", "updated_at": "2021-03-22T00:45:47Z", "author_association": "OWNER", "body": "https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using `python:3.9-slim-buster` or `ubuntu:20.04` - but 20.04 is focal which still has SpatiaLite `4.3.0a-6build1` - It's `20.10` that has 5.0: https://packages.ubuntu.com/groovy/libspatialite-dev", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-803693181", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 803693181, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY5MzE4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T00:41:02Z", "updated_at": "2021-03-22T00:41:02Z", "author_association": "OWNER", "body": "Debian sid has it too: https://packages.debian.org/sid/libspatialite-dev", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-803692673", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 803692673, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T00:38:42Z", "updated_at": "2021-03-22T00:38:42Z", "author_association": "OWNER", "body": "Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-803691236", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 803691236, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-22T00:32:03Z", "updated_at": "2021-03-22T00:32:03Z", "author_association": "OWNER", "body": "Here's something odd: when I run `datasette tuscany_housenumbers.sqlite --load-extension=spatialite` on macOS against SpatiaLite installed using Homebrew (which reports `\"spatialite\": \"5.0.0\"` on the `/-/versions` page) I don't get any weird errors at all, everything works fine.\r\n\r\nBut when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages.\r\n\r\nThis is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-803674728", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 803674728, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY3NDcyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-21T22:55:31Z", "updated_at": "2021-03-21T22:55:31Z", "author_association": "OWNER", "body": "CTEs were added in 2014-02-03 SQLite 3.8.3 - so I think it's OK to depend on them for Datasette.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/647#issuecomment-803673225", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/647", "id": 803673225, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzY3MzIyNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-21T22:44:19Z", "updated_at": "2021-03-21T22:44:19Z", "author_association": "OWNER", "body": "Now that I'm looking at refactoring how views work in #878 it's clear that the gnarliest, most convoluted code I need to deal with relates to this old feature.\r\n\r\nI'm going to remove it entirely. Any performance enhancement or provides can be achieved just as well by using regular URLs and a caching proxy.\r\n\r\nI may provide a 404 handling plugin that attempts to rewrite old URLs that used this mechanism, but I won't do any more than that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 531755959, "label": "Move hashed URL mode out to a plugin"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/942#issuecomment-803631102", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/942", "id": 803631102, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzYzMTEwMg==", "user": {"value": 192568, "label": "mroswell"}, "created_at": "2021-03-21T17:48:42Z", "updated_at": "2021-03-21T17:48:42Z", "author_association": "CONTRIBUTOR", "body": "I like this idea. Though it might be nice to have some kind of automated system from database to file, so that developers could easily track diffs.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681334912, "label": "Support column descriptions in metadata.json"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803502424", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/249", "id": 803502424, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzUwMjQyNA==", "user": {"value": 36287, "label": "prabhur"}, "created_at": "2021-03-21T02:43:32Z", "updated_at": "2021-03-21T02:43:32Z", "author_association": "NONE", "body": "> Did you run `enable-fts` before you inserted the data?\r\n> \r\n> If so you'll need to run `populate-fts` after the insert to populate the FTS index.\r\n> \r\n> A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.\r\n\r\nWow. Wasn't expecting a response this quick, especially during a weekend. :-) Sincerely appreciate it.\r\nI tried the `populate-fts` and that did the trick. My bad for not consulting the docs again. I think I forgot to add that step when I automated the workflow.\r\nThanks for the suggestion. I'll close this issue. Have a great weekend and many many thanks for creating these suite of tools around sqlite.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 836963850, "label": "Full text search possibly broken?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803501756", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/249", "id": 803501756, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzUwMTc1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-21T02:33:45Z", "updated_at": "2021-03-21T02:33:45Z", "author_association": "OWNER", "body": "Did you run `enable-fts` before you inserted the data?\r\n\r\nIf so you'll need to run `populate-fts` after the insert to populate the FTS index.\r\n\r\nA better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 836963850, "label": "Full text search possibly broken?"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1261#issuecomment-803499509", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1261", "id": 803499509, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ5OTUwOQ==", "user": {"value": 812795, "label": "brimstone"}, "created_at": "2021-03-21T02:06:43Z", "updated_at": "2021-03-21T02:06:43Z", "author_association": "NONE", "body": "I can confirm 0.9.2 fixes the problem. Thanks for the fast response!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 832092321, "label": "Some links aren't properly URL encoded."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-803473015", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 803473015, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T22:33:05Z", "updated_at": "2021-03-20T22:33:05Z", "author_association": "OWNER", "body": "Things this mechanism needs to be able to support:\r\n\r\n- Returning a default JSON representation\r\n- Defining \"extra\" JSON representations blocks, which can be requested using `?_extra=`\r\n- Returning rendered HTML, based on the default JSON + one or more extras + a template\r\n- Using Datasette output renderers to return e.g. CSV data\r\n- Potentially also supporting streaming output renderers for streaming CSV/TSV/JSON-nl etc", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-803472595", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 803472595, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T22:28:12Z", "updated_at": "2021-03-20T22:28:12Z", "author_association": "OWNER", "body": "Another idea I had: a view is a class that takes the `datasette` instance in its constructor, and defines a `__call__` method that accepts a request and returns a response. Except `await __call__` looks like it might be a bit messy, discussion in https://github.com/encode/starlette/issues/886", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-803472278", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 803472278, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ3MjI3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T22:25:04Z", "updated_at": "2021-03-20T22:25:04Z", "author_association": "OWNER", "body": "I came up with a slightly wild idea for this that would involve pytest-style dependency injection.\r\n\r\nPrototype here: https://gist.github.com/simonw/496b24fdad44f6f8b7237fe394a0ced7\r\n\r\nCopying from my private notes:\r\n\r\n> Use the lazy evaluated DI mechanism to break up table view into different pieces eg for faceting\r\n> \r\n> Use that to solve JSON vs HTML views\r\n> \r\n> Oh here's an idea: what if the various components of the table view were each defined as async functions.... and then executed using asyncio.gather in order to run the SQL queries in parallel? Then try benchmarking with different numbers of threads?\r\n> \r\n> The async_call_with_arguments function could do this automatically for any awaitable dependencies\r\n> \r\n> This would give me massively parallel dependency injection\r\n> \r\n> (I could build an entire framework around this and call it c64)\r\n> \r\n> Idea: arguments called eg \"count\" are executed and the result passed to the function. If called count_fn then a reference to the not-yet-called function is passed instead \r\n> \r\n> I'm not going to completely combine the views mechanism and the render hooks. Instead, the core view will define a bunch of functions used to compose the page and the render hook will have conditional access to those functions - which will otherwise be asyncio.gather executed directly by the HTML page version\r\n> \r\n> Using asyncio.gather to execute facets and suggest facets in parallel would be VERY interesting \r\n> \r\n> suggest facets should be VERY cachable - doesn't matter if it's wrong unlike actual facets themselves\r\n> \r\n> What if all Datasette views were defined in terms of dependency injection - and those dependency functions could themselves depend on others just like pytest fixtures. Everything would become composable and async stuff could execute in parallel\r\n> \r\n> FURTHER IDEA: use this for the ?_extra= mechanism as well.\r\n> \r\n> Any view in Datasette can be defined as a collection of named keys. Each of those keys maps to a function or an async function that accepts as input other named keys, using DI to handle them.\r\n> \r\n> The HTML view is a defined function. So are the other outputs.\r\n> \r\n> Default original inputs include \u201crequest\u201d and \u201cdatasette\u201d.\r\n> \r\n> So\u2026 maybe a view function is a class methods that use DI. One of those methods as an .html() method used for the default page.\r\n> \r\n> Output formats are a bit more complicated because they are supposed to be defined separately in plugins. They are unified across query, row and table though.\r\n> \r\n> I\u2019m going to try breaking up the TableView to see what happens.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 1}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-803471917", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 803471917, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T22:21:33Z", "updated_at": "2021-03-20T22:21:33Z", "author_association": "OWNER", "body": "This has been blocking things for too long.\r\n\r\nIf this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1258#issuecomment-803471702", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1258", "id": 803471702, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ3MTcwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T22:19:39Z", "updated_at": "2021-03-20T22:19:39Z", "author_association": "OWNER", "body": "This is a good idea. I avoided this initially because it should be possible to run a canned query with a parameter set to the empty string, but that view could definitely be smart enough to differentiate between `?sql=...¶m=` and `?sql=` with no `param` specified at all.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 828858421, "label": "Allow canned query params to specify default values"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-803469623", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 803469623, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ2OTYyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T22:01:23Z", "updated_at": "2021-03-20T22:01:23Z", "author_association": "OWNER", "body": "I'm going to keep `?_shape=array` working on the assumption that many existing uses of the Datasette API are already using that option, so it would be nice not to break them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1261#issuecomment-803468314", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1261", "id": 803468314, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ2ODMxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T21:48:48Z", "updated_at": "2021-03-20T21:48:48Z", "author_association": "OWNER", "body": "That's fixed in this release of `datasette-publish-vercel`: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.9.2", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 832092321, "label": "Some links aren't properly URL encoded."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1261#issuecomment-803466868", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1261", "id": 803466868, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ2Njg2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T21:36:06Z", "updated_at": "2021-03-20T21:36:06Z", "author_association": "OWNER", "body": "This isn't a Datasette bug - it's a Vercel bug: https://github.com/simonw/datasette-publish-vercel/issues/28\r\n\r\nI'm looking at a fix for that now, so watch that issue for updates.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 832092321, "label": "Some links aren't properly URL encoded."}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1266#issuecomment-803466730", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1266", "id": 803466730, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzQ2NjczMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-20T21:35:00Z", "updated_at": "2021-03-20T21:35:00Z", "author_association": "OWNER", "body": "https://docs.datasette.io/en/latest/internals.html#returning-a-response-with-asgi-send-send", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 836273891, "label": "Documentation for Response.asgi_send(send) method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1265#issuecomment-803160804", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1265", "id": 803160804, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzE2MDgwNA==", "user": {"value": 468612, "label": "yunzheng"}, "created_at": "2021-03-19T22:05:12Z", "updated_at": "2021-03-19T22:05:12Z", "author_association": "NONE", "body": "Wow that was fast! Thanks for this very cool project and quick update! \ud83d\udc4d ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 836123030, "label": "Support for HTTP Basic Authentication"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1265#issuecomment-803130332", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1265", "id": 803130332, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMzEzMDMzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-19T21:03:09Z", "updated_at": "2021-03-19T21:03:09Z", "author_association": "OWNER", "body": "This is now available in `datasette-auth-passwords` 0.4! https://github.com/simonw/datasette-auth-passwords/releases/tag/0.4", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 1, \"eyes\": 0}", "issue": {"value": 836123030, "label": "Support for HTTP Basic Authentication"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1265#issuecomment-802923254", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1265", "id": 802923254, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMjkyMzI1NA==", "user": {"value": 7476523, "label": "bobwhitelock"}, "created_at": "2021-03-19T15:39:15Z", "updated_at": "2021-03-19T15:39:15Z", "author_association": "CONTRIBUTOR", "body": "It doesn't use basic auth, but you can put a whole datasette instance, or parts of this, behind a username/password prompt using https://github.com/simonw/datasette-auth-passwords", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 836123030, "label": "Support for HTTP Basic Authentication"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1262#issuecomment-802164134", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1262", "id": 802164134, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMjE2NDEzNA==", "user": {"value": 19328961, "label": "henry501"}, "created_at": "2021-03-18T17:55:00Z", "updated_at": "2021-03-18T17:55:00Z", "author_association": "NONE", "body": "Thanks for the comments. I'll take a look at the documentation to familiarize myself, as I haven't tried to write any plugins yet. With some luck I might be ready to write it when the hook is implemented.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 834602299, "label": "Plugin hook that could support 'order by random()' for table view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1262#issuecomment-802099264", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1262", "id": 802099264, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMjA5OTI2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-18T16:43:09Z", "updated_at": "2021-03-18T16:43:09Z", "author_association": "OWNER", "body": "I often find myself wanting this too, when I'm exploring a new dataset.\r\n\r\ni agree with Bob that this is a good candidate for a plugin. The plugin system isn't quite setup for this yet though - there isn't an obvious mechanism for adding extra sort orders or other interface elements that manipulate the query used by the table view in some way.\r\n\r\nI'm going to promote this issue to status of a plugin hook feature request - I have a hunch that a plugin hook that enables `order by random()` could enable a lot of other useful plugin features too.", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 834602299, "label": "Plugin hook that could support 'order by random()' for table view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1262#issuecomment-802095132", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1262", "id": 802095132, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMjA5NTEzMg==", "user": {"value": 7476523, "label": "bobwhitelock"}, "created_at": "2021-03-18T16:37:45Z", "updated_at": "2021-03-18T16:37:45Z", "author_association": "CONTRIBUTOR", "body": "This sounds like a good use case for a plugin, since this will only be useful for a subset of Datasette users. It shouldn't be too difficult to add a button to do this with the available plugin hooks - have you taken a look at https://docs.datasette.io/en/latest/writing_plugins.html?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 834602299, "label": "Plugin hook that could support 'order by random()' for table view"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/159#issuecomment-802032152", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/159", "id": 802032152, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMjAzMjE1Mg==", "user": {"value": 1025224, "label": "limar"}, "created_at": "2021-03-18T15:42:52Z", "updated_at": "2021-03-18T15:42:52Z", "author_association": "NONE", "body": "I confirm the bug. Happens for me in version 3.6. I use the call to delete all the records:\r\n`table.delete_where()`\r\nThis does not delete anything. \r\n\r\nI see that `delete()` method DOES use context manager `with self.db.conn:` which should help. You may want to align the code of both methods.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 702386948, "label": ".delete_where() does not auto-commit (unlike .insert() or .upsert())"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/246#issuecomment-801816980", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/246", "id": 801816980, "node_id": "MDEyOklzc3VlQ29tbWVudDgwMTgxNjk4MA==", "user": {"value": 37962604, "label": "polyrand"}, "created_at": "2021-03-18T10:40:32Z", "updated_at": "2021-03-18T10:43:04Z", "author_association": "NONE", "body": "I have found a similar problem, but I only when using that type of query (with `*` for doing a prefix search). I'm also building something on top of FTS5/sqlite-utils, and the way I decided to handle it was creating a specific function for prefixes. According to [the docs](https://www2.sqlite.org/fts5.html#fts5_prefix_queries), the query can be done in this 2 ways:\r\n\r\n```sql\r\n... MATCH '\"one two thr\" * '\r\n... MATCH 'one + two + thr*'\r\n```\r\n\r\nI thought I could build a query like the first one using this function:\r\n\r\n```python\r\ndef prefix(query: str):\r\n return f'\"{query}\" *'\r\n```\r\n\r\nAnd then I use the output of that function as the query parameter for the standard `.search()` method in sqlite-utils.\r\n\r\nHowever, my use case is different because I'm the one \"deciding\" when to use a prefix search, not the end user. I also haven't done many tests, but maybe you found that useful. One thing I could think of is checking if the query has an `*` at the end, remove it and build the prefix query using the function above.\r\n\r\nThis is just for prefix queries, I think having the escaping function is still useful for other use cases.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 831751367, "label": "Escaping FTS search strings"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/246#issuecomment-799479175", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/246", "id": 799479175, "node_id": "MDEyOklzc3VlQ29tbWVudDc5OTQ3OTE3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-15T14:47:31Z", "updated_at": "2021-03-15T14:47:31Z", "author_association": "OWNER", "body": "This is a smart feature. I have something that does this in Datasette, extracting it out to `sqlite-utils` makes a lot of sense.\r\n\r\nhttps://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L818-L829", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 831751367, "label": "Escaping FTS search strings"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/236#issuecomment-799066252", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/236", "id": 799066252, "node_id": "MDEyOklzc3VlQ29tbWVudDc5OTA2NjI1Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-15T03:34:52Z", "updated_at": "2021-03-15T03:34:52Z", "author_association": "OWNER", "body": "Yeah the Lambda Docker stuff is pretty odd - you still don't get to speak HTTP, you have to speak their custom event protocol instead.\r\n\r\nhttps://github.com/glassechidna/serverlessish looks interesting here - it adds a proxy inside the container which allows your existing HTTP Docker image to run within Docker-on-Lambda. I've not tried it out yet though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 317001500, "label": "datasette publish lambda plugin"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/236#issuecomment-799003172", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/236", "id": 799003172, "node_id": "MDEyOklzc3VlQ29tbWVudDc5OTAwMzE3Mg==", "user": {"value": 21148, "label": "jacobian"}, "created_at": "2021-03-14T23:42:57Z", "updated_at": "2021-03-14T23:42:57Z", "author_association": "CONTRIBUTOR", "body": "Oh, and the container image can be up to 10GB, so the EFS step might not be needed except for pretty big stuff.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 317001500, "label": "datasette publish lambda plugin"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/236#issuecomment-799002993", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/236", "id": 799002993, "node_id": "MDEyOklzc3VlQ29tbWVudDc5OTAwMjk5Mw==", "user": {"value": 21148, "label": "jacobian"}, "created_at": "2021-03-14T23:41:51Z", "updated_at": "2021-03-14T23:41:51Z", "author_association": "CONTRIBUTOR", "body": "Now that [Lambda supports Docker](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/), this probably is a bit easier and may be able to build on top of the existing package command.\r\n\r\nThere are weirdnesses in how the command actually gets invoked; the [aws-lambda-python image](https://hub.docker.com/r/amazon/aws-lambda-python) shows a bit of that. So Datasette would probably need some sort of Lambda-specific entry point to make this work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 317001500, "label": "datasette publish lambda plugin"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1260#issuecomment-798913090", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1260", "id": 798913090, "node_id": "MDEyOklzc3VlQ29tbWVudDc5ODkxMzA5MA==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-03-14T14:01:30Z", "updated_at": "2021-03-14T14:01:30Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=h1) Report\n> Merging [#1260](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=desc) (90f5fb6) into [main](https://codecov.io/gh/simonw/datasette/commit/8e18c7943181f228ce5ebcea48deb59ce50bee1f?el=desc) (8e18c79) will **not change** coverage.\n> The diff coverage is `83.33%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1260/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=tree)\n\n```diff\n@@ Coverage Diff @@\n## main #1260 +/- ##\n=======================================\n Coverage 91.51% 91.51% \n=======================================\n Files 34 34 \n Lines 4255 4255 \n=======================================\n Hits 3894 3894 \n Misses 361 361 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=tree) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <0.00%> (\u00f8)` | |\n| [datasette/default\\_magic\\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <50.00%> (\u00f8)` | |\n| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.85% <100.00%> (\u00f8)` | |\n| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.01% <100.00%> (\u00f8)` | |\n| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.88% <100.00%> (\u00f8)` | |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=footer). Last update [8e18c79...90f5fb6](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 831163537, "label": "Fix: code quality issues"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798468572", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14", "id": 798468572, "node_id": "MDEyOklzc3VlQ29tbWVudDc5ODQ2ODU3Mg==", "user": {"value": 1234956, "label": "n8henrie"}, "created_at": "2021-03-13T14:47:31Z", "updated_at": "2021-03-13T14:47:31Z", "author_association": "NONE", "body": "Ok, new PR works. I'm not `git` enough so I just force-pushed over the old one.\r\n\r\nI still end up with a lot of activities that are missing an `id` and therefore skipped (since this is used as the primary key). For example:\r\n\r\n```\r\n{'workoutActivityType': 'HKWorkoutActivityTypeRunning', 'duration': '35.31666666666667', 'durationUnit': 'min', 'totalDistance': '4.010870267636999', 'totalDistanceUnit': 'mi', 'totalEnergyBurned': '660.3516235351562', 'totalEnergyBurnedUnit': 'Cal', 'sourceName': 'Strava', 'sourceVersion': '22810', 'creationDate': '2020-07-16 13:38:26 -0700', 'startDate': '2020-07-16 06:38:26 -0700', 'endDate': '2020-07-16 07:13:45 -0700'}\r\n```\r\n\r\nI also end up with some unhappy characters (in the skipped events), such as: `'sourceName': 'Nathan\u2019s Apple\\xa0Watch',`.\r\n\r\nBut it's successfully making it through the file, and the resulting db opens in datasette, so I'd call that progress.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771608692, "label": "UNIQUE constraint failed: workouts.id"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798436026", "issue_url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14", "id": 798436026, "node_id": "MDEyOklzc3VlQ29tbWVudDc5ODQzNjAyNg==", "user": {"value": 1234956, "label": "n8henrie"}, "created_at": "2021-03-13T14:23:16Z", "updated_at": "2021-03-13T14:23:16Z", "author_association": "NONE", "body": "This PR allows my import to succeed.\r\n\r\nIt looks like some events don't have an `id`, but do have `HKExternalUUID` (which gets turned into `metadata_HKExternalUUID`), so I use this as a fallback.\r\n\r\nIf a record has neither of these, I changed it to just print the record (for debugging) and `return`.\r\n\r\nFor some odd reason this ran fine at first, and now (after removing the generated db and trying again) I'm getting a different error (duplicate column name).\r\n\r\nLooks like it may have run when I had two successive runs without remembering to delete the db in between. Will try to refactor.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 771608692, "label": "UNIQUE constraint failed: workouts.id"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-797827038", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 797827038, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzgyNzAzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-13T00:15:40Z", "updated_at": "2021-03-13T00:15:40Z", "author_association": "OWNER", "body": "If all of the facets were being calculated in a single query, I'd be willing to bump the facet time limit up to something a lot higher, maybe even a full second. There's a chance that could work amazingly well with a materialized CTE.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-797804869", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 797804869, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzgwNDg2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T23:05:05Z", "updated_at": "2021-03-12T23:05:05Z", "author_association": "OWNER", "body": "I wonder if I could optimize facet suggestion in the same way?\r\n\r\nOne challenge: the query time limit will apply to the full CTE query, not to the individual columns.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-797801075", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 797801075, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzgwMTA3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T22:53:56Z", "updated_at": "2021-03-12T22:55:16Z", "author_association": "OWNER", "body": "OK, a better comparison:\r\n\r\nhttps://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28*%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28*%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aowner+as+%28%0D%0Aselect%0D%0A++%27owner%27+as+col%2C+owner+as+value%2C+count%28*%29+as+c+from+data+group+by+owner%0D%0A++order+by+c+desc+limit+31%0D%0A%29%0D%0Aselect+*+from+primary_fuel+union+select+*+from+country_long%0D%0Aunion+select+*+from+owner+order+by+col%2C+c+desc calculates facets against three columns. It takes **78.5ms** (and 34.5ms when I refreshed it, presumably after warming some SQLite caches of some sort).\r\n\r\nhttps://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet=country_long&_facet=primary_fuel&_trace=1&_size=0 shows those facets with size=0 on the SQL query - and shows a SQL trace at the bottom of the page.\r\n\r\nThe country_long facet query takes 45.36ms, owner takes 38.45ms, primary_fuel takes 49.04ms - so a total of 132.85ms\r\n\r\nThat's against https://global-power-plants.datasettes.com/-/versions says SQLite 3.27.3 - so even on a SQLite version that doesn't materialize the CTEs there's a significant performance boost to doing all three facets in a single CTE query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1259#issuecomment-797790017", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1259", "id": 797790017, "node_id": "MDEyOklzc3VlQ29tbWVudDc5Nzc5MDAxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T22:22:12Z", "updated_at": "2021-03-12T22:22:12Z", "author_association": "OWNER", "body": "https://sqlite.org/lang_with.html\r\n\r\n> Prior to SQLite 3.35.0, all CTEs where treated as if the NOT MATERIALIZED phrase was present\r\n\r\nIt looks like this optimization is completely unavailable on SQLite prior to 3.35.0 (released 12th March 2021). But I could still rewrite the faceting to work in this way, using the exact same SQL - it would just be significantly faster on 3.35.0+ (assuming it's actually faster in practice - would need to benchmark).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 830567275, "label": "Research using CTEs for faster facet counts"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1193#issuecomment-797159434", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1193", "id": 797159434, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzE1OTQzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T01:01:54Z", "updated_at": "2021-03-12T01:01:54Z", "author_association": "OWNER", "body": "DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python\r\n\r\n```python\r\nimport duckdb\r\ncon = duckdb.connect(database=\"/tmp/blah.db\", read_only=True)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 787173276, "label": "Research plugin hook for alternative database backends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1250#issuecomment-797159221", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1250", "id": 797159221, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzE1OTIyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T01:01:17Z", "updated_at": "2021-03-12T01:01:17Z", "author_association": "OWNER", "body": "This is a duplicate of #1193.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824067604, "label": "Research: Plugin hook for alternative database connections"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/670#issuecomment-797158641", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/670", "id": 797158641, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NzE1ODY0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-12T00:59:49Z", "updated_at": "2021-03-12T00:59:49Z", "author_association": "OWNER", "body": "> Challenge: what's the equivalent for PostgreSQL of opening a database in read only mode? Will I have to talk users through creating read only credentials?\r\n\r\nIt looks like the answer to this is yes - I'll need users to setup read-only credentials. Here's a TIL about that: https://til.simonwillison.net/postgresql/read-only-postgresql-user", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 564833696, "label": "Prototoype for Datasette on PostgreSQL"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1211#issuecomment-796854370", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1211", "id": 796854370, "node_id": "MDEyOklzc3VlQ29tbWVudDc5Njg1NDM3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-11T16:15:29Z", "updated_at": "2021-03-11T16:15:29Z", "author_association": "OWNER", "body": "Thanks very much for this - it's really comprehensive. I need to bake some of these patterns into my coding habits better!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 797649915, "label": "Use context manager instead of plain open"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/838#issuecomment-795950636", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/838", "id": 795950636, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTk1MDYzNg==", "user": {"value": 79913, "label": "tsibley"}, "created_at": "2021-03-10T19:24:13Z", "updated_at": "2021-03-10T19:24:13Z", "author_association": "NONE", "body": "I think this could be solved by one of:\r\n\r\n1. Stop generating absolute URLs, e.g. ones that include an origin. Relative URLs with absolute paths are fine, as long as they take `base_url` into account (as they do now, yay!).\r\n2. Extend `base_url` to include the expected frontend origin, and then use that information when generating absolute URLs.\r\n3. Document which HTTP headers the reverse proxy should set (e.g. the `X-Forwarded-*` family of conventional headers) to pass the frontend origin information to Datasette, and then use that information when generating absolute URLs.\r\n\r\nOption 1 seems like the easiest to me, if you can get away with never having to generate an absolute URL.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 637395097, "label": "Incorrect URLs when served behind a proxy with base_url set"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/838#issuecomment-795939998", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/838", "id": 795939998, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTkzOTk5OA==", "user": {"value": 79913, "label": "tsibley"}, "created_at": "2021-03-10T19:16:55Z", "updated_at": "2021-03-10T19:16:55Z", "author_association": "NONE", "body": "Nod. The problem with the tests is that they're ignoring the origin (hostname, port) of links. In a reverse proxy situation, the frontend request origin is different than the backend request origin. The problem is Datasette generates links with the backend request origin.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 637395097, "label": "Incorrect URLs when served behind a proxy with base_url set"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/838#issuecomment-795918377", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/838", "id": 795918377, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTkxODM3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-10T19:01:48Z", "updated_at": "2021-03-10T19:01:48Z", "author_association": "OWNER", "body": "The biggest challenge here I think is to replicate the exact situation here this happens in a Python unit test. The fix should be easy once we have a test in place.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 637395097, "label": "Incorrect URLs when served behind a proxy with base_url set"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/838#issuecomment-795895436", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/838", "id": 795895436, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTg5NTQzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-10T18:44:46Z", "updated_at": "2021-03-10T18:44:57Z", "author_association": "OWNER", "body": "Let's reopen this.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 637395097, "label": "Incorrect URLs when served behind a proxy with base_url set"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/838#issuecomment-795893813", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/838", "id": 795893813, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTg5MzgxMw==", "user": {"value": 79913, "label": "tsibley"}, "created_at": "2021-03-10T18:43:39Z", "updated_at": "2021-03-10T18:43:39Z", "author_association": "NONE", "body": "@simonw Unfortunately this issue as I reported it is not actually solved in version 0.55.\r\n\r\nEvery link which is returned by the `Datasette.absolute_url` method is still wrong, because it uses the request URL as the base. This still includes the suggested facet links and pagination links.\r\n\r\nWhat I wrote originally still stands:\r\n\r\n> Although many of the URLs in the pages are correct (presumably because they either use absolute paths which include `base_url` or relative paths), the faceting and pagination links still use fully-qualified URLs pointing at `http://localhost:8001`.\r\n> \r\n> I looked into this a little in the source code, and it seems to be an issue anywhere `request.url` or `request.path` is used, as these contain the values for the request between the frontend (Apache) and backend (Datasette) server. Those properties are primarily used via the `path_with_\u2026` family of utility functions and the `Datasette.absolute_url` method.\r\n\r\n Would you prefer to re-open this issue or have me create a new one?\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 637395097, "label": "Incorrect URLs when served behind a proxy with base_url set"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1254#issuecomment-795870524", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1254", "id": 795870524, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTg3MDUyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-10T18:27:45Z", "updated_at": "2021-03-10T18:27:45Z", "author_association": "OWNER", "body": "What other breaks did you spot?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 826613352, "label": "Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1256#issuecomment-795869144", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1256", "id": 795869144, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTg2OTE0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-10T18:26:46Z", "updated_at": "2021-03-10T18:26:46Z", "author_association": "OWNER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 827341657, "label": "Minor type in IP adress"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1256#issuecomment-795112935", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1256", "id": 795112935, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTExMjkzNQ==", "user": {"value": 6371750, "label": "JBPressac"}, "created_at": "2021-03-10T08:59:45Z", "updated_at": "2021-03-10T08:59:45Z", "author_association": "CONTRIBUTOR", "body": "Sorry, I meant \"minor typo\" not \"minor type\".", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 827341657, "label": "Minor type in IP adress"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1256#issuecomment-795085921", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1256", "id": 795085921, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NTA4NTkyMQ==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-03-10T08:35:17Z", "updated_at": "2021-03-10T08:35:17Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=h1) Report\n> Merging [#1256](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=desc) (4eef524) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **not change** coverage.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1256/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=tree)\n\n```diff\n@@ Coverage Diff @@\n## main #1256 +/- ##\n=======================================\n Coverage 91.56% 91.56% \n=======================================\n Files 34 34 \n Lines 4244 4244 \n=======================================\n Hits 3886 3886 \n Misses 358 358 \n```\n\n\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=footer). Last update [d0fd833...4eef524](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 827341657, "label": "Minor type in IP adress"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1254#issuecomment-794518438", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1254", "id": 794518438, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NDUxODQzOA==", "user": {"value": 3200608, "label": "durkie"}, "created_at": "2021-03-09T22:04:23Z", "updated_at": "2021-03-09T22:04:23Z", "author_association": "NONE", "body": "Dang, you're absolutely right. Spatialite 5.0 had been working fine for a plugin I was developing, but it apparently is broken in several other ways.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 826613352, "label": "Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1254#issuecomment-794441034", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1254", "id": 794441034, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NDQ0MTAzNA==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-03-09T20:54:18Z", "updated_at": "2021-03-09T21:12:15Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=h1) Report\n> Merging [#1254](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=desc) (b103204) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **decrease** coverage by `0.04%`.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1254/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=tree)\n\n```diff\n@@ Coverage Diff @@\n## main #1254 +/- ##\n==========================================\n- Coverage 91.56% 91.51% -0.05% \n==========================================\n Files 34 34 \n Lines 4244 4244 \n==========================================\n- Hits 3886 3884 -2 \n- Misses 358 360 +2 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=tree) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1254/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=footer). Last update [d0fd833...b103204](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 826613352, "label": "Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1254#issuecomment-794443710", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1254", "id": 794443710, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NDQ0MzcxMA==", "user": {"value": 3200608, "label": "durkie"}, "created_at": "2021-03-09T20:56:45Z", "updated_at": "2021-03-09T20:56:45Z", "author_association": "NONE", "body": "Oh wow I didn't even see that you had opened an issue about this so recently. I'll check on `/dbname` and report back.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 826613352, "label": "Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1254#issuecomment-794439632", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1254", "id": 794439632, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NDQzOTYzMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-09T20:53:02Z", "updated_at": "2021-03-09T20:53:02Z", "author_association": "OWNER", "body": "Thanks for catching that documentation update!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 826613352, "label": "Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1254#issuecomment-794437715", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1254", "id": 794437715, "node_id": "MDEyOklzc3VlQ29tbWVudDc5NDQzNzcxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-09T20:51:19Z", "updated_at": "2021-03-09T20:51:19Z", "author_association": "OWNER", "body": "Did you see my note on https://github.com/simonw/datasette/issues/1249#issuecomment-792384382 about a weird issue I was having with the `/dbname` page hanging the server? Have you seen anything like that in your work here?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 826613352, "label": "Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1252#issuecomment-793308483", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1252", "id": 793308483, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MzMwODQ4Mw==", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2021-03-09T03:06:10Z", "updated_at": "2021-03-09T03:06:10Z", "author_association": "NONE", "body": "# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=h1) Report\n> Merging [#1252](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=desc) (d22aa32) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **decrease** coverage by `0.04%`.\n> The diff coverage is `n/a`.\n\n[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1252/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=tree)\n\n```diff\n@@ Coverage Diff @@\n## main #1252 +/- ##\n==========================================\n- Coverage 91.56% 91.51% -0.05% \n==========================================\n Files 34 34 \n Lines 4244 4244 \n==========================================\n- Hits 3886 3884 -2 \n- Misses 358 360 +2 \n```\n\n\n| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=tree) | Coverage \u0394 | |\n|---|---|---|\n| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1252/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `\u0394 = absolute (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=footer). Last update [d0fd833...d22aa32](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 825217564, "label": "Add back styling to lists within table cells (fixes #1141)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1250#issuecomment-792386484", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1250", "id": 792386484, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjM4NjQ4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-08T00:29:06Z", "updated_at": "2021-03-08T00:29:06Z", "author_association": "OWNER", "body": "DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python\r\n\r\n```python\r\nimport duckdb\r\ncon = duckdb.connect(database=\"/tmp/blah.db\", read_only=True)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824067604, "label": "Research: Plugin hook for alternative database connections"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1248#issuecomment-792385274", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1248", "id": 792385274, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjM4NTI3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-08T00:25:10Z", "updated_at": "2021-03-08T00:25:10Z", "author_association": "OWNER", "body": "It's not possible yet, unfortunately. This came up on the forums recently: https://github.com/simonw/datasette/discussions/968\r\n\r\nI'm leaning further towards making the database connection layer itself work via a plugin hook, which would open up the possibility of supporting DuckDB and other databases as well. I've not committed to doing this yet though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 823035080, "label": "duckdb database (very low performance in SQLite)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-792384854", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 792384854, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjM4NDg1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-08T00:23:38Z", "updated_at": "2021-03-08T00:23:38Z", "author_association": "OWNER", "body": "One reason to prioritize this issue: Homebrew upgraded to SpatiaLite 5.0 recently https://formulae.brew.sh/formula/spatialite-tools and as a result SpatiaLite database created on my laptop don't appear to be compatible with Datasette when published using `datasette publish`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-792384382", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 792384382, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjM4NDM4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-08T00:22:02Z", "updated_at": "2021-03-08T00:22:02Z", "author_association": "OWNER", "body": "I tried this patch against `Dockerfile`:\r\n```diff\r\ndiff --git a/Dockerfile b/Dockerfile\r\nindex f4b1414..dd659e1 100644\r\n--- a/Dockerfile\r\n+++ b/Dockerfile\r\n@@ -1,25 +1,26 @@\r\n-FROM python:3.7.10-slim-stretch as build\r\n+FROM python:3.9.2-slim-buster as build\r\n \r\n # Setup build dependencies\r\n RUN apt update \\\r\n-&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \\\r\n- && apt clean\r\n+ && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \\\r\n+ libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \\\r\n+ && apt clean\r\n \r\n-\r\n-RUN wget \"https://www.sqlite.org/2020/sqlite-autoconf-3310100.tar.gz\" && tar xzf sqlite-autoconf-3310100.tar.gz \\\r\n- && cd sqlite-autoconf-3310100 && ./configure --disable-static --enable-fts5 --enable-json1 CFLAGS=\"-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1\" \\\r\n+RUN wget \"https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz\" && tar xzf sqlite-autoconf-3340100.tar.gz \\\r\n+ && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \\\r\n+ CFLAGS=\"-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1\" \\\r\n && make && make install\r\n \r\n-RUN wget \"http://www.gaia-gis.it/gaia-sins/freexl-sources/freexl-1.0.5.tar.gz\" && tar zxf freexl-1.0.5.tar.gz \\\r\n- && cd freexl-1.0.5 && ./configure && make && make install\r\n+RUN wget \"http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz\" && tar zxf freexl-1.0.6.tar.gz \\\r\n+ && cd freexl-1.0.6 && ./configure && make && make install\r\n \r\n-RUN wget \"http://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-4.4.0-RC0.tar.gz\" && tar zxf libspatialite-4.4.0-RC0.tar.gz \\\r\n- && cd libspatialite-4.4.0-RC0 && ./configure && make && make install\r\n+RUN wget \"http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz\" && tar zxf libspatialite-5.0.1.tar.gz \\\r\n+ && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install\r\n \r\n RUN wget \"http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz\" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install\r\n \r\n-RUN wget \"http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/spatialite-tools-4.4.0-RC0.tar.gz\" && tar zxf spatialite-tools-4.4.0-RC0.tar.gz \\\r\n- && cd spatialite-tools-4.4.0-RC0 && ./configure && make && make install\r\n+RUN wget \"http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz\" && tar zxf spatialite-tools-5.0.0.tar.gz \\\r\n+ && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install\r\n \r\n \r\n # Add local code to the image instead of fetching from pypi.\r\n@@ -27,7 +28,7 @@ COPY . /datasette\r\n \r\n RUN pip install /datasette\r\n \r\n-FROM python:3.7.10-slim-stretch\r\n+FROM python:3.9.2-slim-buster\r\n \r\n # Copy python dependencies and spatialite libraries\r\n COPY --from=build /usr/local/lib/ /usr/local/lib/\r\n```\r\n\r\nI had to use `--disable-rttopo` from the tip in https://github.com/OSGeo/gdal/pull/3443 and also needed to install `libminizip-dev`.\r\n\r\nThis works, sort of... I'm getting a weird issue where the `/dbname` page is hanging some of the time instead of loading correctly. Other than that it seems to work, but a hanging page is bad!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1249#issuecomment-792383956", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1249", "id": 792383956, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjM4Mzk1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-08T00:20:09Z", "updated_at": "2021-03-08T00:20:09Z", "author_association": "OWNER", "body": "Worth noting that the Docker image used by `datasette publish cloudrun` doesn't actually use that Datasette docker image - it does this:\r\n\r\nhttps://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L349-L353\r\n\r\nWhere the apt extras for SpatiaLite are: https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L344-L345\r\n\r\n`libsqlite3-mod-spatialite` against that official `python:3.8` image doesn't appear to install SpatiaLite 5.0.\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 824064069, "label": "Updated Dockerfile with SpatiaLite version 5.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/858#issuecomment-792308036", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/858", "id": 792308036, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjMwODAzNg==", "user": {"value": 1219001, "label": "robroc"}, "created_at": "2021-03-07T16:41:54Z", "updated_at": "2021-03-07T16:41:54Z", "author_association": "NONE", "body": "Apologies if I sound dense but I don't see where you would pass\n'shell=True'. I'm using the CLI installed via pip.\n\nOn Sun., Mar. 7, 2021, 2:15 a.m. David Smith, \nwrote:\n\n> To get it to work I had to:\n>\n> -\n>\n> add shell=true to the various commands in datasette\n> -\n>\n> use the name argument of the publish command. (\n> https://docs.datasette.io/en/stable/publish.html)\n>\n> \u2014\n> You are receiving this because you commented.\n> Reply to this email directly, view it on GitHub\n> ,\n> or unsubscribe\n> \n> .\n>\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642388564, "label": "publish heroku does not work on Windows 10"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1223#issuecomment-792233255", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1223", "id": 792233255, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjIzMzI1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-07T07:41:01Z", "updated_at": "2021-03-07T07:41:01Z", "author_association": "OWNER", "body": "This is fantastic, thanks so much for tracking this down.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 806918878, "label": "Add compile option to Dockerfile to fix failing test (fixes #696)"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/858#issuecomment-792230560", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/858", "id": 792230560, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjIzMDU2MA==", "user": {"value": 39445562, "label": "smithdc1"}, "created_at": "2021-03-07T07:14:58Z", "updated_at": "2021-03-07T07:14:58Z", "author_association": "NONE", "body": "To get it to work I had to:\n\n- add `shell=true` to the various commands in datasette \n\n- use the name argument of the publish command. (https://docs.datasette.io/en/stable/publish.html)\n\n\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642388564, "label": "publish heroku does not work on Windows 10"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/858#issuecomment-792129022", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/858", "id": 792129022, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MjEyOTAyMg==", "user": {"value": 1219001, "label": "robroc"}, "created_at": "2021-03-07T00:23:34Z", "updated_at": "2021-03-07T00:23:34Z", "author_association": "NONE", "body": "@smithdc1 Can you tell us what you did to get it to publish in Windows? What commands did you pass?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 642388564, "label": "publish heroku does not work on Windows 10"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/766#issuecomment-791509910", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/766", "id": 791509910, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MTUwOTkxMA==", "user": {"value": 6371750, "label": "JBPressac"}, "created_at": "2021-03-05T15:57:35Z", "updated_at": "2021-03-05T16:35:21Z", "author_association": "CONTRIBUTOR", "body": "Hello, \r\nI have the same wildcards search problems with an instance of Datasette. http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwerz&_sort=rowid is OK but http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwe* is not (FTS is activated on \"Reference\" \"IntituleAnalyse\" \"NomDuProducteur\" \"PresentationDuContenu\" \"Notes\"). \r\n\r\nNotice that a SQL query as below launched directly from SQLite in the server's shell, retrieves results.\r\n\r\n`select * from fonds_auguste_dupouy_1872_1967_fts where IntituleAnalyse MATCH \"gwe*\";`\r\n\r\nThanks,", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 617323873, "label": "Enable wildcard-searches by default"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791530093", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 791530093, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MTUzMDA5Mw==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-03-05T16:28:07Z", "updated_at": "2021-03-05T16:28:07Z", "author_association": "NONE", "body": "> I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.\r\n> \r\n> Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?\r\n\r\n@maxhawkins a limitation of the python mbox module is it loads the entire mbox into memory. I did find another approach to this problem that didn't use the builtin python mbox module and created a generator so that it didn't have to load the whole mbox into memory. I was hoping to use standard library modules, but this might be a good reason to investigate that approach a bit more. My worry is making sure a custom processor handles all the ins and outs of the mbox format correctly.\r\n\r\nHm. As I'm writing this, I thought of something. I think I can parse each message one at a time, and then use an mbox function to load each message using the python mbox module. That way the mbox module can still deal with the specifics of the mbox format, but I can use a generator.\r\n\r\nI'll give that a try. Thanks for the feedback @maxhawkins and @simonw. I'll give that a try.\r\n\r\n@simonw can we hold off on merging this until I can test this new approach?", "reactions": "{\"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791089881", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 791089881, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MTA4OTg4MQ==", "user": {"value": 28565, "label": "maxhawkins"}, "created_at": "2021-03-05T02:03:19Z", "updated_at": "2021-03-05T02:03:19Z", "author_association": "NONE", "body": "I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.\r\n\r\nIs it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-791053721", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32", "id": 791053721, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MTA1MzcyMQ==", "user": {"value": 6213, "label": "dsisnero"}, "created_at": "2021-03-05T00:31:27Z", "updated_at": "2021-03-05T00:31:27Z", "author_association": "NONE", "body": "I am getting the same thing for US West (N. California) us-west-1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 803333769, "label": "KeyError: 'Contents' on running upload"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790934616", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4", "id": 790934616, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDkzNDYxNg==", "user": {"value": 203343, "label": "Btibert3"}, "created_at": "2021-03-04T20:54:44Z", "updated_at": "2021-03-04T20:54:44Z", "author_association": "NONE", "body": "Sorry for the delay, I got sidetracked after class last night. I am getting the following error:\r\n\r\n```\r\n/content# google-takeout-to-sqlite mbox takeout.db Takeout/Mail/gmail.mbox \r\nUsage: google-takeout-to-sqlite [OPTIONS] COMMAND [ARGS]...Try 'google-takeout-to-sqlite --help' for help.\r\n\r\nError: No such command 'mbox'.\r\n```\r\n\r\nOn the box, I installed with pip after cloning: https://github.com/UtahDave/google-takeout-to-sqlite.git", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 778380836, "label": "Feature Request: Gmail"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1238#issuecomment-790857004", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1238", "id": 790857004, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDg1NzAwNA==", "user": {"value": 79913, "label": "tsibley"}, "created_at": "2021-03-04T19:06:55Z", "updated_at": "2021-03-04T19:06:55Z", "author_association": "NONE", "body": "@rgieseke Ah, that's super helpful. Thank you for the workaround for now!", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813899472, "label": "Custom pages don't work with base_url setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790695126", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790695126, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDY5NTEyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T15:20:42Z", "updated_at": "2021-03-04T15:20:42Z", "author_association": "MEMBER", "body": "I'm not sure why but my most recent import, when displayed in Datasette, looks like this:\r\n\r\n\"mbox__mbox_emails__753_446_rows\"\r\n\r\nSorting by `id` in the opposite order gives me the data I would expect - so it looks like a bunch of null/blank messages are being imported at some point and showing up first due to ID ordering.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790693674", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790693674, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDY5MzY3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T15:18:36Z", "updated_at": "2021-03-04T15:18:36Z", "author_association": "MEMBER", "body": "I imported my 10GB mbox with 750,000 emails in it, ran this tool (with a hacked fix for the blob column problem) - and now a search that returns 92 results takes 25.37ms! This is fantastic.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790669767", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790669767, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDY2OTc2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T14:46:06Z", "updated_at": "2021-03-04T14:46:06Z", "author_association": "MEMBER", "body": "Solution could be to pre-process that string by splitting on `(` and dropping everything afterwards, assuming that the `(...)` bit isn't necessary for correctly parsing the date.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790668263", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790668263, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDY2ODI2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T14:43:58Z", "updated_at": "2021-03-04T14:43:58Z", "author_association": "MEMBER", "body": "I added this code to output a message ID on errors:\r\n```diff\r\n print(\"Errors: {}\".format(num_errors))\r\n print(traceback.format_exc())\r\n+ print(\"Message-Id: {}\".format(email.get(\"Message-Id\", \"None\")))\r\n continue\r\n```\r\nHaving found a message ID that had an error, I ran this command to see the context:\r\n\r\n rg --text --context 20 '44F289B0.000001.02100@SCHWARZE-DWFXMI' ~/gmail.mbox\r\n\r\nThis was for the following error:\r\n```\r\n File \"/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py\", line 102, in get_mbox\r\n message[\"date\"] = get_message_date(email.get(\"Date\"), email.get_from())\r\n File \"/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py\", line 178, in get_message_date\r\n datetime_tuple = email.utils.parsedate_tz(mail_date)\r\n File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py\", line 50, in parsedate_tz\r\n res = _parsedate_tz(data)\r\n File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py\", line 69, in _parsedate_tz\r\n data = data.split()\r\nAttributeError: 'Header' object has no attribute 'split'\r\n```\r\nHere's what I spotted in the `ripgrep` output:\r\n```\r\n177133570:Message-Id: <44F289B0.000001.02100@SCHWARZE-DWFXMI>\r\n177133571-Date: Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop\ufffdische Sommerzeit)\r\n177133572-X-Mailer: IncrediMail (5002253)\r\n```\r\nSo it could it be that `_parsedate_tz` is having trouble with that `Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop\ufffdische Sommerzeit)` string.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790391711", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790391711, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM5MTcxMQ==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-03-04T07:36:24Z", "updated_at": "2021-03-04T07:36:24Z", "author_association": "NONE", "body": "> Looks like you're doing this:\r\n> \r\n> ```python\r\n> elif message.get_content_type() == \"text/plain\":\r\n> body = message.get_payload(decode=True)\r\n> ```\r\n> \r\n> So presumably that decodes to a unicode string?\r\n> \r\n> I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.\r\n\r\nAh, that's good to know. I think explicitly creating the tables will be a great improvement. I'll add that.\r\n\r\nAlso, I noticed after I opened this PR that the `message.get_payload()` is being deprecated in favor of `message.get_content()` or something like that. I'll see if that handles the decoding better, too.\r\n\r\nThanks for the feedback. I should have time tomorrow to put together some improvements.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790389335", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790389335, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM4OTMzNQ==", "user": {"value": 306240, "label": "UtahDave"}, "created_at": "2021-03-04T07:32:04Z", "updated_at": "2021-03-04T07:32:04Z", "author_association": "NONE", "body": "> The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:\r\n> \r\n> https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67\r\n> \r\n> I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.\r\n\r\nThe wait is from python loading the mbox file. This happens regardless if you're getting the length of the mbox. The mbox module is on the slow side. It is possible to do one's own parsing of the mbox, but I kind of wanted to avoid doing that.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/6#issuecomment-790384087", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6", "id": 790384087, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM4NDA4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:22:51Z", "updated_at": "2021-03-04T07:22:51Z", "author_association": "MEMBER", "body": "#3 also mentions the conflicting version with other tools.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 821841046, "label": "Upgrade to latest sqlite-utils"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790380839, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:17:05Z", "updated_at": "2021-03-04T07:17:05Z", "author_association": "MEMBER", "body": "Looks like you're doing this:\r\n```python\r\n elif message.get_content_type() == \"text/plain\":\r\n body = message.get_payload(decode=True)\r\n```\r\nSo presumably that decodes to a unicode string?\r\n\r\nI imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790379629", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790379629, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3OTYyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:14:41Z", "updated_at": "2021-03-04T07:14:41Z", "author_association": "MEMBER", "body": "Confirmed: removing the `len()` call does not speed things up, so it's reading through the entire file for some other purpose too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790378658", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790378658, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3ODY1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:12:48Z", "updated_at": "2021-03-04T07:12:48Z", "author_association": "MEMBER", "body": "It looks like the `body` is being loaded into a BLOB column - so in Datasette default it looks like this:\r\n\r\n\"mbox__mbox_emails__753_446_rows\"\r\n\r\nIf I `datasette install datasette-render-binary` and then try again I get this:\r\n\r\n\"mbox__mbox_emails__753_446_rows\"\r\n\r\nIt would be great if we could store the `body` as unicode text instead. May have to do something clever to decode it based on some kind of charset header?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790373024", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790373024, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3MzAyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:01:58Z", "updated_at": "2021-03-04T07:04:06Z", "author_association": "MEMBER", "body": "I got 9 warnings that look like this:\r\n```\r\nErrors: 1\r\nTraceback (most recent call last):\r\n File \"/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py\", line 103, in get_mbox\r\n message[\"date\"] = get_message_date(email.get(\"Date\"), email.get_from())\r\n File \"/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py\", line 167, in get_message_date\r\n datetime_tuple = email.utils.parsedate_tz(mail_date)\r\n File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py\", line 50, in parsedate_tz\r\n res = _parsedate_tz(data)\r\n File \"/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py\", line 69, in _parsedate_tz\r\n data = data.split()\r\nAttributeError: 'Header' object has no attribute 'split'\r\n```\r\nIt would be useful if those warnings told me the message ID (or similar) of the affected message so I could grep for it in the `mbox` and see what was going on.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790372621", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790372621, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3MjYyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T07:01:18Z", "updated_at": "2021-03-04T07:01:18Z", "author_association": "MEMBER", "body": "I'm not sure if it would work, but there is an alternative pattern for showing a progress bar against a really large file that I've used in `healthkit-to-sqlite` - you set the progress bar size to the size of the file in bytes, then update a counter as you read the file.\r\n\r\nhttps://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/cli.py#L24-L57 and https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/utils.py#L4-L19 (the `progress_callback()` bit) is where that happens.\r\n\r\nIt can be a bit of a convoluted pattern, and I'm not at all sure it would work for `mbox` files since it looks like that library has other reasons it needs to do a file scan rather than streaming it through one chunk of bytes at a time. So I imagine this would not work here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790370485, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T06:57:25Z", "updated_at": "2021-03-04T06:57:48Z", "author_association": "MEMBER", "body": "The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:\r\n\r\nhttps://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67\r\n\r\nI'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790369076", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790369076, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDM2OTA3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T06:54:46Z", "updated_at": "2021-03-04T06:54:46Z", "author_association": "MEMBER", "body": "The Rich-powered progress bar is pretty:\r\n\r\n![rich](https://user-images.githubusercontent.com/9599/109923307-71f69200-7c73-11eb-9ee2-8f0a240f3994.gif)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790312268", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5", "id": 790312268, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDMxMjI2OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T05:48:16Z", "updated_at": "2021-03-04T05:48:16Z", "author_association": "MEMBER", "body": "Wow, my mbox is a 10.35 GB download!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813880401, "label": "WIP: Add Gmail takeout mbox import"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/1243#issuecomment-790311215", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1243", "id": 790311215, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDMxMTIxNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-04T05:45:57Z", "updated_at": "2021-03-04T05:45:57Z", "author_association": "OWNER", "body": "Thanks!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 815955014, "label": "fix small typo"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-790257263", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 790257263, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDI1NzI2Mw==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2021-03-04T03:20:23Z", "updated_at": "2021-03-04T03:20:23Z", "author_association": "NONE", "body": "It's kind of an ugly hack, but you can try out what using the fts5 table as an actual datasette-accessible table looks like without changing any datasette code by creating yet another view on top of the fts5 table:\r\n\r\n`create view proxyview as select *, rank, table_fts as fts from table_fts;`\r\n\r\nThat's now visible from datasette, just like any other view, but you can use `fts match escape_fts(search_string) order by rank`.\r\n\r\nThis is only good as a proof of concept because you're inefficiently going from view -> fts5 external content table -> view -> data table. However, it does show it works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790198930", "issue_url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4", "id": 790198930, "node_id": "MDEyOklzc3VlQ29tbWVudDc5MDE5ODkzMA==", "user": {"value": 203343, "label": "Btibert3"}, "created_at": "2021-03-04T00:58:40Z", "updated_at": "2021-03-04T00:58:40Z", "author_association": "NONE", "body": "I am just seeing this sorry, yes! I will kick the tires later on tonight. My apologies for the delay.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 778380836, "label": "Feature Request: Gmail"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/283#issuecomment-789680230", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/283", "id": 789680230, "node_id": "MDEyOklzc3VlQ29tbWVudDc4OTY4MDIzMA==", "user": {"value": 605492, "label": "justinpinkney"}, "created_at": "2021-03-03T12:28:42Z", "updated_at": "2021-03-03T12:28:42Z", "author_association": "NONE", "body": "One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`. \r\n\r\nI diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 325958506, "label": "Support cross-database joins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/268#issuecomment-789409126", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/268", "id": 789409126, "node_id": "MDEyOklzc3VlQ29tbWVudDc4OTQwOTEyNg==", "user": {"value": 649467, "label": "mhalle"}, "created_at": "2021-03-03T03:57:15Z", "updated_at": "2021-03-03T03:58:40Z", "author_association": "NONE", "body": "In FTS5, I think doing an FTS search is actually much easier than doing a join against the main table like datasette does now. In fact, FTS5 external content tables provide a transparent interface back to the original table or view.\r\n\r\nHere's what I'm currently doing:\r\n* build a view that joins whatever tables I want and rename the columns to non-joiny names (e.g, `chapter.name AS chapter_name` in the view where needed)\r\n* Create an FTS5 table with `content=\"viewname\"`\r\n* As described in the \"external content tables\" section (https://www.sqlite.org/fts5.html#external_content_tables), sql queries can be made directly to the FTS table, which behind the covers makes select calls to the content table when the content of the original columns are needed.\r\n* In addition, you get \"rank\" and \"bm25()\" available to you when you select on the _fts table.\r\n\r\nUnfortunately, datasette doesn't currently seem happy being coerced into doing a real query on an fts5 table. This works:\r\n```select col1, col2, col3 from table_fts where coll1=\"value\" and table_fts match escape_fts(\"search term\") order by rank```\r\n\r\nBut this doesn't work in the datasette SQL query interface:\r\n```select col1, col2, col3 from table_fts where coll1=\"value\" and table_fts match escape_fts(:search) order by rank``` (the \"search\" input text field doesn't show up)\r\n\r\nFor what datasette is doing right now, I think you could just use contentless fts5 tables (`content=\"\"`), since all you care about is the rowid since all you're doing a subselect to get the rowid anyway. In fts5, that's just a contentless table.\r\n\r\nI guess if you want to follow this suggestion, you'd need a somewhat different code path for fts5.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 323718842, "label": "Mechanism for ranking results from SQLite full-text search"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1238#issuecomment-789186458", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1238", "id": 789186458, "node_id": "MDEyOklzc3VlQ29tbWVudDc4OTE4NjQ1OA==", "user": {"value": 198537, "label": "rgieseke"}, "created_at": "2021-03-02T20:19:30Z", "updated_at": "2021-03-02T20:19:30Z", "author_association": "CONTRIBUTOR", "body": "A custom `templates/index.html` seems to work and custom `pages` as a workaround with moving them to `pages/base_url_dir`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 813899472, "label": "Custom pages don't work with base_url setting"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1247#issuecomment-787616446", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1247", "id": 787616446, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzYxNjQ0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-01T03:50:37Z", "updated_at": "2021-03-01T03:50:37Z", "author_association": "OWNER", "body": "I like the `.add_memory_database()` option. I also like that it makes it more obvious that this is a capability of Datasette, since I'm excited to see more plugins, features and tests that take advantage of it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 818430405, "label": "datasette.add_memory_database() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1247#issuecomment-787616158", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1247", "id": 787616158, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzYxNjE1OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-01T03:49:27Z", "updated_at": "2021-03-01T03:49:27Z", "author_association": "OWNER", "body": "A couple of options:\r\n```python\r\ndatasette.add_memory_database(\"test_json_array\")\r\n# or make that first argument to add_database() optional and support:\r\ndatasette.add_database(memory_name=\"test_json_array\")\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 818430405, "label": "datasette.add_memory_database() method"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1246#issuecomment-787611153", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1246", "id": 787611153, "node_id": "MDEyOklzc3VlQ29tbWVudDc4NzYxMTE1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-03-01T03:30:57Z", "updated_at": "2021-03-01T03:30:57Z", "author_association": "OWNER", "body": "I'm going to try a new pattern for testing this, enabled by #1151 - the test will create a new named in-memory database, write some records to it and then run some test facets against that. This will save me from having to add yet another fixtures table for this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 817597268, "label": "Suggest for ArrayFacet possibly confused by blank values"}, "performed_via_github_app": null}