html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/79#issuecomment-1029317527,https://api.github.com/repos/simonw/sqlite-utils/issues/79,1029317527,IC_kwDOCGYnMM49WiOX,25778,2022-02-03T19:18:02Z,2022-02-03T19:18:02Z,CONTRIBUTOR,"Taking part of the conversation from #385 here. > Would sqlite-utils add-geometry-column ... be a good CLI enhancement. for example? Yes. And also `sqlite-utils create-spatial-index` would be great to have. My plan would be to add those once the Python API is settled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",557842245, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029306428,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029306428,IC_kwDOCGYnMM49Wfg8,25778,2022-02-03T19:03:43Z,2022-02-03T19:03:43Z,CONTRIBUTOR,"I thought about adding these as methods on `Database` and `Table`, and I'm back and forth on it for the same reasons you are. It's certainly cleaner, and it's clearer what you're operating on. I could go either way. I do sort of like having all the Spatialite stuff in its own module, just because it's built around an extension you might not have or want, but I don't know if that's a good reason to have a different API. You could have `init_spatialite` add methods to `Database` and `Table`, so they're only there if you have Spatialite set up. Is that too clever? It feels too clever. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029297971,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029297971,IC_kwDOCGYnMM49Wdcz,9599,2022-02-03T18:52:50Z,2022-02-03T18:52:50Z,OWNER,"I'm not sure I like `name=""geometry""` as the default argument to `add_geometry_column` - mainly because of this example here: ```python add_geometry_column(db[""locations""], ""POINT"") create_spatial_index(db[""locations""], ""geometry"") ``` I had to go and look at the code to figure out if `""POINT""` was the name of the column - and I don't like how inconsistent it looks next to the following `create_spatial_index()` call where you DO need to pass the column name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029296782,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029296782,IC_kwDOCGYnMM49WdKO,9599,2022-02-03T18:51:21Z,2022-02-03T18:51:21Z,OWNER,"What do you think about adding these as methods on the `Database` class instead? Then you could do: ```python # This is with an optional argument, which if omitted runs find_spatialite() for you: db.init_spatialite() # Instead of: init_spatialite(db, find_spatialite()) ``` Likewise, the `add_geometry_column` and `create_spatial_index` methods could live on `Table`: ```python # Instead of this: add_geometry_column(db[""locations""], ""POINT"", ""geometry"") create_spatial_index(db[""locations""], ""geometry"") # Could have this: db[""locations""].add_geometry_column(""POINT"") db[""locations""].create_spatial_index(""geometry"") ``` On the one hand, this is much more consistent with the existing `sqlite-utils` Python API. But on the other hand... this is mixing SpatiaLite functionality directly into the core classes. Is that a good idea, seeing as SpatiaLite is both an optional extension (which can be tricky to install) AND something that has a very different release cadence and quality-of-documentation from SQLite itself? There's a third option: the SpatiaLite could exist on subclasses of `Database` and `Table` - so the above examples would look something like this: ```python from sqlite_utils.gis import SpatiaLiteDatabase db = SpatiaLiteDatabase(""geo.db"") db.init_spatialite() db[""locations""].add_geometry_column(""POINT"") db[""locations""].create_spatial_index(""geometry"") ``` On the one hand, this would keep the SpatiaLite-specific stuff out of the core Database/Table classes. But it feels a bit untidy to me, especially since it raises the spectre of someone who was already subclassing Database for some reason now needing to instead subclass `SpatiaLiteDatabase` (not too keen on that capitalization) - or even (horror) trying to dabble with multiple inheritance, which can only lead to pain. So I don't have a strong opinion formed on this question yet!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029285985,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029285985,IC_kwDOCGYnMM49Wahh,9599,2022-02-03T18:37:48Z,2022-02-03T18:37:48Z,OWNER,"`from sqlite_utils.utils import find_spatialite` is part of the documented API already: https://sqlite-utils.datasette.io/en/3.22.1/python-api.html#finding-spatialite To avoid needing to bump the major version number to 4 to indicate a backwards incompatible change, we should keep a `from .gis import find_spatialite` line at the top of `utils.py` such that any existing code with that documented import continues to work.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029273853,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029273853,IC_kwDOCGYnMM49WXj9,9599,2022-02-03T18:23:30Z,2022-02-03T18:31:21Z,OWNER,"OK, this change makes a bunch of sense to me - and also raises some interesting questions about future additions to `sqlite-utils` with regards to SpatiaLite. Would `sqlite-utils add-geometry-column ...` be a good CLI enhancement. for example? I see you've already talked about that in #79 - moving this conversation there!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029180984,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029180984,IC_kwDOCGYnMM49WA44,25778,2022-02-03T16:42:04Z,2022-02-03T16:42:04Z,CONTRIBUTOR,Fixed my spelling. That's a useful thing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029177015,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029177015,IC_kwDOCGYnMM49V_63,9599,2022-02-03T16:38:02Z,2022-02-03T16:38:02Z,OWNER,Sorry had missed this - tests should run now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312, https://github.com/simonw/sqlite-utils/pull/385#issuecomment-1029175907,https://api.github.com/repos/simonw/sqlite-utils/issues/385,1029175907,IC_kwDOCGYnMM49V_pj,25778,2022-02-03T16:36:54Z,2022-02-03T16:36:54Z,CONTRIBUTOR,"@simonw Not sure if you've seen this, but any chance you can run the tests?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1102899312, https://github.com/simonw/datasette/issues/1545#issuecomment-1028535868,https://api.github.com/repos/simonw/datasette/issues/1545,1028535868,IC_kwDOBm6k_c49TjY8,9599,2022-02-03T02:08:30Z,2022-02-03T02:08:30Z,OWNER,"Filed an issue with Jinja suggesting a documentation update: - https://github.com/pallets/jinja/issues/1578","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249, https://github.com/simonw/datasette/pull/1617#issuecomment-1028419517,https://api.github.com/repos/simonw/datasette/issues/1617,1028419517,IC_kwDOBm6k_c49TG-9,22429695,2022-02-02T22:30:26Z,2022-02-03T01:36:07Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1617](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af293c9) into [main](https://codecov.io/gh/simonw/datasette/commit/2aa686c6554bf6b8230eb5b3019574df6cc99225?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2aa686c) will **increase** coverage by `0.06%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1617/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1617 +/- ## ========================================== + Coverage 92.09% 92.16% +0.06% ========================================== Files 34 34 Lines 4518 4531 +13 ========================================== + Hits 4161 4176 +15 + Misses 357 355 -2 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.37% <100.00%> (ø)` | | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.19% <0.00%> (ø)` | | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.79% <0.00%> (+<0.01%)` | :arrow_up: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.49% <0.00%> (+0.07%)` | :arrow_up: | | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1617/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `95.09% <0.00%> (+2.38%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2aa686c...af293c9](https://codecov.io/gh/simonw/datasette/pull/1617?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806, https://github.com/simonw/datasette/pull/1617#issuecomment-1028519382,https://api.github.com/repos/simonw/datasette/issues/1617,1028519382,IC_kwDOBm6k_c49TfXW,9599,2022-02-03T01:31:25Z,2022-02-03T01:31:25Z,OWNER,"I was hoping to get the test suite running on Windows before merging this PR but that looks like it will be a BIG job, see: - #1627 So I'm going to merge this one as-is for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806, https://github.com/simonw/datasette/issues/1545#issuecomment-1028517268,https://api.github.com/repos/simonw/datasette/issues/1545,1028517268,IC_kwDOBm6k_c49Te2U,9599,2022-02-03T01:26:53Z,2022-02-03T01:26:53Z,OWNER,"I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922 > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1075893249, https://github.com/simonw/datasette/pull/1617#issuecomment-1028517073,https://api.github.com/repos/simonw/datasette/issues/1617,1028517073,IC_kwDOBm6k_c49TezR,9599,2022-02-03T01:26:32Z,2022-02-03T01:26:32Z,OWNER,"Aha I understand the problem now! https://github.com/pallets/jinja/issues/1378#issuecomment-812410922 > Jinja template names/paths are not always filesystem paths. So regardless of the OS Jinja always uses forward slashes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1120990806, https://github.com/simonw/datasette/pull/1626#issuecomment-1028515161,https://api.github.com/repos/simonw/datasette/issues/1626,1028515161,IC_kwDOBm6k_c49TeVZ,9599,2022-02-03T01:22:43Z,2022-02-03T01:22:43Z,OWNER,"OK, the tests do NOT pass against Windows! https://github.com/simonw/datasette/runs/5044105941 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096, https://github.com/simonw/datasette/issues/1534#issuecomment-1028461220,https://api.github.com/repos/simonw/datasette/issues/1534,1028461220,IC_kwDOBm6k_c49TRKk,9599,2022-02-02T23:39:33Z,2022-02-02T23:39:33Z,OWNER,"I've decided not to do this, because of the risk that Cloudflare could cache the JSON version for an HTML page or vice-versa.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065432388, https://github.com/simonw/datasette/pull/1626#issuecomment-1028423514,https://api.github.com/repos/simonw/datasette/issues/1626,1028423514,IC_kwDOBm6k_c49TH9a,22429695,2022-02-02T22:36:37Z,2022-02-02T22:39:52Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1626](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4b4d0e1) into [main](https://codecov.io/gh/simonw/datasette/commit/b5e6b1a9e1332fca3effe45d55dd06ee4249f163?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b5e6b1a) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1626/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1626 +/- ## ======================================= Coverage 92.16% 92.16% ======================================= Files 34 34 Lines 4531 4531 ======================================= Hits 4176 4176 Misses 355 355 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b5e6b1a...4b4d0e1](https://codecov.io/gh/simonw/datasette/pull/1626?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096, https://github.com/simonw/datasette/pull/1626#issuecomment-1028420821,https://api.github.com/repos/simonw/datasette/issues/1626,1028420821,IC_kwDOBm6k_c49THTV,9599,2022-02-02T22:32:26Z,2022-02-02T22:33:31Z,OWNER,"That broke on a macOS test: https://github.com/simonw/datasette/runs/5044036993?check_suite_focus=true I'm going to remove macOS and Ubuntu and just try Windows purely to see what happens there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122451096, https://github.com/simonw/datasette/pull/1616#issuecomment-1028414871,https://api.github.com/repos/simonw/datasette/issues/1616,1028414871,IC_kwDOBm6k_c49TF2X,9599,2022-02-02T22:23:45Z,2022-02-02T22:23:45Z,OWNER,First stable Black release!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1119413338, https://github.com/simonw/datasette/issues/1623#issuecomment-1028397935,https://api.github.com/repos/simonw/datasette/issues/1623,1028397935,IC_kwDOBm6k_c49TBtv,9599,2022-02-02T21:59:43Z,2022-02-02T21:59:43Z,OWNER,Here's the new test: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/tests/test_html.py#L927-L936,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919, https://github.com/simonw/datasette/issues/1624#issuecomment-1028396866,https://api.github.com/repos/simonw/datasette/issues/1624,1028396866,IC_kwDOBm6k_c49TBdC,9599,2022-02-02T21:58:06Z,2022-02-02T21:58:06Z,OWNER,"It looks like this is because `IndexView` extends `BaseView` rather than extending `DataView` which is where all that CORS stuff happens: https://github.com/simonw/datasette/blob/23a09b0f6af33c52acf8c1d9002fe475b42fee10/datasette/views/index.py#L18-L21 Another thing I should address with the refactor project in: - #878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122427321, https://github.com/simonw/datasette/issues/1620#issuecomment-1028393259,https://api.github.com/repos/simonw/datasette/issues/1620,1028393259,IC_kwDOBm6k_c49TAkr,9599,2022-02-02T21:53:02Z,2022-02-02T21:53:02Z,OWNER,"I ran the following on https://www.google.com/ in the console to demonstrate that these work as intended: ```javascript [ ""https://latest.datasette.io/fixtures"", ""https://latest.datasette.io/fixtures?sql=select+1"", ""https://latest.datasette.io/fixtures/facetable"" ].forEach(async (url) => { response = await fetch(url, {method: ""HEAD""}); console.log(response.headers.get(""Link"")); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041, https://github.com/simonw/datasette/issues/1623#issuecomment-1028389953,https://api.github.com/repos/simonw/datasette/issues/1623,1028389953,IC_kwDOBm6k_c49S_xB,9599,2022-02-02T21:48:34Z,2022-02-02T21:48:34Z,OWNER,"A few other pages do that too, including: - https://latest.datasette.io/-/messages - https://latest.datasette.io/-/allow-debug","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122416919, https://github.com/simonw/datasette/pull/1622#issuecomment-1028387529,https://api.github.com/repos/simonw/datasette/issues/1622,1028387529,IC_kwDOBm6k_c49S_LJ,22429695,2022-02-02T21:45:21Z,2022-02-02T21:45:21Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1622](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fbaf317) into [main](https://codecov.io/gh/simonw/datasette/commit/8d5779acf0041cfd0db7f68f468419f9008b86ec?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8d5779a) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1622/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1622 +/- ## ======================================= Coverage 92.11% 92.11% ======================================= Files 34 34 Lines 4525 4525 ======================================= Hits 4168 4168 Misses 357 357 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [8d5779a...fbaf317](https://codecov.io/gh/simonw/datasette/pull/1622?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122414274, https://github.com/simonw/datasette/issues/1620#issuecomment-1028385067,https://api.github.com/repos/simonw/datasette/issues/1620,1028385067,IC_kwDOBm6k_c49S-kr,9599,2022-02-02T21:42:23Z,2022-02-02T21:42:23Z,OWNER,"``` % curl -s -I 'https://latest.datasette.io/' | grep link link: https://latest.datasette.io/.json; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/fixtures' | grep link link: https://latest.datasette.io/fixtures.json; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/fixtures?sql=select+1' | grep link link: https://latest.datasette.io/fixtures.json?sql=select+1; rel=""alternate""; type=""application/json+datasette"" % curl -s -I 'https://latest.datasette.io/-/plugins' | grep link link: https://latest.datasette.io/-/plugins.json; rel=""alternate""; type=""application/json+datasette"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041, https://github.com/simonw/datasette/issues/1620#issuecomment-1028374330,https://api.github.com/repos/simonw/datasette/issues/1620,1028374330,IC_kwDOBm6k_c49S786,9599,2022-02-02T21:28:16Z,2022-02-02T21:28:16Z,OWNER,I just realized I can refactor this to make it much simpler.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121618041, https://github.com/simonw/datasette/issues/1618#issuecomment-1028294089,https://api.github.com/repos/simonw/datasette/issues/1618,1028294089,IC_kwDOBm6k_c49SoXJ,770231,2022-02-02T19:42:03Z,2022-02-02T19:42:03Z,NONE,"Thanks for looking into this. It might have been nice if `explain` surfaced these function calls. Looks like `explain query plan` does, but only for basic queries. ``` sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list(), pragma_database_list(), pragma_module_list()' -t id parent notused detail ---- -------- --------- ------------------------------------------------ 4 0 0 SCAN pragma_function_list VIRTUAL TABLE INDEX 0: 8 0 0 SCAN pragma_database_list VIRTUAL TABLE INDEX 0: 12 0 0 SCAN pragma_module_list VIRTUAL TABLE INDEX 0: ``` ``` sqlite-utils fixtures.db 'explain query plan select * from pragma_function_list() as fl, pragma_database_list() as dl, pragma_module_list() as ml' -t id parent notused detail ---- -------- --------- ------------------------------ 4 0 0 SCAN fl VIRTUAL TABLE INDEX 0: 8 0 0 SCAN dl VIRTUAL TABLE INDEX 0: 12 0 0 SCAN ml VIRTUAL TABLE INDEX 0: ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305, https://github.com/simonw/datasette/issues/1533#issuecomment-1027672617,https://api.github.com/repos/simonw/datasette/issues/1533,1027672617,IC_kwDOBm6k_c49QQop,9599,2022-02-02T07:56:51Z,2022-02-02T07:56:51Z,OWNER,"Demos - these pages both have ` Table-valued functions exist only for PRAGMAs that return results and that have no side-effects. So it's possible I'm being overly paranoid here after all: what I want to block here is people running things like `PRAGMA case_sensitive_like = 1` which could affect the global state for that connection and cause unexpected behaviour later on. So maybe I should allow all pragma functions. I previously allowed an allow-list of them in: - #761","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305, https://github.com/simonw/datasette/issues/1618#issuecomment-1027653005,https://api.github.com/repos/simonw/datasette/issues/1618,1027653005,IC_kwDOBm6k_c49QL2N,9599,2022-02-02T07:22:13Z,2022-02-02T07:22:13Z,OWNER,"There's a workaround for this at the moment, which is to use parameterized SQL queries. For example, this: https://fivethirtyeight.datasettes.com/polls?sql=select+*+from+books+where+title+%3D+%3Atitle&title=The+Pragmatic+Programmer So the SQL query is `select * from books where title = :title` and then `&title=...` is added to the URL. The reason behind the quite aggressive pragma filtering is that SQLite allows you to execute pragmas using function calls, like this one: ```sql SELECT * FROM pragma_index_info('idx52'); ``` These can be nested arbitrarily deeply in sub-queries, so it's difficult to write a regular expression that will definitely catch them. I'm open to relaxing the regex a bit, but I need to be very confident that it's safe to do so. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121121305, https://github.com/simonw/datasette/issues/1586#issuecomment-1027648180,https://api.github.com/repos/simonw/datasette/issues/1586,1027648180,IC_kwDOBm6k_c49QKq0,9599,2022-02-02T07:13:31Z,2022-02-02T07:13:31Z,OWNER,"Running it as part of `datasette publish` is a smart idea - I'm slightly nervous about modifying the database file that has been published though, since part of the undocumented contract right now is that the bytes served are the exact same bytes as the ones you ran the publish against. But there's no reason for that expectation to exist, and I doubt anyone is relying on that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1096536240, https://github.com/simonw/datasette/issues/1619#issuecomment-1027647257,https://api.github.com/repos/simonw/datasette/issues/1619,1027647257,IC_kwDOBm6k_c49QKcZ,9599,2022-02-02T07:11:43Z,2022-02-02T07:11:43Z,OWNER,Weirdly the bug does NOT exhibit itself on this demo: https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1 - which correctly links to https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/no_primary_key/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414, https://github.com/simonw/datasette/issues/1619#issuecomment-1027646659,https://api.github.com/repos/simonw/datasette/issues/1619,1027646659,IC_kwDOBm6k_c49QKTD,9599,2022-02-02T07:10:37Z,2022-02-02T07:10:37Z,OWNER,It's not just the table with slashes in the name. Same thing on http://127.0.0.1:3344/foo/bar/fixtures/attraction_characteristic/1 - the `json` link goes to a JSON-rendered 404 on http://127.0.0.1:3344/foo/bar/foo/bar/fixtures/attraction_characteristic/1.json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414, https://github.com/simonw/datasette/issues/1576#issuecomment-1027635925,https://api.github.com/repos/simonw/datasette/issues/1576,1027635925,IC_kwDOBm6k_c49QHrV,9599,2022-02-02T06:47:20Z,2022-02-02T06:47:20Z,OWNER,"Here's what I was hacking around with when I uncovered this problem: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 77fb285..8c57d08 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import urllib import itertools import json @@ -615,44 +616,37 @@ class TableView(RowTableShared): if request.args.get(""_timelimit""): extra_args[""custom_time_limit""] = int(request.args.get(""_timelimit"")) - # Execute the main query! - results = await db.execute(sql, params, truncate=True, **extra_args) - - # Calculate the total count for this query - filtered_table_rows_count = None - if ( - not db.is_mutable - and self.ds.inspect_data - and count_sql == f""select count(*) from {table} "" - ): - # We can use a previously cached table row count - try: - filtered_table_rows_count = self.ds.inspect_data[database][""tables""][ - table - ][""count""] - except KeyError: - pass - - # Otherwise run a select count(*) ... - if count_sql and filtered_table_rows_count is None and not nocount: - try: - count_rows = list(await db.execute(count_sql, from_sql_params)) - filtered_table_rows_count = count_rows[0][0] - except QueryInterrupted: - pass - - # Faceting - if not self.ds.setting(""allow_facet"") and any( - arg.startswith(""_facet"") for arg in request.args - ): - raise BadRequest(""_facet= is not allowed"") + async def execute_count(): + # Calculate the total count for this query + filtered_table_rows_count = None + if ( + not db.is_mutable + and self.ds.inspect_data + and count_sql == f""select count(*) from {table} "" + ): + # We can use a previously cached table row count + try: + filtered_table_rows_count = self.ds.inspect_data[database][ + ""tables"" + ][table][""count""] + except KeyError: + pass + + if count_sql and filtered_table_rows_count is None and not nocount: + try: + count_rows = list(await db.execute(count_sql, from_sql_params)) + filtered_table_rows_count = count_rows[0][0] + except QueryInterrupted: + pass + + return filtered_table_rows_count + + filtered_table_rows_count = await execute_count() # pylint: disable=no-member facet_classes = list( itertools.chain.from_iterable(pm.hook.register_facet_classes()) ) - facet_results = {} - facets_timed_out = [] facet_instances = [] for klass in facet_classes: facet_instances.append( @@ -668,33 +662,58 @@ class TableView(RowTableShared): ) ) - if not nofacet: - for facet in facet_instances: - ( - instance_facet_results, - instance_facets_timed_out, - ) = await facet.facet_results() - for facet_info in instance_facet_results: - base_key = facet_info[""name""] - key = base_key - i = 1 - while key in facet_results: - i += 1 - key = f""{base_key}_{i}"" - facet_results[key] = facet_info - facets_timed_out.extend(instance_facets_timed_out) - - # Calculate suggested facets - suggested_facets = [] - if ( - self.ds.setting(""suggest_facets"") - and self.ds.setting(""allow_facet"") - and not _next - and not nofacet - and not nosuggest - ): - for facet in facet_instances: - suggested_facets.extend(await facet.suggest()) + async def execute_suggested_facets(): + # Calculate suggested facets + suggested_facets = [] + if ( + self.ds.setting(""suggest_facets"") + and self.ds.setting(""allow_facet"") + and not _next + and not nofacet + and not nosuggest + ): + for facet in facet_instances: + suggested_facets.extend(await facet.suggest()) + return suggested_facets + + async def execute_facets(): + facet_results = {} + facets_timed_out = [] + if not self.ds.setting(""allow_facet"") and any( + arg.startswith(""_facet"") for arg in request.args + ): + raise BadRequest(""_facet= is not allowed"") + + if not nofacet: + for facet in facet_instances: + ( + instance_facet_results, + instance_facets_timed_out, + ) = await facet.facet_results() + for facet_info in instance_facet_results: + base_key = facet_info[""name""] + key = base_key + i = 1 + while key in facet_results: + i += 1 + key = f""{base_key}_{i}"" + facet_results[key] = facet_info + facets_timed_out.extend(instance_facets_timed_out) + + return facet_results, facets_timed_out + + # Execute the main query, facets and facet suggestions in parallel: + ( + results, + suggested_facets, + (facet_results, facets_timed_out), + ) = await asyncio.gather( + db.execute(sql, params, truncate=True, **extra_args), + execute_suggested_facets(), + execute_facets(), + ) + + results = await db.execute(sql, params, truncate=True, **extra_args) # Figure out columns and rows for the query columns = [r[0] for r in results.description] ``` It's a hacky attempt at running some of the table page queries in parallel to see what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1087181951, https://github.com/simonw/datasette/issues/1611#issuecomment-1027635175,https://api.github.com/repos/simonw/datasette/issues/1611,1027635175,IC_kwDOBm6k_c49QHfn,9599,2022-02-02T06:45:47Z,2022-02-02T06:45:47Z,OWNER,"Prototype, not sure that this actually works yet: ```diff diff --git a/datasette/database.py b/datasette/database.py index 6ce8721..0c4aec7 100644 --- a/datasette/database.py +++ b/datasette/database.py @@ -256,18 +256,26 @@ class Database: # Try to get counts for each table, $limit timeout for each count counts = {} for table in await self.table_names(): - try: - table_count = ( - await self.execute( - f""select count(*) from [{table}]"", - custom_time_limit=limit, - ) - ).rows[0][0] - counts[table] = table_count - # In some cases I saw ""SQL Logic Error"" here in addition to - # QueryInterrupted - so we catch that too: - except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError): - counts[table] = None + print(table.lower()) + if table.lower() == ""knn"": + counts[table] = 0 + else: + try: + table_count = ( + await self.execute( + f""select count(*) from [{table}]"", + custom_time_limit=limit, + ) + ).rows[0][0] + counts[table] = table_count + # In some cases I saw ""SQL Logic Error"" here in addition to + # QueryInterrupted - so we catch that too: + except ( + QueryInterrupted, + sqlite3.OperationalError, + sqlite3.DatabaseError, + ): + counts[table] = None if not self.is_mutable: self._cached_table_counts = counts return counts ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1113384383, https://github.com/simonw/datasette/issues/1607#issuecomment-1027634490,https://api.github.com/repos/simonw/datasette/issues/1607,1027634490,IC_kwDOBm6k_c49QHU6,9599,2022-02-02T06:44:30Z,2022-02-02T06:44:30Z,OWNER,"Prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 09d7d03..e2a5aea 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -724,6 +724,47 @@ class Datasette: sqlite_extensions[extension] = None except Exception: pass + # More details on SpatiaLite + if ""spatialite"" in sqlite_extensions: + spatialite_details = {} + fns = ( + ""spatialite_version"", + ""spatialite_target_cpu"", + ""rcheck_strict_sql_quoting"", + ""freexl_version"", + ""proj_version"", + ""geos_version"", + ""rttopo_version"", + ""libxml2_version"", + ""HasIconv"", + ""HasMathSQL"", + ""HasGeoCallbacks"", + ""HasProj"", + ""HasProj6"", + ""HasGeos"", + ""HasGeosAdvanced"", + ""HasGeosTrunk"", + ""HasGeosReentrant"", + ""HasGeosOnlyReentrant"", + ""HasMiniZip"", + ""HasRtTopo"", + ""HasLibXML2"", + ""HasEpsg"", + ""HasFreeXL"", + ""HasGeoPackage"", + ""HasGCP"", + ""HasTopology"", + ""HasKNN"", + ""HasRouting"", + ) + for fn in fns: + try: + result = conn.execute(""select {}()"".format(fn)) + spatialite_details[fn] = result.fetchone()[0] + except Exception: + pass + sqlite_extensions[""spatialite""] = spatialite_details + # Figure out supported FTS versions ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109783030, https://github.com/simonw/datasette/issues/1533#issuecomment-1027633686,https://api.github.com/repos/simonw/datasette/issues/1533,1027633686,IC_kwDOBm6k_c49QHIW,9599,2022-02-02T06:42:53Z,2022-02-02T06:42:53Z,OWNER,"I'm going to apply the hack, then fix it again in: - #1518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1065431383, https://github.com/simonw/datasette/pull/1616#issuecomment-1025732071,https://api.github.com/repos/simonw/datasette/issues/1616,1025732071,IC_kwDOBm6k_c49I23n,22429695,2022-01-31T13:20:18Z,2022-01-31T13:20:18Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1616](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4ebe94b) into [main](https://codecov.io/gh/simonw/datasette/commit/2aa686c6554bf6b8230eb5b3019574df6cc99225?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2aa686c) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1616/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1616 +/- ## ======================================= Coverage 92.09% 92.09% ======================================= Files 34 34 Lines 4518 4518 ======================================= Hits 4161 4161 Misses 357 357 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2aa686c...4ebe94b](https://codecov.io/gh/simonw/datasette/pull/1616?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1119413338, https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1024727476,https://api.github.com/repos/simonw/sqlite-utils/issues/352,1024727476,IC_kwDOCGYnMM49FBm0,9599,2022-01-28T23:01:06Z,2022-01-28T23:01:06Z,OWNER,"Manual test run with that prototype: ``` % echo '{""foo"": ""bar"", ""id"": 1}' | sqlite-utils insert insert-extract.db rows - --pk id --extract foo % sqlite-utils dump insert-extract.db BEGIN TRANSACTION; CREATE TABLE [foo] ( [id] INTEGER PRIMARY KEY, [value] TEXT ); INSERT INTO ""foo"" VALUES(1,'bar'); CREATE TABLE [rows] ( [foo] INTEGER REFERENCES [foo]([id]), [id] INTEGER PRIMARY KEY ); INSERT INTO ""rows"" VALUES(1,1); CREATE UNIQUE INDEX [idx_foo_value] ON [foo] ([value]); COMMIT; ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507, https://github.com/simonw/sqlite-utils/issues/352#issuecomment-1024726879,https://api.github.com/repos/simonw/sqlite-utils/issues/352,1024726879,IC_kwDOCGYnMM49FBdf,9599,2022-01-28T22:59:44Z,2022-01-28T22:59:44Z,OWNER,"Rough prototype, seems to work:; ```diff diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index 771d432..431b93e 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -867,6 +867,12 @@ def insert_upsert_options(*, require_pk=False): ), load_extension_option, click.option(""--silent"", is_flag=True, help=""Do not show progress bar""), + click.option( + ""--extract"", + ""extracts"", + multiple=True, + help=""Columns to extract to another table"", + ), ) ): fn = decorator(fn) @@ -906,6 +912,7 @@ def insert_upsert_implementation( load_extension=None, silent=False, bulk_sql=None, + extracts=None, ): db = sqlite_utils.Database(path) _load_extensions(db, load_extension) @@ -1008,6 +1015,8 @@ def insert_upsert_implementation( extra_kwargs[""defaults""] = dict(default) if upsert: extra_kwargs[""upsert""] = upsert + if extracts is not None: + extra_kwargs[""extracts""] = extracts # Apply {""$base64"": true, ...} decoding, if needed docs = (decode_base64_values(doc) for doc in docs) @@ -1117,6 +1126,7 @@ def insert( truncate, not_null, default, + extracts, ): """""" Insert records from FILE into a table, creating the table if it @@ -1174,6 +1184,7 @@ def insert( silent=silent, not_null=not_null, default=default, + extracts=extracts, ) except UnicodeDecodeError as ex: raise click.ClickException(UNICODE_ERROR.format(ex)) @@ -1207,6 +1218,7 @@ def upsert( analyze, load_extension, silent, + extracts, ): """""" Upsert records based on their primary key. Works like 'insert' but if ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1072792507, https://github.com/simonw/datasette/issues/1615#issuecomment-1023997327,https://api.github.com/repos/simonw/datasette/issues/1615,1023997327,IC_kwDOBm6k_c49CPWP,369053,2022-01-28T08:37:36Z,2022-01-28T08:37:36Z,NONE,"Oops, it feels like this should perhaps be migrated to GitHub Discussions - sorry! I don't think I have the ability to do that 😅","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1117132741, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1022466476,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1022466476,IC_kwDOCGYnMM488Zms,9599,2022-01-26T18:17:43Z,2022-01-26T18:17:43Z,OWNER,"Manually tested it like this: ``` # Create database with an empty ""lines"" table sqlite-utils create-table bulk-test.db lines line text # Stream records every 0.5s, commit every 5 records stream-delay docs/python-api.rst -d 500 | \ sqlite-utils bulk bulk-test.db 'insert into lines (line) values (:line)' - \ --lines --batch-size 5 ``` Running `datasette bulk-test.db` showed that records would show up about every 2.5s five at a time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101, https://github.com/simonw/datasette/issues/1613#issuecomment-1022381732,https://api.github.com/repos/simonw/datasette/issues/1613,1022381732,IC_kwDOBm6k_c488E6k,9599,2022-01-26T16:41:45Z,2022-01-26T16:41:45Z,OWNER,A better interface for modifying the columns used in the SELECT clause would be useful too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238, https://github.com/simonw/datasette/issues/1613#issuecomment-1022255862,https://api.github.com/repos/simonw/datasette/issues/1613,1022255862,IC_kwDOBm6k_c487mL2,9599,2022-01-26T14:35:31Z,2022-01-26T14:37:44Z,OWNER,"Joins are really hard. A mechanism for constructing them in the table view would help a lot: - https://github.com/simonw/datasette/issues/613","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238, https://github.com/simonw/datasette/issues/1613#issuecomment-1022254258,https://api.github.com/repos/simonw/datasette/issues/1613,1022254258,IC_kwDOBm6k_c487lyy,9599,2022-01-26T14:33:46Z,2022-01-26T14:37:31Z,OWNER,"Tool for setting up foreign key relationships. It could even verify the relationship before you apply it - checking that every value in the column does indeed correspond to a value in the other table. Could also detect and suggest possible ones.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238, https://github.com/simonw/datasette/issues/1613#issuecomment-1022257496,https://api.github.com/repos/simonw/datasette/issues/1613,1022257496,IC_kwDOBm6k_c487mlY,9599,2022-01-26T14:37:14Z,2022-01-26T14:37:14Z,OWNER,"Better contextual help on the SQL editor - like in Django SQL Dashboard which shows all available tables and columns. Fancy inline autocomplete would be great too, but that's pretty hard for SQL based on past research.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877769,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021877769,IC_kwDOCGYnMM486J4J,9599,2022-01-26T05:19:48Z,2022-01-26T05:19:48Z,OWNER,Can use this utility function: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/utils.py#L322-L325,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021877058,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021877058,IC_kwDOCGYnMM486JtC,9599,2022-01-26T05:18:12Z,2022-01-26T05:18:18Z,OWNER,"Help for `insert` says: ``` --batch-size INTEGER Commit every X records ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101, https://github.com/simonw/sqlite-utils/issues/392#issuecomment-1021876914,https://api.github.com/repos/simonw/sqlite-utils/issues/392,1021876914,IC_kwDOCGYnMM486Jqy,9599,2022-01-26T05:17:49Z,2022-01-26T05:17:49Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/a9fca7efa4184fbb2a65ca1275c326950ed9d3c1/sqlite_utils/cli.py#L1014-L1018,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114640101, https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876463,https://api.github.com/repos/simonw/sqlite-utils/issues/391,1021876463,IC_kwDOCGYnMM486Jjv,9599,2022-01-26T05:16:51Z,2022-01-26T05:16:51Z,OWNER,"Actually adding a progress bar may not make sense here: it's designed to work with streaming input from stdin, in which case it's impossible for it to know the overall number of rows to be processed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114638930, https://github.com/simonw/sqlite-utils/issues/391#issuecomment-1021876055,https://api.github.com/repos/simonw/sqlite-utils/issues/391,1021876055,IC_kwDOCGYnMM486JdX,9599,2022-01-26T05:15:58Z,2022-01-26T05:15:58Z,OWNER,Could add support for `--batch-size` as seen in `insert`/`upsert` too - causing it to break the list up into batches and commit for each one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114638930, https://github.com/simonw/datasette/issues/1613#issuecomment-1021860694,https://api.github.com/repos/simonw/datasette/issues/1613,1021860694,IC_kwDOBm6k_c486FtW,9599,2022-01-26T04:57:53Z,2022-01-26T04:57:53Z,OWNER,"The existing flow where you can apply filters to a table and then click ""View and edit SQL"" to see the query is a good starting point. Group by queries are both crucially important and difficult to assemble for beginners. Providing a way to see the query that was used by a facet (since facets are really just group-by-counts) would be very useful, which could come out of this: - #1080","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114628238, https://github.com/simonw/sqlite-utils/issues/390#issuecomment-1021825322,https://api.github.com/repos/simonw/sqlite-utils/issues/390,1021825322,IC_kwDOCGYnMM4859Eq,9599,2022-01-26T03:20:12Z,2022-01-26T03:20:25Z,OWNER,"``` % sqlite-utils upsert trees.db blah - Usage: sqlite-utils upsert [OPTIONS] PATH TABLE FILE Try 'sqlite-utils upsert -h' for help. Error: Missing option '--pk'. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114557284, https://github.com/simonw/sqlite-utils/issues/48#issuecomment-1021790707,https://api.github.com/repos/simonw/sqlite-utils/issues/48,1021790707,IC_kwDOCGYnMM4850nz,9599,2022-01-26T02:08:39Z,2022-01-26T02:08:39Z,OWNER,This is a dupe of #308.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939, https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021790136,https://api.github.com/repos/simonw/sqlite-utils/issues/389,1021790136,IC_kwDOCGYnMM4850e4,9599,2022-01-26T02:07:41Z,2022-01-26T02:07:41Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114544727, https://github.com/simonw/sqlite-utils/issues/389#issuecomment-1021789016,https://api.github.com/repos/simonw/sqlite-utils/issues/389,1021789016,IC_kwDOCGYnMM4850NY,9599,2022-01-26T02:05:41Z,2022-01-26T02:05:41Z,OWNER,"Oops, wrong domain in that - https://plausible.io/sqlite-utils.datasette.io is not yet showing data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114544727, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021788260,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021788260,IC_kwDOCGYnMM4850Bk,9599,2022-01-26T02:04:17Z,2022-01-26T02:04:17Z,OWNER,https://sqlite-utils.datasette.io/en/3.20/ now also shows the version warning banner.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021787822,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021787822,IC_kwDOCGYnMM485z6u,9599,2022-01-26T02:03:25Z,2022-01-26T02:03:25Z,OWNER,"https://sqlite-utils.datasette.io/en/latest/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021785268,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021785268,IC_kwDOCGYnMM485zS0,9599,2022-01-26T01:58:19Z,2022-01-26T01:58:19Z,OWNER,"While I'm modifying that template I'm also going to add Plausible analytics: ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475, https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1021784593,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1021784593,IC_kwDOCGYnMM485zIR,9599,2022-01-26T01:56:48Z,2022-01-26T01:56:48Z,OWNER,"On https://readthedocs.org/dashboard/sqlite-utils/advanced/ I checked this box (previously unchecked): ![CleanShot 2022-01-25 at 17 56 22@2x](https://user-images.githubusercontent.com/9599/151090718-72e553a6-826a-43b6-98bd-f70b0cba8784.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475, https://github.com/simonw/sqlite-utils/issues/387#issuecomment-1021773056,https://api.github.com/repos/simonw/sqlite-utils/issues/387,1021773056,IC_kwDOCGYnMM485wUA,9599,2022-01-26T01:37:17Z,2022-01-26T01:37:17Z,OWNER,https://sqlite-utils.datasette.io/en/latest/python-api.html#getting-started,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1111293050, https://github.com/simonw/datasette/issues/1612#issuecomment-1021497165,https://api.github.com/repos/simonw/datasette/issues/1612,1021497165,IC_kwDOBm6k_c484s9N,639012,2022-01-25T18:44:23Z,2022-01-25T18:44:23Z,CONTRIBUTOR,"OMG, this might be the fastest OS ticket I've ever filed, thanks so much @simonw ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905, https://github.com/simonw/datasette/issues/1612#issuecomment-1021489826,https://api.github.com/repos/simonw/datasette/issues/1612,1021489826,IC_kwDOBm6k_c484rKi,9599,2022-01-25T18:34:21Z,2022-01-25T18:34:21Z,OWNER,"OK, that's live on https://latest.datasette.io/fixtures now - I really like it. Thanks for the suggestion!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905, https://github.com/simonw/datasette/issues/1612#issuecomment-1021477220,https://api.github.com/repos/simonw/datasette/issues/1612,1021477220,IC_kwDOBm6k_c484oFk,9599,2022-01-25T18:19:31Z,2022-01-25T18:19:31Z,OWNER,"Here's something I like: I also added a ""Tables"" `

` heading and bumped the tables themselves down from a `

` to a `

`: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905, https://github.com/simonw/datasette/issues/1612#issuecomment-1021472918,https://api.github.com/repos/simonw/datasette/issues/1612,1021472918,IC_kwDOBm6k_c484nCW,9599,2022-01-25T18:14:27Z,2022-01-25T18:15:54Z,OWNER,"They're currently shown at the very bottom of the page, under the list of tables and far away from the SQL query box: https://latest.datasette.io/fixtures I'm also questioning if ""Queries"" is the best header for this. Other options: - **Canned queries** (what the feature is called in the documentation, but I don't think it's a great user-facing term) - **Saved queries** - overlaps with a mechanism by which queries can be saved by the user using a plugin such as [datasette-saved-queries](https://github.com/simonw/datasette-saved-queries) - though that plugin does itself use the canned queries plugin hook so not completely unrelated - **Sample or Example queries** - I don't like these much because they're more than just examples - they are often the core functionality of the specific customized Datasette instance - **Prepared queries** - overlaps with terminology used in other databases, so not great either - **Pre-configured queries** - urgh, don't like that language, feels clumsy - **Query recipes** - bit out of left-field this one, only really makes sense for queries that include named parameters for specific use-cases Maybe ""Queries"" is right after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905, https://github.com/simonw/datasette/issues/1612#issuecomment-1021413700,https://api.github.com/repos/simonw/datasette/issues/1612,1021413700,IC_kwDOBm6k_c484YlE,9599,2022-01-25T17:07:29Z,2022-01-25T17:07:29Z,OWNER,"That's a much better place for them, I like this idea. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114147905, https://github.com/dogsheep/dogsheep.github.io/pull/6#issuecomment-1021264135,https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/6,1021264135,IC_kwDODMzF1s4830EH,1151557,2022-01-25T14:52:40Z,2022-01-25T14:52:40Z,NONE,"@simonw, could you review?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842765105, https://github.com/simonw/datasette/issues/1609#issuecomment-1020456608,https://api.github.com/repos/simonw/datasette/issues/1609,1020456608,IC_kwDOBm6k_c480u6g,9599,2022-01-24T19:20:09Z,2022-01-24T19:20:09Z,OWNER,Uvicorn have a release out now that would have fixed this issue if I hadn't shipped my own fix: https://github.com/encode/uvicorn/releases/tag/0.17.0.post1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1605#issuecomment-1018778667,https://api.github.com/repos/simonw/datasette/issues/1605,1018778667,IC_kwDOBm6k_c48uVQr,25778,2022-01-21T19:00:01Z,2022-01-21T19:00:01Z,CONTRIBUTOR,"Let me know if you want help prototyping any of this, because I'm thinking about it and trying stuff out. Happy to be a sounding board, if it helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952, https://github.com/simonw/datasette/issues/1605#issuecomment-1018766727,https://api.github.com/repos/simonw/datasette/issues/1605,1018766727,IC_kwDOBm6k_c48uSWH,9599,2022-01-21T18:41:21Z,2022-01-21T18:42:03Z,OWNER,"Yeah I think this all hinges on: - #1101 Also this comment about streaming full JSON arrays (not just newline-delimited) using [this trick](https://til.simonwillison.net/python/output-json-array-streaming): - https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553 I'm about ready to figure these out, as with so much it's still a little bit blocked on the refactor stuff from: - #1518 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952, https://github.com/simonw/datasette/issues/1605#issuecomment-1018741262,https://api.github.com/repos/simonw/datasette/issues/1605,1018741262,IC_kwDOBm6k_c48uMIO,25778,2022-01-21T18:05:09Z,2022-01-21T18:05:09Z,CONTRIBUTOR,"Thinking about this more, as well as #1356 and various other tickets related to output formats, I think there's a missing plugin hook for formatting results, separate from `register_output_renderer` (or maybe part of it, depending on #1101). Right now, as I understand it, getting output in any format goes through the normal view stack -- a table, a row or a query -- and so by the time `register_output_renderer` gets it, the results have already been truncated or paginated. What I'd want, I think, is to be able to register ways to format results independent of where those results are sent. It's possible this could be done using [`conn.row_factory`](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.row_factory) (maybe in the `prepare_connection` hook), but I'm not sure that's where it belongs. Another option is some kind of registry of serializers, which `register_output_renderer` and other plugin hooks could use. What I'm trying to avoid here is writing a plugin that also needs plugins for formats I haven't thought of yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952, https://github.com/simonw/datasette/issues/1609#issuecomment-1018104868,https://api.github.com/repos/simonw/datasette/issues/1609,1018104868,IC_kwDOBm6k_c48rwwk,9599,2022-01-21T02:24:13Z,2022-01-21T02:24:13Z,OWNER,Just shipped 0.60.1 with the fix - and tested that `pip install datasette` does indeed work correctly on Python 3.6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018094767,https://api.github.com/repos/simonw/datasette/issues/1609,1018094767,IC_kwDOBm6k_c48ruSv,9599,2022-01-21T02:04:14Z,2022-01-21T02:04:14Z,OWNER,So I don't need to release 0.60.1 AND 0.60.2 after all - I can just release 0.60.1 with a bug fix that it no longer breaks installation for Python 3.6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018092984,https://api.github.com/repos/simonw/datasette/issues/1609,1018092984,IC_kwDOBm6k_c48rt24,9599,2022-01-21T02:00:38Z,2022-01-21T02:00:38Z,OWNER,"Out of curiosity, I installed this latest `setup.py` file using both Python 3.6 and Python 3.10, ran `pip freeze` on both of them and created a Gist to compare the difference. The result is here: https://gist.github.com/simonw/2e7d5b1beba675ef9a5bcd310cadc372/revisions From that, it looks like the Python packages in my dependencies which have released new versions that don't work with Python 3.6 are: - https://pypi.org/project/janus/#history - https://pypi.org/project/Pint/#history - https://pypi.org/project/platformdirs/#history - https://pypi.org/project/uvicorn/#history (already discussed) Sure enough, for the first three of those browsing through their recent versions on PyPI confirms that they switched from e.g. ""Requires: Python >=3.6"" on https://pypi.org/project/janus/0.7.0/ to ""Requires: Python >=3.7"" on https://pypi.org/project/janus/1.0.0/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018091322,https://api.github.com/repos/simonw/datasette/issues/1609,1018091322,IC_kwDOBm6k_c48rtc6,9599,2022-01-21T01:56:42Z,2022-01-21T01:56:42Z,OWNER,"OK, the environment markers approach seems to work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018086697,https://api.github.com/repos/simonw/datasette/issues/1609,1018086697,IC_kwDOBm6k_c48rsUp,9599,2022-01-21T01:46:43Z,2022-01-21T01:46:43Z,OWNER,https://github.com/simonw/datasette/runs/4890775227?check_suite_focus=true - the tests passed on Python 3.6 for this commit with the pinned dependencies: https://github.com/simonw/datasette/commit/41060e7e7cb838328c879de6a98ae794dc1886d0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018086273,https://api.github.com/repos/simonw/datasette/issues/1609,1018086273,IC_kwDOBm6k_c48rsOB,9599,2022-01-21T01:45:46Z,2022-01-21T01:45:46Z,OWNER,"This whole thing reminds me of my ongoing internal debate about version pinning: should the Datasette package released to PyPI pin to the exact versions of the dependencies that are known to work, or should it allow a range of dependencies so users can pick other versions of the dependencies to use in their environment? As I understand it, the general rule is to use exact pinning for applications but use ranges for libraries. Datasette is almost entirely an application... but it can also be used as a library - and in fact I'm hoping to encourage that usage more in the future, see: - #1398 I'd also like to release a packaged version of Datasette that doesn't require Uvicorn, for running on AWS Lambda and other function-as-a-service platforms. Those platforms have their own HTTP layer and hence don't need the Uvicorn dependency. Maybe the answer is to have a `datasette-core` package which provides the core of Datasette with unpinned dependencies and no Uvicorn, and then have the existing `datasette` package provide the Datasette CLI tool with Uvicorn and pinned dependencies?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018082792,https://api.github.com/repos/simonw/datasette/issues/1609,1018082792,IC_kwDOBm6k_c48rrXo,9599,2022-01-21T01:37:11Z,2022-01-21T01:37:11Z,OWNER,"Another option from https://twitter.com/samuel_hames/status/1484327636860293121 - environment markers, described in https://www.python.org/dev/peps/pep-0508/#environment-markers Found some examples of those in use using GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=%22%3Bpython_version%22+path%3Asetup.py - in particular https://github.com/xmendez/wfuzz/blob/1b695ee9a87d66a7d7bf6cae70d60a33fae51541/setup.py#L31-L38 ```python install_requires = [ 'pycurl', 'pyparsing<2.4.2;python_version<=""3.4""', 'pyparsing>=2.4*;python_version>=""3.5""', 'six', 'configparser;python_version<""3.5""', 'chardet', ] ``` So maybe I can ship 0.60.1 with loose dependencies _except_ for the `uvicorn` one on Python 3.6, using an environment marker. Here's my `setup.py` at the moment: https://github.com/simonw/datasette/blob/ffca55dfd7cc9b53522c2e5a2fa1ff67c9beadf2/setup.py#L44-L61 One other problem: there might be packages in that list right now which don't specify their 3.6 Python version but which will, at some point in the future, release a new version that doesn't work with 3.6 (like Uvicorn did) - in which case Python 3.6 installs would break in the future. I think what I'll do then is ship the `0.60.1` Python 3.6 version with strict upper limits on each version which are the current, tested-with-Datasette-on-Python3.6 versions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018077009,https://api.github.com/repos/simonw/datasette/issues/1609,1018077009,IC_kwDOBm6k_c48rp9R,9599,2022-01-21T01:24:15Z,2022-01-21T01:24:43Z,OWNER,"Problem: if I ship this, it will be the most recent release of Datasette - but unlike other previous releases it has exactly pinned versions of all of the dependencies. Which is bad for people who run `pip install datasette` but want to not be stuck to those exact library versions. So maybe I ship this as 0.60.1, then ship a 0.60.2 release directly afterwards which unpins the dependencies again and requires Python 3.7?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018075357,https://api.github.com/repos/simonw/datasette/issues/1609,1018075357,IC_kwDOBm6k_c48rpjd,9599,2022-01-21T01:20:56Z,2022-01-21T01:20:56Z,OWNER,"I used the combo of `pyenv` and `pipenv` to run tests and figure out what the most recent versions of each dependency were that worked on Python 3.6. I also clicked around in the latest releases on pages such as https://pypi.org/project/aiofiles ``` cd /tmp git clone git@github.com:simonw/datasette cd /tmp/datasette pipenv shell --python 3.6.10 pip install -e '.[test]' pytest ``` I also used `pip freeze | grep black` to see which version was installed, since packages with `python_requires=` in them would automatically install the highest compatible version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018064620,https://api.github.com/repos/simonw/datasette/issues/1609,1018064620,IC_kwDOBm6k_c48rm7s,9599,2022-01-21T01:00:12Z,2022-01-21T01:00:12Z,OWNER,"I think there are two possible solutions then: 1. Convince Uvicorn to publish one last `0.16.1` version which includes that `python_requires=` line, such that there's a version of Uvicorn on PyPI that Python 3.6 can still install. 2. Release a `0.60.1` version of Datasette which pins that Uvicorn version, and hence can be installed. I've made the request for 1) in Uvicorn Gitter here: https://gitter.im/encode/community?at=61ea044a6d9ba23328d0fa28 I'm going to investigate option 2) myself now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1609#issuecomment-1018063681,https://api.github.com/repos/simonw/datasette/issues/1609,1018063681,IC_kwDOBm6k_c48rmtB,9599,2022-01-21T00:58:25Z,2022-01-21T00:58:32Z,OWNER,"On Twitter: https://twitter.com/simonw/status/1484317711672877065 Here's the problem: Uvicorn only added `python_requires` to their `setup.py` a few days ago, which means the releases they have out on PyPI at the moment don't specify the Python version they need, which is why this mechanism doesn't work as expected: - https://github.com/encode/uvicorn/pull/1328","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109884720, https://github.com/simonw/datasette/issues/1608#issuecomment-1018017637,https://api.github.com/repos/simonw/datasette/issues/1608,1018017637,IC_kwDOBm6k_c48rbdl,9599,2022-01-20T23:27:59Z,2022-01-20T23:27:59Z,OWNER,"Got a couple of TILs out of this: - [Promoting the stable version of the documentation using rel=canonical](https://til.simonwillison.net/readthedocs/documentation-seo-canonical) - [Linking from /latest/ to /stable/ on Read The Docs](https://til.simonwillison.net/readthedocs/link-from-latest-to-stable)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017998993,https://api.github.com/repos/simonw/datasette/issues/1608,1017998993,IC_kwDOBm6k_c48rW6R,9599,2022-01-20T22:56:00Z,2022-01-20T22:56:00Z,OWNER,"> https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for ""You are looking at v0.36 but you should be looking at 0.40"" but doesn't hand the case I need here which is ""you are looking at /latest/ but you should be looking at /stable/"". Correction! That tool DOES support that, as can be seen in their example configuration for their own documentation: https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38 ```python versionwarning_messages = { 'latest': 'This is a custom message only for version ""latest"" of this documentation.', } versionwarning_admonition_type = 'tip' versionwarning_banner_title = 'Tip' versionwarning_body_selector = 'div[itemprop=""articleBody""]' ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017994925,https://api.github.com/repos/simonw/datasette/issues/1608,1017994925,IC_kwDOBm6k_c48rV6t,9599,2022-01-20T22:48:43Z,2022-01-20T22:54:02Z,OWNER,"https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for ""You are looking at v0.36 but you should be looking at 0.40"" but doesn't hand the case I need here which is ""you are looking at /latest/ but you should be looking at /stable/"". Just shipped my fix here: https://docs.datasette.io/en/latest/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017993482,https://api.github.com/repos/simonw/datasette/issues/1608,1017993482,IC_kwDOBm6k_c48rVkK,316517,2022-01-20T22:46:16Z,2022-01-20T22:46:16Z,NONE,Or you can use https://sphinx-version-warning.readthedocs.io/! 😄 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017988556,https://api.github.com/repos/simonw/datasette/issues/1608,1017988556,IC_kwDOBm6k_c48rUXM,9599,2022-01-20T22:37:51Z,2022-01-20T22:37:51Z,OWNER,"Here's a jQuery recipe that seems to do the right thing: ```javascript jQuery(function ($) { // If this is a /latest/ doc page, show banner linking to /stable/ if (!/\/latest\//.test(location.pathname)) { return; } var stableUrl = location.pathname.replace(""/latest/"", ""/stable/""); // Check it's not a 404 fetch(stableUrl, { method: ""HEAD"" }).then((response) => { if (response.status == 200) { var warning = $( `

Note

This documentation covers the development version of Datasette.

See this page for the current stable release.

` ); warning.find(""a"").attr(""href"", stableUrl); var body = $(""div.body""); if (!body.length) { body = $(""div.document""); } body.prepend(warning); } }); }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017975322,https://api.github.com/repos/simonw/datasette/issues/1608,1017975322,IC_kwDOBm6k_c48rRIa,9599,2022-01-20T22:17:01Z,2022-01-20T22:27:07Z,OWNER,"Turns out that banner is something that ReadTheDocs implemented - I found it using GitHub code search, it's produced by this piece of JavaScript: https://github.com/readthedocs/readthedocs.org/blob/0852d7c10d725d954d3e9a93513171baa1116d9f/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L13-L21 ```javascript function init(data) { var rtd = rtddata.get(); /// Out of date message if (data.is_highest) { return; } var currentURL = window.location.pathname.replace(rtd['version'], data.slug); var warning = $( '
' + '

Note

' + '

' + 'You are not reading the most recent version of this documentation. ' + ' is the latest version available.' + '

' + '
'); warning .find('a') .attr('href', currentURL) .text(data.slug); var body = $(""div.body""); if (!body.length) { body = $(""div.document""); } body.prepend(warning); } ``` And here's where that module is called from the rest of their code: https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/core/static-src/core/js/doc-embed/footer.js#L66-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017981599,https://api.github.com/repos/simonw/datasette/issues/1608,1017981599,IC_kwDOBm6k_c48rSqf,9599,2022-01-20T22:26:32Z,2022-01-20T22:26:32Z,OWNER,I'm tempted to imitate their JavaScript but check for `/latest/` in the URL and use that to append a similar message warning about this being the documentation for the in-development version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017971905,https://api.github.com/repos/simonw/datasette/issues/1608,1017971905,IC_kwDOBm6k_c48rQTB,9599,2022-01-20T22:11:40Z,2022-01-20T22:11:40Z,OWNER,"Huh, I had forgotten I already have a banner on older versions: ![D1A65C68-9A37-4FA2-80C4-534739A9D292](https://user-images.githubusercontent.com/9599/150430410-1e22e23f-ed27-4271-89ff-63467eb5f466.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017970132,https://api.github.com/repos/simonw/datasette/issues/1608,1017970132,IC_kwDOBm6k_c48rP3U,9599,2022-01-20T22:08:55Z,2022-01-20T22:08:55Z,OWNER,"Might want to consider SEO here too - I want people from search engines to land on `/stable/`, I wonder if I should noindex or `rel=canonical` the other documentation versions? Not sure what best practices for that is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1608#issuecomment-1017969452,https://api.github.com/repos/simonw/datasette/issues/1608,1017969452,IC_kwDOBm6k_c48rPss,9599,2022-01-20T22:07:55Z,2022-01-20T22:07:55Z,OWNER,"I think I want a banner at the top of the page making it obvious which version the documentation is talking about. This can be pretty low key for the current stable release, but should be visually more prominent for the `/latest/` branch and for older releases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154, https://github.com/simonw/datasette/issues/1603#issuecomment-1017842366,https://api.github.com/repos/simonw/datasette/issues/1603,1017842366,IC_kwDOBm6k_c48qwq-,9599,2022-01-20T19:19:54Z,2022-01-20T19:19:54Z,OWNER,Wrote up a TIL: https://til.simonwillison.net/pixelmator/pixel-editing-favicon,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694, https://github.com/simonw/datasette/issues/1603#issuecomment-1017808898,https://api.github.com/repos/simonw/datasette/issues/1603,1017808898,IC_kwDOBm6k_c48qogC,9599,2022-01-20T18:42:35Z,2022-01-20T18:42:35Z,OWNER,"Resized it down to 208 bytes with https://squoosh.app ![favicon](https://user-images.githubusercontent.com/9599/150401709-11a50492-a8c8-4eee-848c-c813f8cd3e4e.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694, https://github.com/simonw/datasette/issues/1603#issuecomment-1017806497,https://api.github.com/repos/simonw/datasette/issues/1603,1017806497,IC_kwDOBm6k_c48qn6h,9599,2022-01-20T18:39:27Z,2022-01-20T18:39:57Z,OWNER,"Here's a comparison between my hand-edited version and the one I have now: ![CleanShot 2022-01-20 at 10 38 00@2x](https://user-images.githubusercontent.com/9599/150401244-7e78ee93-1973-4c95-8f91-01e51e0d5366.png) The new 32x32 image: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694, https://github.com/simonw/datasette/issues/1603#issuecomment-1017800506,https://api.github.com/repos/simonw/datasette/issues/1603,1017800506,IC_kwDOBm6k_c48qmc6,9599,2022-01-20T18:31:18Z,2022-01-20T18:31:18Z,OWNER,"One last go at tidying this up. I decided to do a 32x32 pixel version in Pixelmator, using this trick to access a pixel brush: https://osxdaily.com/2016/11/17/enable-pixel-brush-pixelmator-mac/ Frustrating how the white boxes are all exactly four pixels high and ALMOST all four pixels wide, but one of them has to be three pixels wide to fit the space.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108235694,