html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/1034#issuecomment-713176082,https://api.github.com/repos/simonw/datasette/issues/1034,713176082,MDEyOklzc3VlQ29tbWVudDcxMzE3NjA4Mg==,9599,simonw,2020-10-20T22:27:33Z,2020-10-20T22:27:33Z,OWNER,"This feels good to me - it's consistent with how other features in Datasette work, and it means users who need the binary data in CSV (for whatever reason) can get it if they want to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713175741,https://api.github.com/repos/simonw/datasette/issues/1034,713175741,MDEyOklzc3VlQ29tbWVudDcxMzE3NTc0MQ==,9599,simonw,2020-10-20T22:26:45Z,2020-10-20T22:26:45Z,OWNER,"> New idea: since binary in CSV doesn't make sense anyway, emulate Datasette's HTML UI default and output this:
>
> id,title,data
> 1,Some title,
> 2,Other title,
>
> Then allow users to add ?_base64=1 to the URL to get base64 instead
> https://twitter.com/simonw/status/1318679950635888641","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713174690,https://api.github.com/repos/simonw/datasette/issues/1034,713174690,MDEyOklzc3VlQ29tbWVudDcxMzE3NDY5MA==,9599,simonw,2020-10-20T22:23:50Z,2020-10-20T22:23:50Z,OWNER,Or... default to `` and support a `?_base64=1` option which outputs in base64 instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713174341,https://api.github.com/repos/simonw/datasette/issues/1034,713174341,MDEyOklzc3VlQ29tbWVudDcxMzE3NDM0MQ==,9599,simonw,2020-10-20T22:22:53Z,2020-10-20T22:23:14Z,OWNER,"An even easier option: do what the Datasette UI does and output `` for that CSV cell, as seen on https://latest.datasette.io/fixtures/binary_data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713172901,https://api.github.com/repos/simonw/datasette/issues/1034,713172901,MDEyOklzc3VlQ29tbWVudDcxMzE3MjkwMQ==,9599,simonw,2020-10-20T22:19:10Z,2020-10-20T22:20:28Z,OWNER,"I could go with the same format as `datasette-render-binary` but using `0x00` as the format for the hex bytes.
0x15 0x1C 0x02 0xC7 JFIF 0x00 0x01
Problem with this is that it's ambiguous: if the ASCII characters `0x15` occur in the text they will be indistinguishable from those hex bytes.
But since representing binary data in CSV fundamentally doesn't make sense I'm not sure if that really matters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/741#issuecomment-713171742,https://api.github.com/repos/simonw/datasette/issues/741,713171742,MDEyOklzc3VlQ29tbWVudDcxMzE3MTc0Mg==,9599,simonw,2020-10-20T22:16:25Z,2020-10-20T22:16:25Z,OWNER,See also #992 which will rename `--config` to `--setting`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/262#issuecomment-713170979,https://api.github.com/repos/simonw/datasette/issues/262,713170979,MDEyOklzc3VlQ29tbWVudDcxMzE3MDk3OQ==,9599,simonw,2020-10-20T22:14:37Z,2020-10-20T22:14:37Z,OWNER,"I think it's worth having a plugin hook for this - it can be same hook that is used internally. Maybe `register_extra` - it lets you return one or more `extra` implementations, each with a name and an async function that gets called.
Things like suggested facets will become `register_extra` hooks. Maybe actual facets too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/262#issuecomment-713170284,https://api.github.com/repos/simonw/datasette/issues/262,713170284,MDEyOklzc3VlQ29tbWVudDcxMzE3MDI4NA==,9599,simonw,2020-10-20T22:13:01Z,2020-10-20T22:13:01Z,OWNER,In the documentation for `?_extra=` I think I'll emphasize the comma-separated version of it. Also: there will be `?_extra=` values which act as aliases for collection combinations - e.g. `?_extra=full` will toggle everything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/782#issuecomment-712986115,https://api.github.com/repos/simonw/datasette/issues/782,712986115,MDEyOklzc3VlQ29tbWVudDcxMjk4NjExNQ==,9599,simonw,2020-10-20T16:28:46Z,2020-10-20T16:29:51Z,OWNER,"I think this all comes down to how the `?_extras=` mechanism works (see #262), as first hinted at in a30c5b220c15360d575e94b0e67f3255e120b916 (see commit message) when I added this long-forgotten undocumented feature: https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables
Extras need to be able to execute additional SQL, since that would solve the problem we have now where the expensive ""suggested facets"" code runs on all `.json` output even when its results are not being shown.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1035#issuecomment-712976314,https://api.github.com/repos/simonw/datasette/issues/1035,712976314,MDEyOklzc3VlQ29tbWVudDcxMjk3NjMxNA==,9599,simonw,2020-10-20T16:21:42Z,2020-10-20T16:21:42Z,OWNER,"Makes me question if `datasette.urls` should grow functionality equivalent to the other path and querystring manipulation methods in `datasette.utils`:
https://github.com/simonw/datasette/blob/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc/datasette/utils/__init__.py#L216-L279","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1035#issuecomment-712965574,https://api.github.com/repos/simonw/datasette/issues/1035,712965574,MDEyOklzc3VlQ29tbWVudDcxMjk2NTU3NA==,9599,simonw,2020-10-20T16:13:57Z,2020-10-20T16:13:57Z,OWNER,"That `renderers[key] = path_with_format(` is in a base class which can be used for both arbitrary queries, canned queries and the table view. I think that's OK, but it means that the `format=""json""` argument on `datasette.urls.table()` won't be used by Datasette internally, it will just be available for plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1035#issuecomment-712963959,https://api.github.com/repos/simonw/datasette/issues/1035,712963959,MDEyOklzc3VlQ29tbWVudDcxMjk2Mzk1OQ==,9599,simonw,2020-10-20T16:11:25Z,2020-10-20T16:11:25Z,OWNER,"Relevant code:
https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/utils/__init__.py#L681-L696
Only used here:
https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/views/base.py#L498-L502","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1026#issuecomment-712962517,https://api.github.com/repos/simonw/datasette/issues/1026,712962517,MDEyOklzc3VlQ29tbWVudDcxMjk2MjUxNw==,9599,simonw,2020-10-20T16:09:12Z,2020-10-20T16:09:12Z,OWNER,"That `datasette.urls.table(""db"", ""table"") + "".json""` example is bad because if the table name contains a `.` it should be `?_format=json` instead.
Maybe `.table()` should have a `format=""json""` option that knows how to do this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712959034,https://api.github.com/repos/simonw/datasette/issues/1026,712959034,MDEyOklzc3VlQ29tbWVudDcxMjk1OTAzNA==,9599,simonw,2020-10-20T16:03:33Z,2020-10-20T16:03:55Z,OWNER,"Reconsidering this: I think the `.get()` etc methods should automatically add the `base_url` prefix for you, since these APIs are only intended to make internal calls.
The clincher on this is when I went to add a section to the `datasette.client` documentation recommending you use `datasette.urls.path()` for every call to them that you make.
But there's a problem: to handle table name escaping users are likely to want to use `datasette.urls.table()` anyway, like this:
response = await datasette.client.get(datasette.urls.table(""db"", ""table"") + "".json"")
This risks adding the `base_url` prefix twice. Maybe the `.table()` method could return a string-like object that is marked as already having the `base_url` prefix added, so the `client.get()` method knows not to add it again.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/991#issuecomment-712855389,https://api.github.com/repos/simonw/datasette/issues/991,712855389,MDEyOklzc3VlQ29tbWVudDcxMjg1NTM4OQ==,24740,furilo,2020-10-20T13:36:41Z,2020-10-20T13:36:41Z,NONE,"Here is one quick sketch (done in Figma :P) for an idea: a possible filter to switch between showing all tables from all databases, or grouping tables by database.
(the switch is interactive)
All tables: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A2&viewport=536%2C348%2C0.5&scaling=min-zoom
Grouped: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=3%3A974&viewport=536%2C348%2C0.5&scaling=min-zoom
When only 1 database: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A162&viewport=536%2C348%2C0.5&scaling=min-zoom
Is this is useful, I can send some more suggestions/sketches.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/1023#issuecomment-712607608,https://api.github.com/repos/simonw/datasette/issues/1023,712607608,MDEyOklzc3VlQ29tbWVudDcxMjYwNzYwOA==,9599,simonw,2020-10-20T05:47:42Z,2020-10-20T05:47:42Z,OWNER,Requested alpha testers in https://github.com/simonw/datasette/issues/838#issuecomment-712604364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712607227,https://api.github.com/repos/simonw/datasette/issues/1026,712607227,MDEyOklzc3VlQ29tbWVudDcxMjYwNzIyNw==,9599,simonw,2020-10-20T05:46:44Z,2020-10-20T05:46:44Z,OWNER,We have a solution for this now: `datasette.urls` from #1033 can be used by plugins to assemble the correct URLs to pass to `.get()` and friends.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-712604541,https://api.github.com/repos/simonw/datasette/issues/1023,712604541,MDEyOklzc3VlQ29tbWVudDcxMjYwNDU0MQ==,9599,simonw,2020-10-20T05:39:44Z,2020-10-20T05:39:44Z,OWNER,Here's the alpha with most of this work ready for people to preview: https://github.com/simonw/datasette/releases/tag/0.51a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/838#issuecomment-712604364,https://api.github.com/repos/simonw/datasette/issues/838,712604364,MDEyOklzc3VlQ29tbWVudDcxMjYwNDM2NA==,9599,simonw,2020-10-20T05:39:15Z,2020-10-20T05:39:15Z,OWNER,"OK, I've made a ton of improvements to how the `base_url` setting works - see tickets linked from #1023. I've just pushed out an alpha release with those changes in it: https://github.com/simonw/datasette/releases/tag/0.51a0
@tsibley @tballison @ChristopherWilks I'd really appreciate your help testing this alpha!
You can install it with:
pip install datasette==0.51a0
It should work with just `ProxyPass`, without needing the `ProxyPassReverse` setting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/865#issuecomment-712597762,https://api.github.com/repos/simonw/datasette/issues/865,712597762,MDEyOklzc3VlQ29tbWVudDcxMjU5Nzc2Mg==,9599,simonw,2020-10-20T05:22:59Z,2020-10-20T05:22:59Z,OWNER,"OK, this is definitely working now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/1025#issuecomment-712593790,https://api.github.com/repos/simonw/datasette/issues/1025,712593790,MDEyOklzc3VlQ29tbWVudDcxMjU5Mzc5MA==,9599,simonw,2020-10-20T05:12:36Z,2020-10-20T05:12:36Z,OWNER,"I'm going to leave the cookies code setting cookies to default to the `""/""` top level.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/782#issuecomment-712590398,https://api.github.com/repos/simonw/datasette/issues/782,712590398,MDEyOklzc3VlQ29tbWVudDcxMjU5MDM5OA==,9599,simonw,2020-10-20T05:03:46Z,2020-10-20T05:04:09Z,OWNER,"OK, https://latest-with-plugins.datasette.io/ is running that now - e.g. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions.json-preview or https://latest-with-plugins.datasette.io/fixtures/compound_three_primary_keys.json-preview
```json
{
""rows"": [
{
""pk"": 1,
""name"": ""The Mystery Spot"",
""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
""latitude"": 37.0167,
""longitude"": -122.0024
},
{
""pk"": 2,
""name"": ""Winchester Mystery House"",
""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
""latitude"": 37.3184,
""longitude"": -121.9511
},
{
""pk"": 3,
""name"": ""Burlingame Museum of PEZ Memorabilia"",
""address"": ""214 California Drive, Burlingame, CA 94010"",
""latitude"": 37.5793,
""longitude"": -122.3442
},
{
""pk"": 4,
""name"": ""Bigfoot Discovery Museum"",
""address"": ""5497 Highway 9, Felton, CA 95018"",
""latitude"": 37.0414,
""longitude"": -122.0725
}
],
""total"": 4,
""next_url"": null
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-712585921,https://api.github.com/repos/simonw/datasette/issues/782,712585921,MDEyOklzc3VlQ29tbWVudDcxMjU4NTkyMQ==,9599,simonw,2020-10-20T04:48:01Z,2020-10-20T04:48:01Z,OWNER,I'll update `datasette-json-preview` with that now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-712585687,https://api.github.com/repos/simonw/datasette/issues/782,712585687,MDEyOklzc3VlQ29tbWVudDcxMjU4NTY4Nw==,9599,simonw,2020-10-20T04:47:02Z,2020-10-20T04:47:12Z,OWNER,"Great point about CORS, I hadn't considered that.
I think I'm going to keep the `Link:` header (added in #1014) because I quite enjoy using it with GitHub and WordPress, but I'm not going to have it be the default way of doing pagination. For the default shape I'm now leaning towards this:
```json
{
""total"": 36,
""rows"": [{""id"": 1, ""name"": ""Cleo""}],
""next_url"": ""https://latest-with-plugins.datasette.io/fixtures/facetable.json?_next=5""
}
```
So three keys: `total`, `rows` and `next_url`. Then extra keys can be added using `?_extra=` with various named bundles.","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1034#issuecomment-712582699,https://api.github.com/repos/simonw/datasette/issues/1034,712582699,MDEyOklzc3VlQ29tbWVudDcxMjU4MjY5OQ==,9599,simonw,2020-10-20T04:36:04Z,2020-10-20T04:36:14Z,OWNER,Asked for ideas on Twitter: https://twitter.com/simonw/status/1318409558805467136,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-712581994,https://api.github.com/repos/simonw/datasette/issues/1034,712581994,MDEyOklzc3VlQ29tbWVudDcxMjU4MTk5NA==,9599,simonw,2020-10-20T04:33:28Z,2020-10-20T04:33:28Z,OWNER,"The [datasette-render-binary](https://github.com/simonw/datasette-render-binary) plugin does this, which I really like - but without the different coloured fonts I'm not sure how readable it would be as just plain text:
![image](https://user-images.githubusercontent.com/9599/96540435-9c125f00-1252-11eb-85aa-5fc8d0e63728.png)
Really the goal here is to find the most human-friendly option, so that people looking at the output have a vague idea what's going on. That's why I'm not leaping at the chance to use base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-712580976,https://api.github.com/repos/simonw/datasette/issues/1034,712580976,MDEyOklzc3VlQ29tbWVudDcxMjU4MDk3Ng==,9599,simonw,2020-10-20T04:29:23Z,2020-10-20T04:29:23Z,OWNER,Most obvious option is base64. Any other potential solutions I'm missing?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1025#issuecomment-712579674,https://api.github.com/repos/simonw/datasette/issues/1025,712579674,MDEyOklzc3VlQ29tbWVudDcxMjU3OTY3NA==,9599,simonw,2020-10-20T04:24:10Z,2020-10-20T04:24:10Z,OWNER,"Changed my mind, `error.html` needs access to `urls` in order to link to its CSS file. Passing it after all (it already got passed `ds.config(""base_url"")` so `ds` was available previously).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/782#issuecomment-712569695,https://api.github.com/repos/simonw/datasette/issues/782,712569695,MDEyOklzc3VlQ29tbWVudDcxMjU2OTY5NQ==,222245,carlmjohnson,2020-10-20T03:45:48Z,2020-10-20T03:46:14Z,NONE,"I vote against headers. It has a lot of strikes against it: poor discoverability, new developers often don’t know how to use them, makes CORS harder, makes it hard to use eg with JQ, needs ad hoc specification for each bit of metadata, etc.
The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1025#issuecomment-712481127,https://api.github.com/repos/simonw/datasette/issues/1025,712481127,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTEyNw==,9599,simonw,2020-10-19T22:40:37Z,2020-10-20T01:21:36Z,OWNER,Was blocked on #904 - now unblocked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1033#issuecomment-712529413,https://api.github.com/repos/simonw/datasette/issues/1033,712529413,MDEyOklzc3VlQ29tbWVudDcxMjUyOTQxMw==,9599,simonw,2020-10-20T01:21:12Z,2020-10-20T01:21:12Z,OWNER,Also refs #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/1025#issuecomment-712525557,https://api.github.com/repos/simonw/datasette/issues/1025,712525557,MDEyOklzc3VlQ29tbWVudDcxMjUyNTU1Nw==,9599,simonw,2020-10-20T01:07:02Z,2020-10-20T01:07:02Z,OWNER,"I fixed the `queries.html` one. I'm not going to fix these two:
```
datasette/templates/error.html: home
datasette/templates/patterns.html: home /
```
Because the `error.html` one does not get passed a context (which makes sense since an error has occurred) and the pattern portfolio doesn't need to link to anywhere in particular.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/904#issuecomment-712524699,https://api.github.com/repos/simonw/datasette/issues/904,712524699,MDEyOklzc3VlQ29tbWVudDcxMjUyNDY5OQ==,9599,simonw,2020-10-20T01:04:12Z,2020-10-20T01:04:12Z,OWNER,Documentation is https://docs.datasette.io/en/latest/writing_plugins.html#building-urls-within-plugins and https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712483066,https://api.github.com/repos/simonw/datasette/issues/904,712483066,MDEyOklzc3VlQ29tbWVudDcxMjQ4MzA2Ng==,9599,simonw,2020-10-19T22:46:48Z,2020-10-19T22:46:48Z,OWNER,"OK, I'm committing to `datasette.urls.table()` and friends, which will be available to (and documented for use in) plugins and will be exposed to templates as `{{ urls.table(...) }}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/1020#issuecomment-712482504,https://api.github.com/repos/simonw/datasette/issues/1020,712482504,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjUwNA==,9599,simonw,2020-10-19T22:45:01Z,2020-10-19T22:45:01Z,OWNER,"I'm having trouble coming up with the syntax for this. Here's one option:
```python
response = await datasette.client.get(path, request=request)
```
But this feels confusing to me. We're not using the `request=` argument as a request - we're using it as a source of some default request values (the cookies and incoming headers, but not the path).
We're essentially combining that request with the other arguments passed to `.get()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712482015,https://api.github.com/repos/simonw/datasette/issues/1020,712482015,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjAxNQ==,9599,simonw,2020-10-19T22:43:24Z,2020-10-19T22:43:24Z,OWNER,"... unless I want to support authentication mechanisms that work based on incoming IP address instead, in which case there's an argument for copying more over from the incoming request.
Tailscale is a good example of a system where authentication based on IP address can actually work well, so this is worth doing. Also, there might be authentication mechanisms which work by setting a custom header on the incoming request (not to mention the `Authorization` header).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712481568,https://api.github.com/repos/simonw/datasette/issues/1020,712481568,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTU2OA==,9599,simonw,2020-10-19T22:41:59Z,2020-10-19T22:41:59Z,OWNER,"It turns out this works just fine:
```python
response = await datasette.client.get(path, cookies=request.cookies)
```
So I don't need a mechanism for this. I'm going to add this to the documentation instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1028#issuecomment-712480866,https://api.github.com/repos/simonw/datasette/issues/1028,712480866,MDEyOklzc3VlQ29tbWVudDcxMjQ4MDg2Ng==,9599,simonw,2020-10-19T22:39:51Z,2020-10-19T22:39:51Z,OWNER,Documentation: https://docs.datasette.io/en/latest/spatialite.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723803777,--load-extension=spatialite shortcut,
https://github.com/simonw/datasette/issues/1032#issuecomment-712397537,https://api.github.com/repos/simonw/datasette/issues/1032,712397537,MDEyOklzc3VlQ29tbWVudDcxMjM5NzUzNw==,236498,saulpw,2020-10-19T19:37:55Z,2020-10-19T19:37:55Z,NONE,"python-dateutil is awesome, but it can only guess at one date at a time. So if you have a column of dates that are (presumably) in the same format, it can't use the full set of dates to deduce the format. Also, once it has parsed a date, you can't get the format it used, whether to parse or render other dates. These limitations prevent it from being a silver bullet for date parsing, though they're not enough for me to stop using it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712367285,https://api.github.com/repos/simonw/datasette/issues/1032,712367285,MDEyOklzc3VlQ29tbWVudDcxMjM2NzI4NQ==,9599,simonw,2020-10-19T18:39:32Z,2020-10-19T18:39:32Z,OWNER,https://github.com/digital-land/brownfield-land-collection/blob/a09ddf9960a6af59e72dc02448f7b645e59bf227/bin/harmonise.py#L217-L247 is a beautiful example of this problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712365439,https://api.github.com/repos/simonw/datasette/issues/1032,712365439,MDEyOklzc3VlQ29tbWVudDcxMjM2NTQzOQ==,9599,simonw,2020-10-19T18:35:50Z,2020-10-19T18:37:57Z,OWNER,"Maybe I don't need to add `dateutil` as a dependency here?
`pendulum` and `arrow` both depend on it so it's pretty deeply embedded in the Python date ecosystem. Adding it as a dependency seems reasonable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712365236,https://api.github.com/repos/simonw/datasette/issues/1032,712365236,MDEyOklzc3VlQ29tbWVudDcxMjM2NTIzNg==,9599,simonw,2020-10-19T18:35:25Z,2020-10-19T18:35:25Z,OWNER,"Since I'm dealing with tables of data I usually have a whole column of examples, so heuristics that check for numbers-greater-than-12 could actually work well for many cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712364532,https://api.github.com/repos/simonw/datasette/issues/1032,712364532,MDEyOklzc3VlQ29tbWVudDcxMjM2NDUzMg==,9599,simonw,2020-10-19T18:34:10Z,2020-10-19T18:34:10Z,OWNER,Lots of great example date data in https://biglocal.datasettes.com/COVID_WARN_Notices,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712364317,https://api.github.com/repos/simonw/datasette/issues/1032,712364317,MDEyOklzc3VlQ29tbWVudDcxMjM2NDMxNw==,9599,simonw,2020-10-19T18:33:42Z,2020-10-19T18:33:42Z,OWNER,"Related challenge: timezones. I think I'll punt on those for the moment and just concentrate on dates, but I should keep them in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712363825,https://api.github.com/repos/simonw/datasette/issues/1032,712363825,MDEyOklzc3VlQ29tbWVudDcxMjM2MzgyNQ==,9599,simonw,2020-10-19T18:32:43Z,2020-10-19T18:32:43Z,OWNER,"Horrible thought: I bet there is data out there that uses more than one date format in the same table! So this needs to be exposed as a visible per-column setting.
Column action menu can help here, although it's not yet the full solution because it isn't yet visible on mobile.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712363344,https://api.github.com/repos/simonw/datasette/issues/1032,712363344,MDEyOklzc3VlQ29tbWVudDcxMjM2MzM0NA==,9599,simonw,2020-10-19T18:31:46Z,2020-10-19T18:31:46Z,OWNER,"As always with dates, the challenge is America. `mm/dd/yy` is indistinguishable from the more sensible `dd/mm/yy`. It's crucially important to visibly tell people which format you assumed, otherwise all kinds of weird invisible bugs can manifest.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/904#issuecomment-712355877,https://api.github.com/repos/simonw/datasette/issues/904,712355877,MDEyOklzc3VlQ29tbWVudDcxMjM1NTg3Nw==,9599,simonw,2020-10-19T18:17:22Z,2020-10-19T18:17:22Z,OWNER,I quite like `datasette.urls.table()` but I'm not sure why.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712355706,https://api.github.com/repos/simonw/datasette/issues/904,712355706,MDEyOklzc3VlQ29tbWVudDcxMjM1NTcwNg==,9599,simonw,2020-10-19T18:17:03Z,2020-10-19T18:17:03Z,OWNER,"Options:
- `datasette.urls.instance()` (or `datasette.urls.root()`), `datasette.urls.table()` etc
- `datasette.url_for.instance()`...
- `datasette.url.instance()`...
- `datasette.root_url()`, `datasette.table_url()`...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712354600,https://api.github.com/repos/simonw/datasette/issues/904,712354600,MDEyOklzc3VlQ29tbWVudDcxMjM1NDYwMA==,9599,simonw,2020-10-19T18:15:03Z,2020-10-19T18:15:39Z,OWNER,"Related: #1026 (How should datasette.client interact with base_url)
Also this comment from https://github.com/simonw/datasette/issues/943#issuecomment-675752436
> One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted.
>
> If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712324077,https://api.github.com/repos/simonw/datasette/issues/904,712324077,MDEyOklzc3VlQ29tbWVudDcxMjMyNDA3Nw==,9599,simonw,2020-10-19T17:39:38Z,2020-10-19T17:39:38Z,OWNER,"If I do these methods I think this should be available on the `datasette` object too, as an internal API for plugins to use to construct redirects and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-710487083,https://api.github.com/repos/simonw/datasette/issues/904,710487083,MDEyOklzc3VlQ29tbWVudDcxMDQ4NzA4Mw==,9599,simonw,2020-10-16T19:34:10Z,2020-10-19T17:39:04Z,OWNER,"Alternatively, I could expose a single object that knows how to construct all kinds of URLs. Something like this:
```
{{ urls.instance() }}
{{ urls.database(database_name) }}
{{ urls.table(database_name, table_name) }}
etc
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/1027#issuecomment-712320103,https://api.github.com/repos/simonw/datasette/issues/1027,712320103,MDEyOklzc3VlQ29tbWVudDcxMjMyMDEwMw==,9599,simonw,2020-10-19T17:35:04Z,2020-10-19T17:35:04Z,OWNER,Still need to configure proxying though. https://www.netnea.com/cms/apache-tutorial-9_setting-up-a-reverse-proxy/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/991#issuecomment-712317638,https://api.github.com/repos/simonw/datasette/issues/991,712317638,MDEyOklzc3VlQ29tbWVudDcxMjMxNzYzOA==,9599,simonw,2020-10-19T17:30:56Z,2020-10-19T17:30:56Z,OWNER,https://biglocal.datasettes.com/ is one of my larger Datasettes in terms of number of databases.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-712266834,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,712266834,MDEyOklzc3VlQ29tbWVudDcxMjI2NjgzNA==,9599,simonw,2020-10-19T16:01:23Z,2020-10-19T16:01:23Z,MEMBER,Might just be a documented pattern for how to configure this in YAML templates.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/simonw/datasette/pull/1030#issuecomment-711407607,https://api.github.com/repos/simonw/datasette/issues/1030,711407607,MDEyOklzc3VlQ29tbWVudDcxMTQwNzYwNw==,22429695,codecov[bot],2020-10-18T19:31:31Z,2020-10-19T08:01:51Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=h1) Report
> Merging [#1030](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/568bd7bbf590861687db8c318f3d8cfcd1dfb47a?el=desc) will **decrease** coverage by `0.10%`.
> The diff coverage is `68.75%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1030/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1030 +/- ##
==========================================
- Coverage 84.63% 84.53% -0.11%
==========================================
Files 28 28
Lines 3892 3905 +13
==========================================
+ Hits 3294 3301 +7
- Misses 598 604 +6
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1030/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.35% <68.75%> (-0.79%)` | :arrow_down: |
| [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1030/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=footer). Last update [568bd7b...e082533](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723982480,Make `package` command deal with a configuration directory argument,
https://github.com/simonw/datasette/pull/1031#issuecomment-711792622,https://api.github.com/repos/simonw/datasette/issues/1031,711792622,MDEyOklzc3VlQ29tbWVudDcxMTc5MjYyMg==,22429695,codecov[bot],2020-10-19T07:57:17Z,2020-10-19T07:57:17Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=h1) Report
> Merging [#1031](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/568bd7bbf590861687db8c318f3d8cfcd1dfb47a?el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1031/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #1031 +/- ##
==========================================
- Coverage 84.63% 84.60% -0.03%
==========================================
Files 28 28
Lines 3892 3892
==========================================
- Hits 3294 3293 -1
- Misses 598 599 +1
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1031/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `74.22% <ø> (ø)` | |
| [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1031/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=footer). Last update [568bd7b...7e7eaa4](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-711569063,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,711569063,MDEyOklzc3VlQ29tbWVudDcxMTU2OTA2Mw==,9599,simonw,2020-10-19T05:01:29Z,2020-10-19T05:01:29Z,MEMBER,"Demo of `--accept`:
github-to-sqlite get /repos/simonw/datasette/readme --accept 'application/vnd.github.VERSION.html'
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/dogsheep-beta/issues/28#issuecomment-711089647,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28,711089647,MDEyOklzc3VlQ29tbWVudDcxMTA4OTY0Nw==,9599,simonw,2020-10-17T22:43:13Z,2020-10-17T22:43:13Z,MEMBER,"Since my personal Dogsheep uses Datasette authentication, I'm going to need to pass through cookies. https://github.com/simonw/datasette/issues/1020 will solve that in the future but for now I need to solve it explicitly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723861683,Switch to using datasette.client,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711083698,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711083698,MDEyOklzc3VlQ29tbWVudDcxMTA4MzY5OA==,572,jarib,2020-10-17T21:39:15Z,2020-10-17T21:39:15Z,NONE,Nice! Works perfectly. Thanks for the quick response and great tooling in general.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711081703,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711081703,MDEyOklzc3VlQ29tbWVudDcxMTA4MTcwMw==,9599,simonw,2020-10-17T21:18:35Z,2020-10-17T21:18:35Z,MEMBER,"OK, if you upgrade to the just-released 1.0 this should work (it worked against my Spanish export).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings,
https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079760,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711079760,MDEyOklzc3VlQ29tbWVudDcxMTA3OTc2MA==,9599,simonw,2020-10-17T21:00:05Z,2020-10-17T21:00:05Z,MEMBER,Checking for either `
""
TEMPLATE = """"""
{}
"""""".strip()
EN_MEDIA_SCRIPT = """"""
Array.from(document.querySelectorAll('en-media')).forEach(el => {
let hash = el.getAttribute('hash');
let type = el.getAttribute('type');
let path = `/evernote/resources_data/${hash}.json?_shape=array`;
fetch(path).then(r => r.json()).then(rows => {
let b64 = rows[0].data.encoded;
let data = `data:${type};base64,${b64}`;
el.innerHTML = ``;
});
});
""""""
@hookimpl
def render_cell(value, table):
if not table:
# Don't render content from arbitrary SQL queries, could be XSS hole
return
if not value or not isinstance(value, str):
return
value = value.strip()
if value.startswith(START) and value.endswith(END):
trimmed = value[len(START) : -len(END)]
trimmed = trimmed.split("">"", 1)[1]
# Replace those horrible double newlines
trimmed = trimmed.replace(""
"", "" "")
return jinja2.Markup(TEMPLATE.format(trimmed))
@hookimpl
def extra_body_script():
return EN_MEDIA_SCRIPT
```
It works!
It does however demonstrate that Evernote's ""clip this webpage"" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from tags inline in Datasette,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710461468,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710461468,MDEyOklzc3VlQ29tbWVudDcxMDQ2MTQ2OA==,9599,simonw,2020-10-16T19:18:19Z,2020-10-16T19:18:19Z,OWNER,"Reconsidering: #89 was a feature request that relates to this, so maybe this is worth implementing after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242,https://api.github.com/repos/simonw/sqlite-utils/issues/89,710460242,MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==,9599,simonw,2020-10-16T19:17:27Z,2020-10-16T19:17:50Z,OWNER,"I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works:
```python
fresh_db.table(""tree"", extracts=[Extract(
columns=(""CommonName"", ""LatinName""),
table=""Species"",
fk_column=""species_id"",
rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature,
https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710456981,https://api.github.com/repos/simonw/sqlite-utils/issues/187,710456981,MDEyOklzc3VlQ29tbWVudDcxMDQ1Njk4MQ==,9599,simonw,2020-10-16T19:15:13Z,2020-10-16T19:15:13Z,OWNER,This is a duplicate of #79.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723460107,Maybe: Utility method / CLI tool for initializing SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710440853,https://api.github.com/repos/simonw/sqlite-utils/issues/187,710440853,MDEyOklzc3VlQ29tbWVudDcxMDQ0MDg1Mw==,9599,simonw,2020-10-16T19:04:19Z,2020-10-16T19:04:19Z,OWNER,I split this off from #136.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723460107,Maybe: Utility method / CLI tool for initializing SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/136#issuecomment-710428802,https://api.github.com/repos/simonw/sqlite-utils/issues/136,710428802,MDEyOklzc3VlQ29tbWVudDcxMDQyODgwMg==,9599,simonw,2020-10-16T18:56:47Z,2020-10-16T18:56:47Z,OWNER,"To keep the code cleaner, I'm tempted to support this instead:
--load-extension=spatialite
Where `spatialite` is a special shortcut value that triggers a search for that module in known locations.
Users could still load a module in a file called `spatialite` in the current directory using:
--load-extension=./spatialite
In fact, `--load-extension=spatialite` could handle that case too by always checking for a file called `spatialite` before attempting to search for it in known locations.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642,--load-extension=spatialite shortcut option,
https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710405658,MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==,9599,simonw,2020-10-16T18:42:48Z,2020-10-16T18:42:48Z,OWNER,"Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d
I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142,Feature request: enable extensions loading,
https://github.com/simonw/sqlite-utils/issues/48#issuecomment-710402331,https://api.github.com/repos/simonw/sqlite-utils/issues/48,710402331,MDEyOklzc3VlQ29tbWVudDcxMDQwMjMzMQ==,9599,simonw,2020-10-16T18:41:06Z,2020-10-16T18:41:06Z,OWNER,I could use this demo from JupyterCon 2020 https://gist.github.com/simonw/656c21b5800d5e4624dec9930f00e093,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder",
https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593,https://api.github.com/repos/simonw/sqlite-utils/issues/58,710399593,MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==,9599,simonw,2020-10-16T18:39:31Z,2020-10-16T18:39:31Z,OWNER,I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488293926,Support enabling FTS on views,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710397574,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710397574,MDEyOklzc3VlQ29tbWVudDcxMDM5NzU3NA==,9599,simonw,2020-10-16T18:38:21Z,2020-10-16T18:38:21Z,OWNER,"I'm not going to implement this. I'll leave `extract=...` as it is right now, suitable for quick simple single-column operations on input, but if users want to do something more complicated involving multiple columns they should use the `table.extract()` method after the initial insert instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710395444,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710395444,MDEyOklzc3VlQ29tbWVudDcxMDM5NTQ0NA==,9599,simonw,2020-10-16T18:37:10Z,2020-10-16T18:37:10Z,OWNER,"But this begins to feel too complicated, given that `table.extract()` can already be used to achieve the same thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710393550,MDEyOklzc3VlQ29tbWVudDcxMDM5MzU1MA==,9599,simonw,2020-10-16T18:35:57Z,2020-10-16T18:36:39Z,OWNER,"If I want to support that most complicated example, I think the option to pass a `Extracts()` object to `extracts=` is the best way to do it:
```python
fresh_db.table(""tree"", extracts=[Extract(
columns=(""CommonName"", ""LatinName""),
table=""Species"",
fk_column=""species_id"",
rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710390915,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710390915,MDEyOklzc3VlQ29tbWVudDcxMDM5MDkxNQ==,9599,simonw,2020-10-16T18:34:26Z,2020-10-16T18:34:50Z,OWNER,"Here's the most complex example of `.extracts()`:
```python
db[""Trees""].extract(
[""CommonName"", ""LatinName""],
table=""Species"",
fk_column=""species_id"",
rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)
```
Resulting in:
```sql
CREATE TABLE [Species] (
[id] INTEGER PRIMARY KEY,
[name] TEXT,
[latin] TEXT
)
```
From https://sqlite-utils.readthedocs.io/en/stable/python-api.html#extracting-columns-into-a-separate-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710364942,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710364942,MDEyOklzc3VlQ29tbWVudDcxMDM2NDk0Mg==,9599,simonw,2020-10-16T18:18:48Z,2020-10-16T18:18:48Z,OWNER,"I think there is. It's a nice existing feature, and I don't think adding tuple support to it would be a huge lift.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710363789,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710363789,MDEyOklzc3VlQ29tbWVudDcxMDM2Mzc4OQ==,9599,simonw,2020-10-16T18:18:05Z,2020-10-16T18:18:05Z,OWNER,I wonder if there's value in extending the `extracts=` option at all given the existence of `table.extract()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710359724,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710359724,MDEyOklzc3VlQ29tbWVudDcxMDM1OTcyNA==,9599,simonw,2020-10-16T18:15:31Z,2020-10-16T18:15:31Z,OWNER,"Using a tuple would work:
```python
fresh_db.table(""tree"", extracts=[(""common_name"", ""latin_name"")])
```
Or to define a custom name:
```python
fresh_db.table(""tree"", extracts={(""common_name"", ""latin_name""): ""names""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710346830,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710346830,MDEyOklzc3VlQ29tbWVudDcxMDM0NjgzMA==,9599,simonw,2020-10-16T18:08:52Z,2020-10-16T18:09:21Z,OWNER,"The new `.extract()` method can handle multiple columns:
https://github.com/simonw/sqlite-utils/blob/2c541fac352632e23e40b0d21e3f233f7a744a57/tests/test_extract.py#L70-L87","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710258736,https://api.github.com/repos/simonw/sqlite-utils/issues/182,710258736,MDEyOklzc3VlQ29tbWVudDcxMDI1ODczNg==,9599,simonw,2020-10-16T17:20:41Z,2020-10-16T17:20:41Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-710198162,https://api.github.com/repos/simonw/sqlite-utils/issues/186,710198162,MDEyOklzc3VlQ29tbWVudDcxMDE5ODE2Mg==,9599,simonw,2020-10-16T16:41:00Z,2020-10-16T16:41:00Z,OWNER,"Failing test:
```python
def test_extract_null_values(fresh_db):
fresh_db[""species""].insert({""id"": 1, ""species"": ""Wolf""}, pk=""id"")
fresh_db[""individuals""].insert_all(
[
{""id"": 10, ""name"": ""Terriana"", ""species"": ""Fox""},
{""id"": 11, ""name"": ""Spenidorm"", ""species"": None},
{""id"": 12, ""name"": ""Grantheim"", ""species"": ""Wolf""},
{""id"": 13, ""name"": ""Turnutopia"", ""species"": None},
{""id"": 14, ""name"": ""Wargal"", ""species"": ""Wolf""},
],
pk=""id"",
)
fresh_db[""individuals""].extract(""species"")
assert fresh_db[""species""].schema == (
""CREATE TABLE [species] (\n""
"" [id] INTEGER PRIMARY KEY,\n""
"" [species] TEXT\n""
"")""
)
assert fresh_db[""individuals""].schema == (
'CREATE TABLE ""individuals"" (\n'
"" [id] INTEGER PRIMARY KEY,\n""
"" [name] TEXT,\n""
"" [species_id] INTEGER,\n""
"" FOREIGN KEY(species_id) REFERENCES species(id)\n""
"")""
)
assert list(fresh_db[""species""].rows) == [
{""id"": 1, ""species"": ""Wolf""},
{""id"": 2, ""species"": ""Fox""},
]
assert list(fresh_db[""individuals""].rows) == [
{""id"": 10, ""name"": ""Terriana"", ""species_id"": 2},
{""id"": 11, ""name"": ""Spenidorm"", ""species_id"": None},
{""id"": 12, ""name"": ""Grantheim"", ""species_id"": 1},
{""id"": 13, ""name"": ""Turnutopia"", ""species_id"": None},
{""id"": 14, ""name"": ""Wargal"", ""species_id"": 1},
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710178871,https://api.github.com/repos/simonw/sqlite-utils/issues/182,710178871,MDEyOklzc3VlQ29tbWVudDcxMDE3ODg3MQ==,9599,simonw,2020-10-16T16:27:39Z,2020-10-16T16:28:14Z,OWNER,"The file is opened for me by `click.File()`, which also handles things like `-` for stdin. But i neee to be able to switch the encoding used to read from that based on the `--encoding` option.
I think the way to do that is to open the file in binary mode and then wrap it in a codec reader:
```python
fp = codecs.getreader(encoding)(fp)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706260,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709706260,MDEyOklzc3VlQ29tbWVudDcwOTcwNjI2MA==,9599,simonw,2020-10-16T03:17:02Z,2020-10-16T03:17:17Z,OWNER,Actually I think this should be an option to `.extract()` which controls if nulls are extracted or left alone. Maybe called `extract_nulls=True/False`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706065,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709706065,MDEyOklzc3VlQ29tbWVudDcwOTcwNjA2NQ==,9599,simonw,2020-10-16T03:16:22Z,2020-10-16T03:16:22Z,OWNER,Either way I think I'm going to need to add some SQL which uses `where a = b or (a is null and b is null)`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705885,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709705885,MDEyOklzc3VlQ29tbWVudDcwOTcwNTg4NQ==,9599,simonw,2020-10-16T03:15:39Z,2020-10-16T03:15:39Z,OWNER,The alternative solution here would be that a single `null` value DOES get extracted. To implement this I would need to add some logic that uses `is null` instead of `=`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705624,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709705624,MDEyOklzc3VlQ29tbWVudDcwOTcwNTYyNA==,9599,simonw,2020-10-16T03:14:39Z,2020-10-16T03:14:39Z,OWNER,"How should this work with extractions covering multiple columns?
If there's a single column then it makes sense that a `null` value would not be extracted into the lookup table, but would instead become stay as `null`.
For a multiple column extraction, provided at least one of those columns is not null It should map to a record in the lookup table. Only if ALL of the extracted columns are null should the lookup value stay null.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/datasette/issues/1027#issuecomment-709647525,https://api.github.com/repos/simonw/datasette/issues/1027,709647525,MDEyOklzc3VlQ29tbWVudDcwOTY0NzUyNQ==,9599,simonw,2020-10-15T23:49:51Z,2020-10-15T23:51:39Z,OWNER,"I'll install Apache on macOS to figure this out using https://formulae.brew.sh/formula/httpd
`brew install httpd` output this at the end:
```
==> httpd
DocumentRoot is /usr/local/var/www.
The default ports have been set in /usr/local/etc/httpd/httpd.conf to 8080 and in
/usr/local/etc/httpd/extra/httpd-ssl.conf to 8443 so that httpd can run without sudo.
To have launchd start httpd now and restart at login:
brew services start httpd
Or, if you don't want/need a background service you can just run:
apachectl start
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1027#issuecomment-709646865,https://api.github.com/repos/simonw/datasette/issues/1027,709646865,MDEyOklzc3VlQ29tbWVudDcwOTY0Njg2NQ==,9599,simonw,2020-10-15T23:47:08Z,2020-10-15T23:47:08Z,OWNER,It should cover both nginx and Apache. nginx config is here: https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-709636372,https://api.github.com/repos/simonw/datasette/issues/1026,709636372,MDEyOklzc3VlQ29tbWVudDcwOTYzNjM3Mg==,9599,simonw,2020-10-15T23:09:34Z,2020-10-15T23:09:34Z,OWNER,"I'm inclined to say that internal requests should ignore `base_url` - since that seems like the right thing for plugins that need to access default Datasette APIs.
The one catch here is plugins that might want to proxy the current incoming URL for some reason - where that incoming `request.path` could include the `base_url`.
Actually those should be fine - because it will have been stripped off earlier:
https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/app.py#L963-L968","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/904#issuecomment-709635276,https://api.github.com/repos/simonw/datasette/issues/904,709635276,MDEyOklzc3VlQ29tbWVudDcwOTYzNTI3Ng==,9599,simonw,2020-10-15T23:05:54Z,2020-10-15T23:05:54Z,OWNER,Could have `instance_url()` take an optional path argument which is then turned into the correct path.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709635021,https://api.github.com/repos/simonw/datasette/issues/904,709635021,MDEyOklzc3VlQ29tbWVudDcwOTYzNTAyMQ==,9599,simonw,2020-10-15T23:05:11Z,2020-10-15T23:05:11Z,OWNER,"I think this should be a family of functions:
- `instance_url()` - the root URL of the instance (usually `/` unless `base_url` is set)
- `database_url(database_name)` - already got this
- `table_url(database_name, table_name)`
- `row_url(database_name, table_name, row)` - not sure about this one. The idea would be for `row` to be correctly turned into a URL by introspecting the primary keys for that table, then pulling those values out of the SQLite `row` object. Might not be necessary though.
I also need a way for plugins to link to e.g. `/-/configure-fts` - or even `/-/configure-fts/database-name/table-name`. What should that look like?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709634261,https://api.github.com/repos/simonw/datasette/issues/904,709634261,MDEyOklzc3VlQ29tbWVudDcwOTYzNDI2MQ==,9599,simonw,2020-10-15T23:02:43Z,2020-10-15T23:02:43Z,OWNER,"Here's the current implementation of `database_url` - on the `BaseView` class, but only because it needs access to a `datasette` instance (to read `base_url`):
https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/datasette/views/base.py#L102-L108","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709633823,https://api.github.com/repos/simonw/datasette/issues/904,709633823,MDEyOklzc3VlQ29tbWVudDcwOTYzMzgyMw==,9599,simonw,2020-10-15T23:01:13Z,2020-10-15T23:01:13Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/988#issuecomment-709633762,https://api.github.com/repos/simonw/datasette/issues/988,709633762,MDEyOklzc3VlQ29tbWVudDcwOTYzMzc2Mg==,9599,simonw,2020-10-15T23:01:01Z,2020-10-15T23:01:01Z,OWNER,This is a dupe of https://github.com/simonw/datasette/issues/904,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",713209404,Mechanism for plugins to construct URLs that respect base_url,
https://github.com/simonw/datasette/issues/865#issuecomment-709633080,https://api.github.com/repos/simonw/datasette/issues/865,709633080,MDEyOklzc3VlQ29tbWVudDcwOTYzMzA4MA==,9599,simonw,2020-10-15T22:58:51Z,2020-10-15T22:58:51Z,OWNER,"It looks like there are places where Datasette might return a redirect that doesn't take `base_url` into account - I'm planning on fixing those here, after which I think `ProxyPassReverse` should no longer be necessary. https://github.com/simonw/datasette/issues/1025#issuecomment-709632136","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/900#issuecomment-709632765,https://api.github.com/repos/simonw/datasette/issues/900,709632765,MDEyOklzc3VlQ29tbWVudDcwOTYzMjc2NQ==,9599,simonw,2020-10-15T22:57:55Z,2020-10-15T22:57:55Z,OWNER,"I believe this particular bug has been fixed, based on my testing here: https://github.com/simonw/datasette/issues/1024#issuecomment-709622973
Please re-open the ticket if you are still experiencing it.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url,
https://github.com/simonw/datasette/issues/1025#issuecomment-709632314,https://api.github.com/repos/simonw/datasette/issues/1025,709632314,MDEyOklzc3VlQ29tbWVudDcwOTYzMjMxNA==,9599,simonw,2020-10-15T22:56:25Z,2020-10-15T22:56:34Z,OWNER,"That `utils/asgi.py` line is the default path for setting cookies. That should likely take `base_url` into account too:
https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/utils/asgi.py#L331-L342","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-709632136,https://api.github.com/repos/simonw/datasette/issues/1025,709632136,MDEyOklzc3VlQ29tbWVudDcwOTYzMjEzNg==,9599,simonw,2020-10-15T22:55:44Z,2020-10-15T22:55:44Z,OWNER,"It looks like there are also some generated redirect responses that don't take `base_url` into account:
```
datasette % git grep '""/' -- '*.py' ':(exclude)*test_*.py' ':(exclude)datasette/app.py'
datasette/_version.py: for i in cfg.versionfile_source.split(""/""):
datasette/utils/asgi.py: path=""/"",
datasette/views/base.py: should_redirect = ""/{}-{}"".format(name, expected)
datasette/views/base.py: should_redirect += ""/"" + urllib.parse.quote_plus(kwargs[""table""])
datasette/views/base.py: should_redirect += ""/"" + kwargs[""pk_path""]
datasette/views/special.py: response = Response.redirect(""/"")
datasette/views/special.py: return Response.redirect(""/"")
datasette/views/special.py: response = Response.redirect(""/"")
datasette/views/special.py: return Response.redirect(""/"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-709629920,https://api.github.com/repos/simonw/datasette/issues/1025,709629920,MDEyOklzc3VlQ29tbWVudDcwOTYyOTkyMA==,9599,simonw,2020-10-15T22:48:20Z,2020-10-15T22:48:20Z,OWNER,"Also these:
```
datasette % git grep '""/' -- '*.html' ':(exclude)*/patterns.html'
datasette/templates/allow_debug.html:
-
{% for table in database.tables_and_views_truncated %}{{ table.name }}{% if table.private %} 🔒{% endif %}{% if not loop.last %}, {% endif %}{% endfor %}{% if database.tables_and_views_more %}, ...{% endif %}
+ {% for table in database.tables_and_views_truncated %}
+ {% if show_hidden or not table.hidden %}
+
+
{{ table.name }}{% if table.private %} 🔒{% endif %}{% if table.hidden %} (hidden){% endif %}
+
{% for column in table.columns[:9] %}{{ column }}{% if not loop.last %}, {% endif %}{% endfor %}{% if table.columns|length > 9 %}...{% endif %}
+
{% if table.count is none %}Many rows{% else %}{{ ""{:,}"".format(table.count) }} row{% if table.count == 1 %}{% else %}s{% endif %}{% endif %}
+ {% endif %}
+
{% endfor %}
{% endblock %}
```
![image](https://user-images.githubusercontent.com/9599/95398019-9c594480-08b9-11eb-90c9-f322d647ee3b.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/996#issuecomment-705119801,https://api.github.com/repos/simonw/datasette/issues/996,705119801,MDEyOklzc3VlQ29tbWVudDcwNTExOTgwMQ==,9599,simonw,2020-10-07T18:34:58Z,2020-10-07T18:34:58Z,OWNER,One option: compare the length of the matches and pick the shortest one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716756082,Better handling of multiple matching template wildcard paths,
https://github.com/simonw/datasette/issues/996#issuecomment-705116123,https://api.github.com/repos/simonw/datasette/issues/996,705116123,MDEyOklzc3VlQ29tbWVudDcwNTExNjEyMw==,9599,simonw,2020-10-07T18:27:57Z,2020-10-07T18:28:36Z,OWNER,"This line was meant to handle this case, by only matching non-/ characters. Looks like it did not have the desired effect: https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1204
Maybe because the finished regex doesn't include `^` and `$` anchors?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716756082,Better handling of multiple matching template wildcard paths,
https://github.com/simonw/datasette/issues/996#issuecomment-705115185,https://api.github.com/repos/simonw/datasette/issues/996,705115185,MDEyOklzc3VlQ29tbWVudDcwNTExNTE4NQ==,9599,simonw,2020-10-07T18:26:11Z,2020-10-07T18:26:45Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1030-L1037
https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L951-L954
https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1197-L1207","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716756082,Better handling of multiple matching template wildcard paths,
https://github.com/dogsheep/github-to-sqlite/pull/48#issuecomment-704553385,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48,704553385,MDEyOklzc3VlQ29tbWVudDcwNDU1MzM4NQ==,9599,simonw,2020-10-06T21:07:44Z,2020-10-06T21:07:44Z,MEMBER,"Sorry for not looking at this sooner, trying it out now - pull request looks great!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681228542,Add pull requests,
https://github.com/dogsheep/github-to-sqlite/pull/48#issuecomment-704503719,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48,704503719,MDEyOklzc3VlQ29tbWVudDcwNDUwMzcxOQ==,755825,adamjonas,2020-10-06T19:26:59Z,2020-10-06T19:26:59Z,CONTRIBUTOR,ref #46 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681228542,Add pull requests,
https://github.com/simonw/datasette/pull/995#issuecomment-704395913,https://api.github.com/repos/simonw/datasette/issues/995,704395913,MDEyOklzc3VlQ29tbWVudDcwNDM5NTkxMw==,9599,simonw,2020-10-06T16:25:20Z,2020-10-06T16:25:20Z,OWNER,"This is great, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",715779909,Document setting Google Cloud SDK properties,
https://github.com/simonw/datasette/pull/995#issuecomment-704347565,https://api.github.com/repos/simonw/datasette/issues/995,704347565,MDEyOklzc3VlQ29tbWVudDcwNDM0NzU2NQ==,22429695,codecov[bot],2020-10-06T15:22:55Z,2020-10-06T15:22:55Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=h1) Report
> Merging [#995](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/5a184a5d211d3226e0417ee5cf8476cd887cd35e?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/995/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #995 +/- ##
=======================================
Coverage 84.34% 84.34%
=======================================
Files 28 28
Lines 3865 3865
=======================================
Hits 3260 3260
Misses 605 605
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=footer). Last update [5a184a5...aed2cf9](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",715779909,Document setting Google Cloud SDK properties,
https://github.com/simonw/datasette/issues/992#issuecomment-704013305,https://api.github.com/repos/simonw/datasette/issues/992,704013305,MDEyOklzc3VlQ29tbWVudDcwNDAxMzMwNQ==,9599,simonw,2020-10-06T04:00:00Z,2020-10-06T04:00:00Z,OWNER,I can add this well in advance of Datasette 1.0 as an alias to the existing `--config` option (but taking two arguments instead of one).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/993#issuecomment-703963211,https://api.github.com/repos/simonw/datasette/issues/993,703963211,MDEyOklzc3VlQ29tbWVudDcwMzk2MzIxMQ==,9599,simonw,2020-10-06T00:40:48Z,2020-10-06T00:40:48Z,OWNER,"Now live at https://latest.datasette.io/fixtures/facetable
New `await db.table_column_details(table)` method is documented here: https://docs.datasette.io/en/latest/internals.html#database-introspection","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",715072935,Column action menu should show column type,
https://github.com/simonw/datasette/issues/993#issuecomment-703928029,https://api.github.com/repos/simonw/datasette/issues/993,703928029,MDEyOklzc3VlQ29tbWVudDcwMzkyODAyOQ==,9599,simonw,2020-10-05T22:42:45Z,2020-10-05T22:42:59Z,OWNER,"
The `NOT NULL` text shows up only for columns that are not null.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",715072935,Column action menu should show column type,
https://github.com/simonw/datasette/issues/992#issuecomment-703351641,https://api.github.com/repos/simonw/datasette/issues/992,703351641,MDEyOklzc3VlQ29tbWVudDcwMzM1MTY0MQ==,9599,simonw,2020-10-05T01:31:36Z,2020-10-05T01:31:36Z,OWNER,If I rename metadata to config in #493 then this should be `--setting` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/741#issuecomment-703350401,https://api.github.com/repos/simonw/datasette/issues/741,703350401,MDEyOklzc3VlQ29tbWVudDcwMzM1MDQwMQ==,9599,simonw,2020-10-05T01:26:12Z,2020-10-05T01:26:12Z,OWNER,Relevant support question on Twitter: https://twitter.com/simonw/status/1312926103627988993,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/778#issuecomment-702493047,https://api.github.com/repos/simonw/datasette/issues/778,702493047,MDEyOklzc3VlQ29tbWVudDcwMjQ5MzA0Nw==,9599,simonw,2020-10-02T02:26:25Z,2020-10-02T02:26:25Z,OWNER,"I think this could work for arbitrary SQL queries too. Those would need querystring configuration that specifies which sorted column(s) should be used for the ""next"" cursor.
One example: I'd like to be able to offer a paginated list of counts of values in a table - e.g. this query:
https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=select+replies%2C+count%28*%29+from+%5Btwitter-ratio%2Fsenators%5D+group+by+replies+order+by+count%28*%29+desc%3B
That could even become a query that gets linked to from the column actions menu.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626211658,Ability to configure keyset pagination for views and queries,
https://github.com/simonw/datasette/issues/565#issuecomment-702459747,https://api.github.com/repos/simonw/datasette/issues/565,702459747,MDEyOklzc3VlQ29tbWVudDcwMjQ1OTc0Nw==,9599,simonw,2020-10-02T00:09:55Z,2020-10-02T00:09:55Z,OWNER,"Today Datasette uses `""click~=7.1.1""` and Uvicorn uses `""click==7.*""` so I think we are OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473307794,Conflict between datasette and uvicorn click versions,
https://github.com/simonw/datasette/issues/446#issuecomment-702459432,https://api.github.com/repos/simonw/datasette/issues/446,702459432,MDEyOklzc3VlQ29tbWVudDcwMjQ1OTQzMg==,9599,simonw,2020-10-02T00:08:47Z,2020-10-02T00:08:47Z,OWNER,"I've decided not to bother with this, especially having redesigned some hooks to return Datasette standard `Response` objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440134714,Define mechanism for plugins to return structured data,
https://github.com/simonw/datasette/issues/865#issuecomment-702418045,https://api.github.com/repos/simonw/datasette/issues/865,702418045,MDEyOklzc3VlQ29tbWVudDcwMjQxODA0NQ==,9599,simonw,2020-10-01T21:54:00Z,2020-10-01T21:54:00Z,OWNER,"Had a thought: this is likely to break in plugins too, such as `datasette-edit-schema` which constructs URLs for redirects e.g. here: https://github.com/simonw/datasette-edit-schema/blob/dbd0abee6dd3385b114cfe9671f7ead1c4855b60/datasette_edit_schema/__init__.py#L46-L48","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/pull/986#issuecomment-702265255,https://api.github.com/repos/simonw/datasette/issues/986,702265255,MDEyOklzc3VlQ29tbWVudDcwMjI2NTI1NQ==,9599,simonw,2020-10-01T16:51:45Z,2020-10-01T16:51:45Z,OWNER,Thanks for taking a look! The fix ended up being a little different from this because I still want to disable faceting on regular single primary keys (since faceting by those won't ever produce interesting results) - here's what I used: https://github.com/simonw/datasette/commit/5d6bc4c268f9f155e59561671f8617addd3e91bc,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712889459,"Allow facet by primary keys, fixes #985",
https://github.com/simonw/datasette/issues/987#issuecomment-702241397,https://api.github.com/repos/simonw/datasette/issues/987,702241397,MDEyOklzc3VlQ29tbWVudDcwMjI0MTM5Nw==,9599,simonw,2020-10-01T16:11:31Z,2020-10-01T16:11:31Z,OWNER,"This should be a new page in the documentation, which will be expanded later on to cover things like the proposed JavaScript plugin hooks system from #983.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/987#issuecomment-702240916,https://api.github.com/repos/simonw/datasette/issues/987,702240916,MDEyOklzc3VlQ29tbWVudDcwMjI0MDkxNg==,9599,simonw,2020-10-01T16:10:44Z,2020-10-01T16:10:44Z,OWNER,https://github.com/simonw/datasette/blob/141544613f9e76ddb74eee38d6f8ee1e0e70f833/datasette/templates/_table.html#L6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/pull/986#issuecomment-702171636,https://api.github.com/repos/simonw/datasette/issues/986,702171636,MDEyOklzc3VlQ29tbWVudDcwMjE3MTYzNg==,22429695,codecov[bot],2020-10-01T14:24:11Z,2020-10-01T14:24:11Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=h1) Report
> Merging [#986](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/141544613f9e76ddb74eee38d6f8ee1e0e70f833?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/986/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #986 +/- ##
=======================================
Coverage 84.28% 84.28%
=======================================
Files 28 28
Lines 3850 3850
=======================================
Hits 3245 3245
Misses 605 605
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=footer). Last update [1415446...76f7094](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712889459,"Allow facet by primary keys, fixes #985",
https://github.com/simonw/datasette/issues/985#issuecomment-702132019,https://api.github.com/repos/simonw/datasette/issues/985,702132019,MDEyOklzc3VlQ29tbWVudDcwMjEzMjAxOQ==,9599,simonw,2020-10-01T13:23:48Z,2020-10-01T13:23:48Z,OWNER,"The code that skips the ""facet by this"" menu option for primary key columns could take into account if there are multiple primary keys.
https://github.com/simonw/datasette/blob/141544613f9e76ddb74eee38d6f8ee1e0e70f833/datasette/static/table.js#L91-L98","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712839383,Column actions should support facet by compound primary keys,
https://github.com/simonw/datasette/issues/530#issuecomment-701720034,https://api.github.com/repos/simonw/datasette/issues/530,701720034,MDEyOklzc3VlQ29tbWVudDcwMTcyMDAzNA==,9599,simonw,2020-10-01T00:42:08Z,2020-10-01T00:42:08Z,OWNER,"I've changed my mind - especially since Datasette now ships with default JavaScript for the column menus in #981. I'm not going to extract CodeMirror into a plugin, I'll leave it where it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",460540321,Extract codemirror SQL editor out into a plugin,
https://github.com/simonw/datasette/issues/984#issuecomment-701708072,https://api.github.com/repos/simonw/datasette/issues/984,701708072,MDEyOklzc3VlQ29tbWVudDcwMTcwODA3Mg==,9599,simonw,2020-10-01T00:01:36Z,2020-10-01T00:01:36Z,OWNER,Column action menus are the cog icons on https://latest.datasette.io/fixtures/facetable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712368432,Review accessibility of new column action menus,
https://github.com/simonw/datasette/issues/981#issuecomment-701707116,https://api.github.com/repos/simonw/datasette/issues/981,701707116,MDEyOklzc3VlQ29tbWVudDcwMTcwNzExNg==,9599,simonw,2020-09-30T23:58:17Z,2020-09-30T23:58:17Z,OWNER,"Now live at https://latest.datasette.io/fixtures/facetable
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701704716,https://api.github.com/repos/simonw/datasette/issues/981,701704716,MDEyOklzc3VlQ29tbWVudDcwMTcwNDcxNg==,9599,simonw,2020-09-30T23:48:36Z,2020-09-30T23:48:36Z,OWNER,"Since this menu doesn't provide new functionality, I'm going to ignore the fact that it doesn't exist on portrait mobile view for the moment. Likewise, I'm going to skip making it accessible for the moment since lacking accessibility doesn't prevent functionality from being accessed - the menu-less experience currently works the same as the portrait mobile experience.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701703472,https://api.github.com/repos/simonw/datasette/issues/981,701703472,MDEyOklzc3VlQ29tbWVudDcwMTcwMzQ3Mg==,9599,simonw,2020-09-30T23:43:30Z,2020-09-30T23:43:30Z,OWNER,"I'm going to go based just on the visible values on the current page. I think that's good enough, and it avoids the complexity involved in doing a server-side check for blank values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701697918,https://api.github.com/repos/simonw/datasette/issues/981,701697918,MDEyOklzc3VlQ29tbWVudDcwMTY5NzkxOA==,9599,simonw,2020-09-30T23:24:14Z,2020-09-30T23:25:28Z,OWNER,"I could provide the ""Show non-blank values"" option only for columns where there are blank values visible on the current page.
This could be a bit confusing though, since the absence of that option could suggest that there are no blank values at all when that's actually not true.
One option: run a separate `fetch()` call that figures out if any of the columns contain blank values, which gets a bit of extra time to execute. Only show the ""Show non-blank values"" option in the menu once that has returned.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701679729,https://api.github.com/repos/simonw/datasette/issues/981,701679729,MDEyOklzc3VlQ29tbWVudDcwMTY3OTcyOQ==,9599,simonw,2020-09-30T22:26:33Z,2020-09-30T22:26:33Z,OWNER,"Bug: https://latest.datasette.io/fixtures/sortable?_sort=sortable
![sort-bug](https://user-images.githubusercontent.com/9599/94746304-4e7c9380-0331-11eb-8e81-7245529c4654.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/sqlite-utils/issues/183#issuecomment-701678659,https://api.github.com/repos/simonw/sqlite-utils/issues/183,701678659,MDEyOklzc3VlQ29tbWVudDcwMTY3ODY1OQ==,9599,simonw,2020-09-30T22:23:36Z,2020-09-30T22:23:36Z,OWNER,"It just ran and didn't find anything. I'll leave it enabled for a while but I may turn it off again, it could be that this kind of Python library isn't really what it's useful for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712316959,Try out GitHub code scanning,
https://github.com/simonw/datasette/issues/981#issuecomment-701668526,https://api.github.com/repos/simonw/datasette/issues/981,701668526,MDEyOklzc3VlQ29tbWVudDcwMTY2ODUyNg==,9599,simonw,2020-09-30T21:57:22Z,2020-09-30T21:57:22Z,OWNER,"A bunch of things to fix:
- It clobbers existing querystring parameters - it needs to leave these alone (but replace the current sort order)
- Facet option should not show up if you are already faceting by that column
- There's no way to close the menu once it has opened!
- Accessibility: SVG icon doesn't even have an alt attribute yet. Should use ARIA when the thing appears.
It's also not visible on mobile, need to think about how that will work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701664306,https://api.github.com/repos/simonw/datasette/issues/981,701664306,MDEyOklzc3VlQ29tbWVudDcwMTY2NDMwNg==,9599,simonw,2020-09-30T21:47:08Z,2020-09-30T21:47:08Z,OWNER,"The arrow icon didn't make sense because I already have a triangle icon showing sort order. I'm trying a cog icon instead:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701659197,https://api.github.com/repos/simonw/datasette/issues/981,701659197,MDEyOklzc3VlQ29tbWVudDcwMTY1OTE5Nw==,9599,simonw,2020-09-30T21:34:44Z,2020-09-30T21:34:44Z,OWNER,"Showing ""facet by this"" on the primary key column doesn't make sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701642448,https://api.github.com/repos/simonw/datasette/issues/981,701642448,MDEyOklzc3VlQ29tbWVudDcwMTY0MjQ0OA==,9599,simonw,2020-09-30T20:59:33Z,2020-09-30T20:59:33Z,OWNER,I think I've got everything I need to implement this now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/983#issuecomment-701629984,https://api.github.com/repos/simonw/datasette/issues/983,701629984,MDEyOklzc3VlQ29tbWVudDcwMTYyOTk4NA==,9599,simonw,2020-09-30T20:34:43Z,2020-09-30T20:34:43Z,OWNER,"I had a look around and there isn't an obvious pluggy equivalent in JavaScript world at the moment. Lots of frameworks like jQuery and Vue have their own custom plugin mechanisms.
https://github.com/rekit/js-plugin is a simple standalone plugin mechanism. Not quite as full-featured as Pluggy though - in particular I like how Pluggy supports multiple plugins returning results for the same hook that get concatenated into a list of results.
https://css-tricks.com/designing-a-javascript-plugin-system/ has some ideas.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/sqlite-utils/pull/178#issuecomment-701627158,https://api.github.com/repos/simonw/sqlite-utils/issues/178,701627158,MDEyOklzc3VlQ29tbWVudDcwMTYyNzE1OA==,9599,simonw,2020-09-30T20:29:11Z,2020-09-30T20:29:11Z,OWNER,Thanks for the fix!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709043182,Update README.md,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-701626134,https://api.github.com/repos/simonw/sqlite-utils/issues/182,701626134,MDEyOklzc3VlQ29tbWVudDcwMTYyNjEzNA==,9599,simonw,2020-09-30T20:27:09Z,2020-09-30T20:27:42Z,OWNER,"It looks like http://maps.natalian.org/data.txt is encoded as `latin-1`, but `sqlite-utils` assumes `utf-8` and hence breaks.
It would be worth improving the error message here. I could also add a `--encoding latin-1` option to `sqlite-utils insert` to help in consuming files that are stored in charsets other than `utf-8`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/datasette/issues/981#issuecomment-701616922,https://api.github.com/repos/simonw/datasette/issues/981,701616922,MDEyOklzc3VlQ29tbWVudDcwMTYxNjkyMg==,9599,simonw,2020-09-30T20:08:02Z,2020-09-30T20:08:02Z,OWNER,It would be neat to provide a JavaScript plugin hook that plugins can use to add their own options to this menu. No idea what that would look like though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701157010,https://api.github.com/repos/simonw/datasette/issues/981,701157010,MDEyOklzc3VlQ29tbWVudDcwMTE1NzAxMA==,9599,simonw,2020-09-30T05:00:42Z,2020-09-30T20:06:59Z,OWNER,"Maybe use this as the icon:
```svg
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701615291,https://api.github.com/repos/simonw/datasette/issues/981,701615291,MDEyOklzc3VlQ29tbWVudDcwMTYxNTI5MQ==,9599,simonw,2020-09-30T20:04:34Z,2020-09-30T20:05:37Z,OWNER,"Another potential action:
* Show rows where this is not blank (equivalent to `is not blank` filter)
This could be displayed conditionally based on if the column is detected to have any blank rows in it?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/982#issuecomment-701585695,https://api.github.com/repos/simonw/datasette/issues/982,701585695,MDEyOklzc3VlQ29tbWVudDcwMTU4NTY5NQ==,9599,simonw,2020-09-30T19:06:29Z,2020-09-30T19:06:29Z,OWNER,This is a little related to the error display issue #619 in that both will require some reworking of how the code is structured.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712202333,"SQL editor should allow execution of write queries, if you have permission",
https://github.com/simonw/datasette/issues/982#issuecomment-701585075,https://api.github.com/repos/simonw/datasette/issues/982,701585075,MDEyOklzc3VlQ29tbWVudDcwMTU4NTA3NQ==,9599,simonw,2020-09-30T19:05:11Z,2020-09-30T19:05:11Z,OWNER,The form needs to switch from GET to POST if the query is a write query. JavaScript can handle this based on the checkbox - if a user does not have JavaScript submitting the form will cause the form action to be changed to POST and the form to be redisplayed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712202333,"SQL editor should allow execution of write queries, if you have permission",
https://github.com/simonw/datasette/issues/981#issuecomment-701153543,https://api.github.com/repos/simonw/datasette/issues/981,701153543,MDEyOklzc3VlQ29tbWVudDcwMTE1MzU0Mw==,9599,simonw,2020-09-30T04:46:05Z,2020-09-30T05:11:45Z,OWNER,"Prototype:
```html
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701153982,https://api.github.com/repos/simonw/datasette/issues/981,701153982,MDEyOklzc3VlQ29tbWVudDcwMTE1Mzk4Mg==,9599,simonw,2020-09-30T04:47:54Z,2020-09-30T04:47:54Z,OWNER,"I think the accessible way to do this is with absolute positioning - have a menu icon in the `
` which, when clicked, causes the dropdown menu to appear as an absolutely positioned `
` that is not located within the DOM hierarchy of the`
` itself but is positioned to show up in the correct place.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701153822,https://api.github.com/repos/simonw/datasette/issues/981,701153822,MDEyOklzc3VlQ29tbWVudDcwMTE1MzgyMg==,9599,simonw,2020-09-30T04:47:10Z,2020-09-30T04:47:10Z,OWNER,"Future version could have expanding out nested side menus that let you do things like ""calculate sum/avg for this column against this-other-column"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701153600,https://api.github.com/repos/simonw/datasette/issues/981,701153600,MDEyOklzc3VlQ29tbWVudDcwMTE1MzYwMA==,9599,simonw,2020-09-30T04:46:18Z,2020-09-30T04:46:18Z,OWNER,"More options:
```html
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/980#issuecomment-700929721,https://api.github.com/repos/simonw/datasette/issues/980,700929721,MDEyOklzc3VlQ29tbWVudDcwMDkyOTcyMQ==,9599,simonw,2020-09-29T19:21:50Z,2020-09-29T19:21:50Z,OWNER,"That fixed it: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule(%27FREQ%3DHOURLY%3BCOUNT%3D5%27)%2C%0D%0A++dateutil_rrule_date(%0D%0A++++%27FREQ%3DDAILY%3BCOUNT%3D3%27%2C%0D%0A++++%271st+jan+2020%27%0D%0A++)%3B
```html
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710819020,Another rendering glitch with column headers on mobile,
https://github.com/simonw/datasette/issues/980#issuecomment-700490225,https://api.github.com/repos/simonw/datasette/issues/980,700490225,MDEyOklzc3VlQ29tbWVudDcwMDQ5MDIyNQ==,9599,simonw,2020-09-29T06:53:37Z,2020-09-29T06:53:37Z,OWNER,"This time it's because there are newlines in the column header:
```html
```
Those need to be escaped somehow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710819020,Another rendering glitch with column headers on mobile,
https://github.com/simonw/datasette/issues/979#issuecomment-700343373,https://api.github.com/repos/simonw/datasette/issues/979,700343373,MDEyOklzc3VlQ29tbWVudDcwMDM0MzM3Mw==,9599,simonw,2020-09-28T23:56:27Z,2020-09-28T23:56:27Z,OWNER,This would benefit https://github.com/simonw/datasette-import-table - which currently ignores the `CREATE TABLE` and derives the schema by inserting rows.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710650633,Default table view JSON should include CREATE TABLE,
https://github.com/simonw/datasette/issues/979#issuecomment-700343229,https://api.github.com/repos/simonw/datasette/issues/979,700343229,MDEyOklzc3VlQ29tbWVudDcwMDM0MzIyOQ==,9599,simonw,2020-09-28T23:55:55Z,2020-09-28T23:55:55Z,OWNER,Here's the code that adds it to the HTML context: https://github.com/simonw/datasette/blob/c11383e6284e000b2641569457efa16ac9e0d6ae/datasette/views/table.py#L835-L837,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710650633,Default table view JSON should include CREATE TABLE,
https://github.com/simonw/datasette/issues/978#issuecomment-700320480,https://api.github.com/repos/simonw/datasette/issues/978,700320480,MDEyOklzc3VlQ29tbWVudDcwMDMyMDQ4MA==,9599,simonw,2020-09-28T22:39:18Z,2020-09-28T22:39:18Z,OWNER,"```python
def escape_css_string(s):
return _css_re.sub(lambda m: ""\\"" + (""{:X}"".format(ord(m.group())).zfill(6)), s)
```
That fixes it:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700319656,https://api.github.com/repos/simonw/datasette/issues/978,700319656,MDEyOklzc3VlQ29tbWVudDcwMDMxOTY1Ng==,9599,simonw,2020-09-28T22:36:44Z,2020-09-28T22:36:44Z,OWNER,"Weirdly even those leading 0s doesn't fix it:
But... padding to six characters does! See https://www.w3.org/International/questions/qa-escapes
```
In [32]: print('\\' + ""{:X}"".format(ord('""')).zfill(6))
\000022
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700317760,https://api.github.com/repos/simonw/datasette/issues/978,700317760,MDEyOklzc3VlQ29tbWVudDcwMDMxNzc2MA==,9599,simonw,2020-09-28T22:30:25Z,2020-09-28T22:30:25Z,OWNER,"```python
print('\\' + ""{:X}"".format(ord('""')).zfill(4))
\0022
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700316511,https://api.github.com/repos/simonw/datasette/issues/978,700316511,MDEyOklzc3VlQ29tbWVudDcwMDMxNjUxMQ==,9599,simonw,2020-09-28T22:26:38Z,2020-09-28T22:26:38Z,OWNER,The fix may be to use `\0022` instead of `\22`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700314509,https://api.github.com/repos/simonw/datasette/issues/978,700314509,MDEyOklzc3VlQ29tbWVudDcwMDMxNDUwOQ==,9599,simonw,2020-09-28T22:20:51Z,2020-09-28T22:20:51Z,OWNER,"Here's the HTML for the broken example above:
```html
```
The glitch affects the ones where the quote is followed by digits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700313836,https://api.github.com/repos/simonw/datasette/issues/978,700313836,MDEyOklzc3VlQ29tbWVudDcwMDMxMzgzNg==,9599,simonw,2020-09-28T22:19:05Z,2020-09-28T22:19:05Z,OWNER,Looks like a bug in this function: https://github.com/simonw/datasette/blob/1f021c37110fc9019b0ef70062c28c335e568ae2/datasette/utils/__init__.py#L269-L274,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/pull/977#issuecomment-700012161,https://api.github.com/repos/simonw/datasette/issues/977,700012161,MDEyOklzc3VlQ29tbWVudDcwMDAxMjE2MQ==,22429695,codecov[bot],2020-09-28T13:37:44Z,2020-09-28T13:37:44Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=h1) Report
> Merging [#977](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/9a6d0dce282e7fb58c5610e24c74098c923abfdc?el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/977/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #977 +/- ##
=======================================
Coverage 84.27% 84.27%
=======================================
Files 28 28
Lines 3847 3847
=======================================
Hits 3242 3242
Misses 605 605
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=footer). Last update [9a6d0dc...5c01344](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710269200,"Update pytest requirement from <6.1.0,>=5.2.2 to >=5.2.2,<6.2.0",
https://github.com/simonw/sqlite-utils/issues/181#issuecomment-699762881,https://api.github.com/repos/simonw/sqlite-utils/issues/181,699762881,MDEyOklzc3VlQ29tbWVudDY5OTc2Mjg4MQ==,9599,simonw,2020-09-28T04:29:23Z,2020-09-28T04:29:23Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/94fc62857ee2655a21d85f6dae84b67bbfa5956d/sqlite_utils/db.py#L331-L367,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709920027,"pk=[""id""] should have same effect as pk=""id""",
https://github.com/simonw/sqlite-utils/issues/180#issuecomment-699718788,https://api.github.com/repos/simonw/sqlite-utils/issues/180,699718788,MDEyOklzc3VlQ29tbWVudDY5OTcxODc4OA==,9599,simonw,2020-09-28T01:11:45Z,2020-09-28T01:11:45Z,OWNER,https://hypothesis.readthedocs.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709861194,Try running some tests using Hypothesis,
https://github.com/simonw/datasette/issues/858#issuecomment-699690034,https://api.github.com/repos/simonw/datasette/issues/858,699690034,MDEyOklzc3VlQ29tbWVudDY5OTY5MDAzNA==,39445562,smithdc1,2020-09-27T21:23:04Z,2020-09-27T21:23:04Z,NONE,"Hi Simon,
Thanks so much for all your work on datasette, it's an excellent project and I wish you all the best with it. I particularly enjoyed your talk at the Django London Meetup a short while back.
I've been trying to publish to Heroku from Windows 10 and I was running into this error. I'm not sure why it can't be run without `shell=True` on Windows but this seems to help. With this change, I am able to publish if I pass in a `name` to the `publish` command. When a `name` is not passed the default of `datasette` is used and therefore this line here fails (as datasette at heroku already exists) and causes the recession error mentioned above.
https://github.com/simonw/datasette/blob/9a6d0dce282e7fb58c5610e24c74098c923abfdc/datasette/publish/heroku.py#L126
I tried to write a patch for this but I am really struggling with being on Windows (many of the tests seem to fail anyway?), and my lack of knowledge of Mock, so sorry for this. Hope this is of some help. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699524671,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699524671,MDEyOklzc3VlQ29tbWVudDY5OTUyNDY3MQ==,9599,simonw,2020-09-26T17:31:23Z,2020-09-27T20:31:50Z,OWNER,"SQL query for detecting integers:
```sql
select
'contains_non_integer' as result
from
mytable
where
cast(cast(mycolumn AS INTEGER) AS TEXT) != mycolumn
limit
1
```
This will return a single row with a 1 as soon as it comes across a column that contains a non-integer - so it short circuits quickly on TEXT columns with non-integers in them.
If everything in the column is an integer it will scan the whole thing before returning no rows.
More extensive demo:
```sql
select
value,
cast(cast(value AS INTEGER) AS TEXT) = value as is_valid_int
from
(
select
'1' as value
union
select
'1.1' as value
union
select
'dog' as value
union
select
null as value
)
```
https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+INTEGER%29+AS+TEXT%29+%3D+value+as+is_valid_int%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29
value | is_valid_int
-- | --
|
1 | 1
1.1 | 0
dog | 0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699684535,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699684535,MDEyOklzc3VlQ29tbWVudDY5OTY4NDUzNQ==,9599,simonw,2020-09-27T20:30:31Z,2020-09-27T20:30:31Z,OWNER,"This recipe looks like it might be the way to detect floats:
```sql
select
value,
cast(cast(value AS REAL) AS TEXT) in (value, value || '.0') as is_valid_float
from
(
select
'1' as value
union
select
'1.1' as value
union
select
'dog' as value
union
select
null as value
)
```
Demo: https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+REAL%29+AS+TEXT%29+in+%28value%2C+value+%7C%7C+%27.0%27%29+as+is_valid_float%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29
value | is_valid_float
-- | --
|
1 | 1
1.1 | 1
dog | 0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699526149,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699526149,MDEyOklzc3VlQ29tbWVudDY5OTUyNjE0OQ==,9599,simonw,2020-09-26T17:43:28Z,2020-09-26T17:43:28Z,OWNER,Posed a question about this on the SQLite forum here: https://sqlite.org/forum/forumpost/ab0dcd66ef,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/138#issuecomment-698626768,https://api.github.com/repos/simonw/sqlite-utils/issues/138,698626768,MDEyOklzc3VlQ29tbWVudDY5ODYyNjc2OA==,9599,simonw,2020-09-24T22:46:56Z,2020-09-24T22:46:56Z,OWNER,"Yeah this works fine, added a new confirmatory test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684118950,extracts= doesn't configure foreign keys,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698578959,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698578959,MDEyOklzc3VlQ29tbWVudDY5ODU3ODk1OQ==,9599,simonw,2020-09-24T20:44:35Z,2020-09-24T20:50:19Z,OWNER,"I'm using a `click.File()` at the moment: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L496
I'll need to change that to be something that I can easily measure progress through. Also I should change its name - `json_file` is a bad name when it sometimes handles `csv` or `tsv` instead.
It looks like the argument provided by `click.File` doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698579389,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698579389,MDEyOklzc3VlQ29tbWVudDY5ODU3OTM4OQ==,9599,simonw,2020-09-24T20:45:29Z,2020-09-24T20:45:29Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L550-L560
Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698577508,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698577508,MDEyOklzc3VlQ29tbWVudDY5ODU3NzUwOA==,9599,simonw,2020-09-24T20:41:18Z,2020-09-24T20:41:18Z,OWNER,"I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.
Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/119#issuecomment-698575545,https://api.github.com/repos/simonw/sqlite-utils/issues/119,698575545,MDEyOklzc3VlQ29tbWVudDY5ODU3NTU0NQ==,9599,simonw,2020-09-24T20:36:59Z,2020-09-24T20:36:59Z,OWNER,This was implemented in #161.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770,Ability to remove a foreign key,
https://github.com/simonw/sqlite-utils/issues/176#issuecomment-698572493,https://api.github.com/repos/simonw/sqlite-utils/issues/176,698572493,MDEyOklzc3VlQ29tbWVudDY5ODU3MjQ5Mw==,9599,simonw,2020-09-24T20:30:18Z,2020-09-24T20:30:18Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/cli.html#transforming-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708293114,sqlite-utils transform column order option,
https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698572264,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698572264,MDEyOklzc3VlQ29tbWVudDY5ODU3MjI2NA==,9599,simonw,2020-09-24T20:29:48Z,2020-09-24T20:29:48Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=),
https://github.com/simonw/datasette/issues/976#issuecomment-698488971,https://api.github.com/repos/simonw/datasette/issues/976,698488971,MDEyOklzc3VlQ29tbWVudDY5ODQ4ODk3MQ==,9599,simonw,2020-09-24T17:42:09Z,2020-09-24T17:42:35Z,OWNER,This is complex enough new logic that it will need test coverage - specifically covering tables or databases with strange names.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708289783,Idea: -o could open to a more convenient location,
https://github.com/simonw/sqlite-utils/issues/177#issuecomment-698444567,https://api.github.com/repos/simonw/sqlite-utils/issues/177,698444567,MDEyOklzc3VlQ29tbWVudDY5ODQ0NDU2Nw==,9599,simonw,2020-09-24T16:14:47Z,2020-09-24T16:14:47Z,OWNER,"This is a backwards incompatible change, so technically I should bump the major version to 3. I'm not going to do that, because the feature is brand new and the chance that anyone has written code or shell scripts that use it is vanishingly small.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708301810,Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key,
https://github.com/simonw/sqlite-utils/issues/176#issuecomment-698438043,https://api.github.com/repos/simonw/sqlite-utils/issues/176,698438043,MDEyOklzc3VlQ29tbWVudDY5ODQzODA0Mw==,9599,simonw,2020-09-24T16:02:55Z,2020-09-24T16:02:55Z,OWNER,I think I'll call this option `--column-order` with a shortcut of `-o`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708293114,sqlite-utils transform column order option,
https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698434811,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698434811,MDEyOklzc3VlQ29tbWVudDY5ODQzNDgxMQ==,9599,simonw,2020-09-24T15:57:17Z,2020-09-24T15:57:17Z,OWNER,Landed that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=),
https://github.com/simonw/datasette/issues/970#issuecomment-698434236,https://api.github.com/repos/simonw/datasette/issues/970,698434236,MDEyOklzc3VlQ29tbWVudDY5ODQzNDIzNg==,9599,simonw,2020-09-24T15:56:18Z,2020-09-24T15:56:50Z,OWNER,"Idea: if a database only has a single table, this could open straight to `/db/table`. If it has multiple tables but a single database it could open straight to `/db`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url",
https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698412692,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698412692,MDEyOklzc3VlQ29tbWVudDY5ODQxMjY5Mg==,9599,simonw,2020-09-24T15:19:28Z,2020-09-24T15:19:28Z,OWNER,Need to land #174 first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=),
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698400790,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698400790,MDEyOklzc3VlQ29tbWVudDY5ODQwMDc5MA==,9599,simonw,2020-09-24T14:59:50Z,2020-09-24T14:59:50Z,OWNER,For reusing the lookup table: I'm going to raise an error if a lookup table exists but without the correct columns. The caller can then add those columns and try again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698184166,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698184166,MDEyOklzc3VlQ29tbWVudDY5ODE4NDE2Ng==,9599,simonw,2020-09-24T08:01:07Z,2020-09-24T08:01:07Z,OWNER,I may revert the now unnecessary undocumented tweaks to the `.update()` method made in 66d506587eba9f0715267d6560b97c1fa44cc781 as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182656,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698182656,MDEyOklzc3VlQ29tbWVudDY5ODE4MjY1Ng==,9599,simonw,2020-09-24T07:58:08Z,2020-09-24T07:58:08Z,OWNER,"The way the lookup table works here differs from the previous implementation. In the previous implementation the usage of `.lookup()` meant that an existing table would be modified to fit the new purpose. That no longer happens in this version. Need to make a design decision about how this should work.
It should definitely be possible to use an existing lookup table - imagine a database where several tables have a ""Departments"" column and we want to extract all of those values out to a single shared ""Departments"" table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182037,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698182037,MDEyOklzc3VlQ29tbWVudDY5ODE4MjAzNw==,9599,simonw,2020-09-24T07:56:50Z,2020-09-24T07:56:50Z,OWNER,I could also be a bit smarter about transaction handling. I think it may be possible to run this entire operation in a single transaction now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698181478,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698181478,MDEyOklzc3VlQ29tbWVudDY5ODE4MTQ3OA==,9599,simonw,2020-09-24T07:55:45Z,2020-09-24T07:55:45Z,OWNER,`import functools` is no longer needed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180705,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698180705,MDEyOklzc3VlQ29tbWVudDY5ODE4MDcwNQ==,9599,simonw,2020-09-24T07:54:10Z,2020-09-24T07:54:10Z,OWNER,"After running through the steps in https://simonwillison.net/2020/Sep/23/sqlite-utils-extract/ I get a table that looks like this:
The foreign key columns are all at the end of the table. It would be nicer if they were arranged in the same order as the columns they replaced.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180113,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698180113,MDEyOklzc3VlQ29tbWVudDY5ODE4MDExMw==,9599,simonw,2020-09-24T07:53:03Z,2020-09-24T07:53:03Z,OWNER,This could do with a little bit more testing - I'm worried there may be column or table name edge cases that are not covered yet. I also need to remove the progress bar code since that no longer makes sense for this implementation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-698178101,https://api.github.com/repos/simonw/sqlite-utils/issues/172,698178101,MDEyOklzc3VlQ29tbWVudDY5ODE3ODEwMQ==,9599,simonw,2020-09-24T07:48:57Z,2020-09-24T07:49:20Z,OWNER,"> I wonder if I could make this faster by separating it out into a few steps:
>
> * Create the new lookup table with all of the distinct rows
>
> * Add the blank foreign key column
>
> * run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)`
>
> * Drop the value columns
My prototype of this knocked the time down from 10 minutes to 4 seconds, so I think the change is worth it!
```
% date
sqlite-utils extract salaries.db salaries \
'Department Code' 'Department' \
--table 'departments' \
--fk-column 'department_id' \
--rename 'Department Code' code \
--rename 'Department' name
date
sqlite-utils extract salaries.db salaries \
'Union Code' 'Union' \
--table 'unions' \
--fk-column 'union_id' \
--rename 'Union Code' code \
--rename 'Union' name
date
sqlite-utils extract salaries.db salaries \
'Job Family Code' 'Job Family' \
--table 'job_families' \
--fk-column 'job_family_id' \
--rename 'Job Family Code' code \
--rename 'Job Family' name
date
sqlite-utils extract salaries.db salaries \
'Job Code' 'Job' \
--table 'jobs' \
--fk-column 'job_id' \
--rename 'Job Code' code \
--rename 'Job' name
date
Thu Sep 24 00:48:16 PDT 2020
Thu Sep 24 00:48:20 PDT 2020
Thu Sep 24 00:48:24 PDT 2020
Thu Sep 24 00:48:28 PDT 2020
Thu Sep 24 00:48:32 PDT 2020
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/datasette/issues/123#issuecomment-698174957,https://api.github.com/repos/simonw/datasette/issues/123,698174957,MDEyOklzc3VlQ29tbWVudDY5ODE3NDk1Nw==,45416,obra,2020-09-24T07:42:05Z,2020-09-24T07:42:05Z,NONE,"
Oh. Awesome.
On Thu, Sep 24, 2020 at 12:28:53AM -0700, Simon Willison wrote:
> @obra there's a plugin for that! https://github.com/simonw/
> datasette-upload-csvs
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.*
>
--
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/simonw/datasette/issues/123#issuecomment-698168648,https://api.github.com/repos/simonw/datasette/issues/123,698168648,MDEyOklzc3VlQ29tbWVudDY5ODE2ODY0OA==,9599,simonw,2020-09-24T07:28:38Z,2020-09-24T07:28:38Z,OWNER,@obra there's a plugin for that! https://github.com/simonw/datasette-upload-csvs,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/simonw/datasette/issues/974#issuecomment-698110492,https://api.github.com/repos/simonw/datasette/issues/974,698110492,MDEyOklzc3VlQ29tbWVudDY5ODExMDQ5Mg==,9599,simonw,2020-09-24T04:50:56Z,2020-09-24T04:51:05Z,OWNER,"Come to think of it I've noticed that in the logs when it's running on my laptop, definitely worth fixing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707849175,static assets and favicon aren't cached by the browser,
https://github.com/simonw/datasette/issues/123#issuecomment-698110186,https://api.github.com/repos/simonw/datasette/issues/123,698110186,MDEyOklzc3VlQ29tbWVudDY5ODExMDE4Ng==,45416,obra,2020-09-24T04:49:51Z,2020-09-24T04:49:51Z,NONE,"As a half-measure, I'd get value out of being able to upload a CSV and have datasette run csv-to-sqlite on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/simonw/datasette/issues/619#issuecomment-698024773,https://api.github.com/repos/simonw/datasette/issues/619,698024773,MDEyOklzc3VlQ29tbWVudDY5ODAyNDc3Mw==,9599,simonw,2020-09-23T23:31:46Z,2020-09-23T23:31:46Z,OWNER,"I'm going to have to untangle Datasette's error handling a bit for this - currently the expectation is that exceptions will be handled at a higher level, but I need to rethink that to make it cleaner for views like the ""execute custom SQL"" view to add their own error handling (and still be able to return the correct HTTP status codes, even with custom pages).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697998045,https://api.github.com/repos/simonw/datasette/issues/619,697998045,MDEyOklzc3VlQ29tbWVudDY5Nzk5ODA0NQ==,9599,simonw,2020-09-23T22:09:06Z,2020-09-23T22:09:06Z,OWNER,"I'll add this to the succesful JSON format:
```json
{
""ok"": true,
""error"": null
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697995885,https://api.github.com/repos/simonw/datasette/issues/619,697995885,MDEyOklzc3VlQ29tbWVudDY5Nzk5NTg4NQ==,9599,simonw,2020-09-23T22:02:44Z,2020-09-23T22:08:28Z,OWNER,"So the JSON (still served with a 500 code) will look something like this:
```json
{
""ok"": false,
""status"": 500,
""database"": ""fixtures"",
""query_name"": null,
""rows"": [],
""truncated"": false,
""error"": ""Error message goes here"",
""columns"": [],
""query"": {
""sql"": ""the query that broke goes here"",
""params"": {}
},
""private"": false,
""allow_execute_sql"": true,
""query_ms"": 0.8716583251953125,
""source"": ""tests/fixtures.py"",
""source_url"": ""https://github.com/simonw/datasette/blob/master/tests/fixtures.py"",
""license"": ""Apache License 2.0"",
""license_url"": ""https://github.com/simonw/datasette/blob/master/LICENSE""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697995303,https://api.github.com/repos/simonw/datasette/issues/619,697995303,MDEyOklzc3VlQ29tbWVudDY5Nzk5NTMwMw==,9599,simonw,2020-09-23T22:01:08Z,2020-09-23T22:01:08Z,OWNER,"This is a little tricky to solve, because of the location of the form and the need to return JSON as well as HTML. It would be weird if a JSON request came in and got back the standard output from https://latest.datasette.io/fixtures.json when they were expecting to get back JSON in the shape of https://latest.datasette.io/fixtures.json?sql=select%20*%20from%20sqlite_master
I'm going to return the HTML view that you would get for 0 results for a query - https://latest.datasette.io/fixtures?sql=select%201%20limit%200 - but with an error message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697980061,https://api.github.com/repos/simonw/datasette/issues/619,697980061,MDEyOklzc3VlQ29tbWVudDY5Nzk4MDA2MQ==,9599,simonw,2020-09-23T21:22:42Z,2020-09-23T21:22:42Z,OWNER,Yeah that sucks. Bumping this up the priority list.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697973420,https://api.github.com/repos/simonw/datasette/issues/619,697973420,MDEyOklzc3VlQ29tbWVudDY5Nzk3MzQyMA==,45416,obra,2020-09-23T21:07:58Z,2020-09-23T21:07:58Z,NONE,"I've just run into this after crafting a complex query and discovered that hitting back loses my query.
Even showing me the whole bad query would be a huge improvement over the current status quo.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697869886,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697869886,MDEyOklzc3VlQ29tbWVudDY5Nzg2OTg4Ng==,9599,simonw,2020-09-23T18:45:30Z,2020-09-23T18:45:30Z,OWNER,"There's something to be said for making this operation pausable and resumable, especially if I'm going to make it available in a Datasette plugin at some point.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697866885,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697866885,MDEyOklzc3VlQ29tbWVudDY5Nzg2Njg4NQ==,9599,simonw,2020-09-23T18:43:37Z,2020-09-23T18:43:37Z,OWNER,Also what would happen if the table had new rows added to it while that command was running?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697863116,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697863116,MDEyOklzc3VlQ29tbWVudDY5Nzg2MzExNg==,9599,simonw,2020-09-23T18:41:06Z,2020-09-23T18:41:06Z,OWNER,Problem with this approach is it's not compatible with progress bars - but if it's a multiple of times faster it's worth it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697859772,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697859772,MDEyOklzc3VlQ29tbWVudDY5Nzg1OTc3Mg==,9599,simonw,2020-09-23T18:38:43Z,2020-09-23T18:38:52Z,OWNER,"I wonder if I could make this faster by separating it out into a few steps:
- Create the new lookup table with all of the distinct rows
- Add the blank foreign key column
- run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)`
- Drop the value columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697835956,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697835956,MDEyOklzc3VlQ29tbWVudDY5NzgzNTk1Ng==,9599,simonw,2020-09-23T18:22:49Z,2020-09-23T18:22:49Z,OWNER,"I ran `sudo py-spy top -p 123` against the process while it was running and the most time is definitely spent in `.update()`:
```
Total Samples 1000
GIL: 0.00%, Active: 90.00%, Threads: 1
%Own %Total OwnTime TotalTime Function (filename:line)
38.00% 38.00% 3.85s 3.85s update (sqlite_utils/db.py:1283)
27.00% 27.00% 2.12s 2.12s execute (sqlite_utils/db.py:161)
10.00% 10.00% 0.890s 0.890s execute (sqlite_utils/db.py:163)
10.00% 17.00% 0.870s 1.54s columns (sqlite_utils/db.py:553)
0.00% 0.00% 0.110s 0.210s (sqlite_utils/db.py:554)
0.00% 3.00% 0.100s 0.320s table_names (sqlite_utils/db.py:191)
0.00% 0.00% 0.100s 0.100s __new__ (:1)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-697577646,https://api.github.com/repos/simonw/sqlite-utils/issues/173,697577646,MDEyOklzc3VlQ29tbWVudDY5NzU3NzY0Ng==,9599,simonw,2020-09-23T15:48:51Z,2020-09-23T15:48:51Z,OWNER,"This can only work when it's reading from a file, not when it's reading from standard input.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/datasette/issues/111#issuecomment-697545290,https://api.github.com/repos/simonw/datasette/issues/111,697545290,MDEyOklzc3VlQ29tbWVudDY5NzU0NTI5MA==,9599,simonw,2020-09-23T15:29:11Z,2020-09-23T15:29:11Z,OWNER,This is still a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697473247,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697473247,MDEyOklzc3VlQ29tbWVudDY5NzQ3MzI0Nw==,9599,simonw,2020-09-23T14:45:13Z,2020-09-23T14:45:13Z,OWNER,"`lookup_table.lookup(lookups)` is doing a SQL lookup. This could be cached in-memory, maybe with a LRU cache, to avoid looking up the primary key for records that we have recently used.
The `.update()` method it is calling first does a `get()` and then does a SQL `UPDATE ... WHERE`:
https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L1244-L1264
Batching those updates may have an effect. Or finding a way to skip the `.get()` since we already know we have a valid record.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697467833,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697467833,MDEyOklzc3VlQ29tbWVudDY5NzQ2NzgzMw==,9599,simonw,2020-09-23T14:42:03Z,2020-09-23T14:42:03Z,OWNER,Here's the loop that's taking the time: https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L892-L897,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697466497,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697466497,MDEyOklzc3VlQ29tbWVudDY5NzQ2NjQ5Nw==,9599,simonw,2020-09-23T14:41:17Z,2020-09-23T14:41:17Z,OWNER,"Steps to produce that database:
```
curl -o salaries.csv 'https://data.sfgov.org/api/views/88g8-5mnd/rows.csv?accessType=DOWNLOAD'
sqlite-utils insert salaries.db salaries salaries.csv --csv
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/datasette/issues/970#issuecomment-697073465,https://api.github.com/repos/simonw/datasette/issues/970,697073465,MDEyOklzc3VlQ29tbWVudDY5NzA3MzQ2NQ==,2861690,secretGeek,2020-09-23T01:49:05Z,2020-09-23T01:49:05Z,NONE,"Oh wow oh wow. Thanks so much Simon. In an astoundingly rough week, this is a shining jewel. 🤣","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url",
https://github.com/simonw/sqlite-utils/issues/170#issuecomment-697047591,https://api.github.com/repos/simonw/sqlite-utils/issues/170,697047591,MDEyOklzc3VlQ29tbWVudDY5NzA0NzU5MQ==,9599,simonw,2020-09-23T00:14:52Z,2020-09-23T00:14:52Z,OWNER,"
@simonw
@db.register_function decorator, closes #162
4824775
@simonw
table.transform() method - closes #114
987dd12
@simonw
Keyword only arguments for transform()
f8e10df
Also renamed columns= to types=
Closes #165
Commits on Sep 22, 2020
@simonw
Implemented sqlite-utils transform command, closes #164
752d261
@simonw
Applied Black
f29f682
@simonw
table.extract() method, refs #42
f855379
@simonw
Docstring for sqlite-utils transform
c755f28
@simonw
Added table.extract(rename=) option, refs #42
c3210f2
@simonw
Applied Black
317071a
@simonw
New .rows_where(select=) argument
7178231
@simonw
table.extract() now works with rowid tables, refs #42
2db6c5b
@simonw
sqlite-utils extract, closes #42
55cf928
@simonw
Progress bar for ""sqlite-utils extract"", closes #169
5c4d58d
@simonw
Fixed PRAGMA foreign_keys handling for .transform, closes #167 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706768798,Release notes for 2.20,
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697037974,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697037974,MDEyOklzc3VlQ29tbWVudDY5NzAzNzk3NA==,9599,simonw,2020-09-22T23:39:31Z,2020-09-22T23:39:31Z,OWNER,Documentation for `sqlite-utils extract`: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697031174,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697031174,MDEyOklzc3VlQ29tbWVudDY5NzAzMTE3NA==,9599,simonw,2020-09-22T23:16:00Z,2020-09-22T23:16:00Z,OWNER,"Trying this demo again:
```
wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv
sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name
```
It worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697025403,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697025403,MDEyOklzc3VlQ29tbWVudDY5NzAyNTQwMw==,9599,simonw,2020-09-22T22:57:53Z,2020-09-22T22:57:53Z,OWNER,The documentation for the `.extract()` method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697019944,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697019944,MDEyOklzc3VlQ29tbWVudDY5NzAxOTk0NA==,9599,simonw,2020-09-22T22:40:00Z,2020-09-22T22:40:00Z,OWNER,"I tried out the prototype of the CLI on the Global Power Plants data:
```
wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv
sqlite-utils extract global.db power_plants country country_long
```
This threw an error because `rowid` columns are not yet supported. I fixed that like so:
```
sqlite-utils transform global.db power_plants --rename rowid id
sqlite-utils extract global.db power_plants country country_long
```
That worked! But it didn't play great with Datasette, because the resulting extracted table had columns `country` and `country_long` and neither of those are called `name` or `value` or `title`.
Based on this I need to add `rowid` table support AND I need to implement the proposed `rename=` argument for renaming columns on their way into the new table.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697013681,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697013681,MDEyOklzc3VlQ29tbWVudDY5NzAxMzY4MQ==,9599,simonw,2020-09-22T22:22:49Z,2020-09-22T22:22:49Z,OWNER,"The command-line version of this needs to accept a table and one or more columns, then a `--table` and `--fk-column` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697012111,MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ==,9599,simonw,2020-09-22T22:18:13Z,2020-09-22T22:18:13Z,OWNER,"Here's how I'm generating the examples for the documentation:
```
In [2]: import sqlite_utils
In [3]: db = sqlite_utils.Database(memory=True)
In [4]: db[""Trees""].insert({""id"": 1, ""TreeAddress"": ""52 Vine St"", ""CommonName"":
...: ""Palm"", ""LatinName"": ""foo""}, pk=""id"")
Out[4]:
In [5]: db[""Trees""].extract([""CommonName"", ""LatinName""], table=""Species"", fk_col
...: umn=""species_id"")
In [6]: print(db[""Trees""].schema)
CREATE TABLE ""Trees"" (
[id] INTEGER PRIMARY KEY,
[TreeAddress] TEXT,
[species_id] INTEGER,
FOREIGN KEY(species_id) REFERENCES Species(id)
)
In [7]: print(db[""Species""].schema)
CREATE TABLE [Species] (
[id] INTEGER PRIMARY KEY,
[CommonName] TEXT,
[LatinName] TEXT
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987925,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987925,MDEyOklzc3VlQ29tbWVudDY5Njk4NzkyNQ==,9599,simonw,2020-09-22T21:19:04Z,2020-09-22T21:19:04Z,OWNER,Need to make sure this works correctly for `rowid` tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987257,MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw==,9599,simonw,2020-09-22T21:17:34Z,2020-09-22T21:17:34Z,OWNER,"What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980709,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980709,MDEyOklzc3VlQ29tbWVudDY5Njk4MDcwOQ==,9599,simonw,2020-09-22T21:05:07Z,2020-09-22T21:05:07Z,OWNER,"So `.extract()` probably takes a `batch_size=` argument too, which defaults to maybe 1000.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980503,MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw==,9599,simonw,2020-09-22T21:04:45Z,2020-09-22T21:04:45Z,OWNER,"`table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979626,MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg==,9599,simonw,2020-09-22T21:03:11Z,2020-09-22T21:03:11Z,OWNER,"And if you want to rename some of the columns in the new table:
```python
db[""trees""].extract([""common_name"", ""latin_name""], table=""species"", rename={""common_name"": ""name""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979168,MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA==,9599,simonw,2020-09-22T21:02:24Z,2020-09-22T21:02:24Z,OWNER,"In Python it looks like this:
```python
# Simple case - species column species_id pointing to species table
db[""trees""].extract(""species"")
# Setting a custom table
db[""trees""].extract(""species"", table=""Species"")
# Custom foreign key column on trees
db[""trees""].extract(""species"", fk_column=""species"")
# Extracting multiple columns
db[""trees""].extract([""common_name"", ""latin_name""])
# (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id)
# Or with explicit table (fk_column here defaults to species_id because of the table name)
db[""trees""].extract([""common_name"", ""latin_name""], table=""species"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696976678,MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA==,9599,simonw,2020-09-22T20:57:57Z,2020-09-22T20:57:57Z,OWNER,"I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID.
That integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893774,MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA==,9599,simonw,2020-09-22T18:15:33Z,2020-09-22T18:15:33Z,OWNER,I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893244,MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA==,9599,simonw,2020-09-22T18:14:33Z,2020-09-22T18:14:45Z,OWNER,"Thinking more about this one:
```
$ sqlite-utils extract my.db \
dea_sales company_name company_address \
--table companies
```
The goal here is to pull the company name and address pair out into a separate table.
Some questions:
- should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this.
- what should the foreign key column that gets added to the `companies` table be called?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513262013,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513262013,MDEyOklzc3VlQ29tbWVudDUxMzI2MjAxMw==,9599,simonw,2019-07-19T14:58:23Z,2020-09-22T18:12:11Z,OWNER,"CLI design idea:
$ sqlite-utils extract my.db \
dea_sales company_name
Here we just specify the original table and column - the new extracted table will automatically be called ""company_name"" and will have ""id"" and ""value"" columns, by default.
To set a custom extract table:
$ sqlite-utils extract my.db \
dea_sales company_name \
--table companies
And for extracting multiple columns and renaming them on the created table, maybe something like this:
$ sqlite-utils extract my.db \
dea_sales company_name company_address \
--table companies \
--column company_name name \
--column company_address address
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/datasette/issues/973#issuecomment-696800410,https://api.github.com/repos/simonw/datasette/issues/973,696800410,MDEyOklzc3VlQ29tbWVudDY5NjgwMDQxMA==,9599,simonw,2020-09-22T15:35:28Z,2020-09-22T15:35:28Z,OWNER,"Confirmed in local dev:
```
% datasette fixtures.db --inspect-file inspect.json
Traceback (most recent call last):
File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette"", line 11, in
load_entry_point('datasette', 'console_scripts', 'datasette')()
File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
return self.main(*args, **kwargs)
File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 782, in main
rv = self.invoke(ctx)
File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
return callback(*args, **kwargs)
File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 406, in serve
inspect_data = json.load(open(inspect_file))
TypeError: 'bool' object is not callable
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706486323,'bool' object is not callable error,
https://github.com/simonw/datasette/issues/973#issuecomment-696798114,https://api.github.com/repos/simonw/datasette/issues/973,696798114,MDEyOklzc3VlQ29tbWVudDY5Njc5ODExNA==,9599,simonw,2020-09-22T15:31:25Z,2020-09-22T15:31:25Z,OWNER,D'oh because I have a new variable called `open`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706486323,'bool' object is not callable error,
https://github.com/simonw/datasette/issues/969#issuecomment-696788109,https://api.github.com/repos/simonw/datasette/issues/969,696788109,MDEyOklzc3VlQ29tbWVudDY5Njc4ODEwOQ==,9599,simonw,2020-09-22T15:15:14Z,2020-09-22T15:15:14Z,OWNER,"I don't think a standard ""pass these extra arguments to the publish tool"" mechanism will work because there's no guarantee that a publisher uses a CLI tool - or if it does, it might make several calls to different CLI tools. The Cloud Run one runs a couple of commands, as illustrated by this test:
https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/tests/test_publish_cloudrun.py#L63-L73
Adding a `--tar` option for `datasette publish heroku` is a good fix for this though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705057955,"Add --tar option to ""datasette publish heroku""",
https://github.com/simonw/datasette/issues/943#issuecomment-696778735,https://api.github.com/repos/simonw/datasette/issues/943,696778735,MDEyOklzc3VlQ29tbWVudDY5Njc3ODczNQ==,9599,simonw,2020-09-22T15:00:13Z,2020-09-22T15:00:39Z,OWNER,"Am I going to rewrite ALL of my tests to use this instead? It would clean up a lot of test code, at the cost of quite a bit of work.
It would make for much neater plugin tests too, and neater testing documentation: https://docs.datasette.io/en/stable/testing_plugins.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696777886,https://api.github.com/repos/simonw/datasette/issues/943,696777886,MDEyOklzc3VlQ29tbWVudDY5Njc3Nzg4Ng==,9599,simonw,2020-09-22T14:58:54Z,2020-09-22T14:58:54Z,OWNER,"```python
class DatasetteClient:
def __init__(self, ds):
self._client = httpx.AsyncClient(app=ds.app())
def _fix(self, path):
if path.startswith(""/""):
path = ""http://localhost{}"".format(path)
return path
async def get(self, path, **kwargs):
return await self._client.get(self._fix(path), **kwargs)
async def options(self, path, **kwargs):
return await self._client.options(self._fix(path), **kwargs)
async def head(self, path, **kwargs):
return await self._client.head(self._fix(path), **kwargs)
async def post(self, path, **kwargs):
return await self._client.post(self._fix(path), **kwargs)
async def put(self, path, **kwargs):
return await self._client.put(self._fix(path), **kwargs)
async def patch(self, path, **kwargs):
return await self._client.patch(self._fix(path), **kwargs)
async def delete(self, path, **kwargs):
return await self._client.delete(self._fix(path), **kwargs)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696776828,https://api.github.com/repos/simonw/datasette/issues/943,696776828,MDEyOklzc3VlQ29tbWVudDY5Njc3NjgyOA==,9599,simonw,2020-09-22T14:57:13Z,2020-09-22T14:57:13Z,OWNER,"I may as well implement all of the HTTP methods supported by the `httpx` client:
- get
- options
- head
- post
- put
- patch
- delete","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696775516,https://api.github.com/repos/simonw/datasette/issues/943,696775516,MDEyOklzc3VlQ29tbWVudDY5Njc3NTUxNg==,9599,simonw,2020-09-22T14:55:10Z,2020-09-22T14:55:10Z,OWNER,"Even smaller `DatasetteClient` implementation:
```python
class DatasetteClient:
def __init__(self, ds):
self._client = httpx.AsyncClient(app=ds.app())
def _fix(self, path):
if path.startswith(""/""):
path = ""http://localhost{}"".format(path)
return path
async def get(self, path, **kwargs):
return await self._client.get(self._fix(path), **kwargs)
async def post(self, path, **kwargs):
return await self._client.post(self._fix(path), **kwargs)
async def options(self, path, **kwargs):
return await self._client.options(self._fix(path), **kwargs)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696774711,https://api.github.com/repos/simonw/datasette/issues/943,696774711,MDEyOklzc3VlQ29tbWVudDY5Njc3NDcxMQ==,9599,simonw,2020-09-22T14:53:56Z,2020-09-22T14:53:56Z,OWNER,"How important is it to use `httpx.AsyncClient` with a context manager?
https://www.python-httpx.org/async/#opening-and-closing-clients says:
> Alternatively, use `await client.aclose()` if you want to close a client explicitly:
>
> ```
> client = httpx.AsyncClient()
> ...
> await client.aclose()
> ```
The `.aclose()` method has a comment saying ""Close transport and proxies"" - I'm not using proxies, so the relevant implementation seems to be a call to `await self._transport.aclose()` in https://github.com/encode/httpx/blob/f932af9172d15a803ad40061a4c2c0cd891645cf/httpx/_client.py#L1741-L1751
The transport I am using is a class called `ASGITransport` in https://github.com/encode/httpx/blob/master/httpx/_transports/asgi.py
The `aclose()` method on that class does nothing. So it looks like I can instantiate a client without bothering with the `async with httpx.AsyncClient` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696769853,https://api.github.com/repos/simonw/datasette/issues/943,696769853,MDEyOklzc3VlQ29tbWVudDY5Njc2OTg1Mw==,9599,simonw,2020-09-22T14:46:21Z,2020-09-22T14:46:21Z,OWNER,This adds `httpx` as a dependency - I think I'm OK with that. I use it for testing in all of my plugins anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696769501,https://api.github.com/repos/simonw/datasette/issues/943,696769501,MDEyOklzc3VlQ29tbWVudDY5Njc2OTUwMQ==,9599,simonw,2020-09-22T14:45:49Z,2020-09-22T14:45:49Z,OWNER,"I put together a minimal prototype of this and it feels pretty good:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 20aae7d..fb3bdad 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -4,6 +4,7 @@ import collections
import datetime
import glob
import hashlib
+import httpx
import inspect
import itertools
from itsdangerous import BadSignature
@@ -312,6 +313,7 @@ class Datasette:
self._register_renderers()
self._permission_checks = collections.deque(maxlen=200)
self._root_token = secrets.token_hex(32)
+ self.client = DatasetteClient(self)
async def invoke_startup(self):
for hook in pm.hook.startup(datasette=self):
@@ -1209,3 +1211,25 @@ def route_pattern_from_filepath(filepath):
class NotFoundExplicit(NotFound):
pass
+
+
+class DatasetteClient:
+ def __init__(self, ds):
+ self.app = ds.app()
+
+ def _fix(self, path):
+ if path.startswith(""/""):
+ path = ""http://localhost{}"".format(path)
+ return path
+
+ async def get(self, path, **kwargs):
+ async with httpx.AsyncClient(app=self.app) as client:
+ return await client.get(self._fix(path), **kwargs)
+
+ async def post(self, path, **kwargs):
+ async with httpx.AsyncClient(app=self.app) as client:
+ return await client.post(self._fix(path), **kwargs)
+
+ async def options(self, path, **kwargs):
+ async with httpx.AsyncClient(app=self.app) as client:
+ return await client.options(self._fix(path), **kwargs)
```
Used like this in `ipython`:
```
In [1]: from datasette.app import Datasette
In [2]: ds = Datasette([""fixtures.db""])
In [3]: (await ds.client.get(""/-/config.json"")).json()
Out[3]:
{'default_page_size': 100,
'max_returned_rows': 1000,
'num_sql_threads': 3,
'sql_time_limit_ms': 1000,
'default_facet_size': 30,
'facet_time_limit_ms': 200,
'facet_suggest_time_limit_ms': 50,
'hash_urls': False,
'allow_facet': True,
'allow_download': True,
'suggest_facets': True,
'default_cache_ttl': 5,
'default_cache_ttl_hashed': 31536000,
'cache_size_kb': 0,
'allow_csv_stream': True,
'max_csv_mb': 100,
'truncate_cells_html': 2048,
'force_https_urls': False,
'template_debug': False,
'base_url': '/'}
In [4]: (await ds.client.get(""/fixtures/facetable.json?_shape=array"")).json()
Out[4]:
[{'pk': 1,
'created': '2019-01-14 08:00:00',
'planet_int': 1,
'on_earth': 1,
'state': 'CA',
'city_id': 1,
'neighborhood': 'Mission',
'tags': '[""tag1"", ""tag2""]',
'complex_array': '[{""foo"": ""bar""}]',
'distinct_some_null': 'one'},
{'pk': 2,
'created': '2019-01-14 08:00:00',
'planet_int': 1,
'on_earth': 1,
'state': 'CA',
'city_id': 1,
'neighborhood': 'Dogpatch',
'tags': '[""tag1"", ""tag3""]',
'complex_array': '[]',
'distinct_some_null': 'two'},
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693009048,https://api.github.com/repos/simonw/datasette/issues/943,693009048,MDEyOklzc3VlQ29tbWVudDY5MzAwOTA0OA==,9599,simonw,2020-09-15T22:17:30Z,2020-09-22T14:37:00Z,OWNER,"Maybe instead of implementing `datasette.get()` and `datasette.post()` and `datasette.request()` and `datasette.stream()` I could instead have a nested object called `datasette.client` which is a preconfigured `AsyncClient` instance.
```python
response = await datasette.client.get(""/"")
```
Or perhaps this should be a method in case I ever need to be able to `await` it:
```python
response = await (await datasette.client()).get(""/"")
```
This is a bit cosmetically ugly though, I'd rather avoid that if possible.
Maybe I could get this working by returning an object from `.client()` which provides a `await obj.get()` method:
```python
response = await datasette.client().get(""/"")
```
I don't think there's any benefit to that over `await datasette.client.get()` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/sqlite-utils/issues/168#issuecomment-696573944,https://api.github.com/repos/simonw/sqlite-utils/issues/168,696573944,MDEyOklzc3VlQ29tbWVudDY5NjU3Mzk0NA==,9599,simonw,2020-09-22T08:11:30Z,2020-09-22T08:11:30Z,OWNER,Huh... maybe I don't need to do anything here? It looks like it's been kept up to date: https://github.com/Homebrew/homebrew-core/commits/master/Formula/sqlite-utils.rb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706167456,Automate (as much as possible) updates published to Homebrew,
https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696567988,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696567988,MDEyOklzc3VlQ29tbWVudDY5NjU2Nzk4OA==,9599,simonw,2020-09-22T07:57:50Z,2020-09-22T07:57:50Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#transforming-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command,
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696567460,MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA==,9599,simonw,2020-09-22T07:56:42Z,2020-09-22T07:56:42Z,OWNER,`.transform()` has landed now which should make this a lot easier to solve.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/26#issuecomment-696566750,https://api.github.com/repos/simonw/sqlite-utils/issues/26,696566750,MDEyOklzc3VlQ29tbWVudDY5NjU2Njc1MA==,9599,simonw,2020-09-22T07:55:00Z,2020-09-22T07:55:00Z,OWNER,"Problem: `extract` means something else now, see #47 and the upcoming work in #42.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455486286,Mechanism for turning nested JSON into foreign keys / many-to-many,
https://github.com/simonw/sqlite-utils/issues/167#issuecomment-696565981,https://api.github.com/repos/simonw/sqlite-utils/issues/167,696565981,MDEyOklzc3VlQ29tbWVudDY5NjU2NTk4MQ==,9599,simonw,2020-09-22T07:53:13Z,2020-09-22T07:53:13Z,OWNER,"Confirmed this is a bug, https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes explicitly says you should do the `PRAGMA foreign_keys` bits before and after the transaction, not during.
Right now my code does this INSIDE the transaction: https://github.com/simonw/sqlite-utils/blob/f29f6821f2d08e91c5c6d65d885a1bbc0c743bdd/sqlite_utils/db.py#L790-L793
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706098005,Review the foreign key pragma stuff,
https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696520928,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696520928,MDEyOklzc3VlQ29tbWVudDY5NjUyMDkyOA==,9599,simonw,2020-09-22T05:50:17Z,2020-09-22T05:50:17Z,OWNER,"Idea for CLI options:
```
--type age integer
--drop colname
--rename oldname newname
--not-null col
--not-null-false col
--pk new_id
--pk-none
--default col value
--default-none column
--drop-foreign-key col other_table other_column
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command,
https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696500922,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696500922,MDEyOklzc3VlQ29tbWVudDY5NjUwMDkyMg==,9599,simonw,2020-09-22T04:22:40Z,2020-09-22T04:22:40Z,OWNER,Documentation for the `.transform()` method #114 (now landed) is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696500767,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696500767,MDEyOklzc3VlQ29tbWVudDY5NjUwMDc2Nw==,9599,simonw,2020-09-22T04:21:45Z,2020-09-22T04:21:45Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696494070,MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA==,9599,simonw,2020-09-22T03:48:58Z,2020-09-22T03:48:58Z,OWNER,"One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be:
> If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF.
And the last steps should be:
> If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints.
>
> Commit the transaction started in step 2.
>
> If foreign keys constraints were originally enabled, reenable them now.
I need to implement that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696490851,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696490851,MDEyOklzc3VlQ29tbWVudDY5NjQ5MDg1MQ==,9599,simonw,2020-09-22T03:33:54Z,2020-09-22T03:33:54Z,OWNER,It would be neat if `.transform(pk=None)` converted a primary key table to a rowid table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696488201,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696488201,MDEyOklzc3VlQ29tbWVudDY5NjQ4ODIwMQ==,9599,simonw,2020-09-22T03:21:16Z,2020-09-22T03:21:16Z,OWNER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696485791,MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ==,9599,simonw,2020-09-22T03:10:15Z,2020-09-22T03:10:15Z,OWNER,"Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key?
Since I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this:
```python
table.transform(drop_foreign_keys=[(""author_id"", ""author"", ""id"")])
```
It's a bit crufty but it's such a rare use-case that I think this will be good enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696480925,MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ==,9599,simonw,2020-09-22T02:45:47Z,2020-09-22T02:45:47Z,OWNER,"I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this:
INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs;
So passing `conversions={""name"": ""upper(?)""})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696473559,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696473559,MDEyOklzc3VlQ29tbWVudDY5NjQ3MzU1OQ==,9599,simonw,2020-09-22T02:10:37Z,2020-09-22T02:10:37Z,OWNER,"Maybe something like this:
sqlite-utils transform mydb.db mytable -c age integer --rename age dog_age
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command,
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-696465788,https://api.github.com/repos/simonw/sqlite-utils/issues/163,696465788,MDEyOklzc3VlQ29tbWVudDY5NjQ2NTc4OA==,9599,simonw,2020-09-22T01:33:04Z,2020-09-22T01:33:04Z,OWNER,This would apply to `.transform()` in #114 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696454485,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696454485,MDEyOklzc3VlQ29tbWVudDY5NjQ1NDQ4NQ==,9599,simonw,2020-09-22T00:42:35Z,2020-09-22T00:42:35Z,OWNER,The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696454084,https://api.github.com/repos/simonw/sqlite-utils/issues/162,696454084,MDEyOklzc3VlQ29tbWVudDY5NjQ1NDA4NA==,9599,simonw,2020-09-22T00:40:44Z,2020-09-22T00:40:44Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705995722,A decorator for registering custom SQL functions,
https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696449345,https://api.github.com/repos/simonw/sqlite-utils/issues/162,696449345,MDEyOklzc3VlQ29tbWVudDY5NjQ0OTM0NQ==,9599,simonw,2020-09-22T00:22:46Z,2020-09-22T00:22:46Z,OWNER,Inspired by the idea of adding `conversions=` to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705995722,A decorator for registering custom SQL functions,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696446658,MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA==,9599,simonw,2020-09-22T00:13:55Z,2020-09-22T00:14:21Z,OWNER,"Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.:
```python
table.transform({""age"": int}, conversions={""name"": ""upper(?)""})
```
https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696445766,MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng==,9599,simonw,2020-09-22T00:10:50Z,2020-09-22T00:11:12Z,OWNER,"A less horrible interface might be the following:
```python
# Ensure the 'age' column is not null:
table.transform(not_null={""age""})
# The 'age' column is not null but I don't want it to be:
table.transform(not_null={""age"": False})
```
So if the argument is a set it means ""make sure these are all not null"" - if the argument is a dictionary it means ""set these to be null or not null depending on if their dictionary value is true or false"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444842,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg==,9599,simonw,2020-09-22T00:07:43Z,2020-09-22T00:09:05Z,OWNER,"Syntax challenge: I could use `.transform(defaults={""age"": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it?
I could do this: `.transform(not_not_null={""age""})` - it's a bit gross but it's also kind of funny. I actually like it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444353,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw==,9599,simonw,2020-09-22T00:06:12Z,2020-09-22T00:06:12Z,OWNER,I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443845,MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ==,9599,simonw,2020-09-22T00:04:31Z,2020-09-22T00:04:44Z,OWNER,"Good news: the `.columns` introspection does tell me those things:
```
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> db.create_table(""foo"", {""id"": int, ""name"": str, ""age"": int}, defaults={""age"": 1}, not_null={""name"", ""age""})
>>> db[""foo""]
>>> print(db[""foo""].schema)
CREATE TABLE [foo] (
[id] INTEGER,
[name] TEXT NOT NULL,
[age] INTEGER NOT NULL DEFAULT 1
)
>>> db[""foo""].columns
[Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0),
Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0),
Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443190,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA==,9599,simonw,2020-09-22T00:02:22Z,2020-09-22T00:02:22Z,OWNER,How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443042,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg==,9599,simonw,2020-09-22T00:01:50Z,2020-09-22T00:01:50Z,OWNER,"When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696442621,MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ==,9599,simonw,2020-09-22T00:00:23Z,2020-09-22T00:00:23Z,OWNER,I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696435194,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696435194,MDEyOklzc3VlQ29tbWVudDY5NjQzNTE5NA==,9599,simonw,2020-09-21T23:34:14Z,2020-09-21T23:35:00Z,OWNER,"I think the fiddliest part of the implementation here is code that takes the existing `columns_dict` of the table and the incoming `columns=` and `drop=` and `rename=` parameters and produces the columns dictionary for the new table, ready to be fed to `.create_table()`.
This logic probably also needs to return a structure that can be used to build the `INSERT INTO ... SELECT ... FROM` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434638,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434638,MDEyOklzc3VlQ29tbWVudDY5NjQzNDYzOA==,9599,simonw,2020-09-21T23:32:26Z,2020-09-21T23:32:26Z,OWNER,A test that confirms that this mechanism can turn a `rowid` into a non-rowid table would be good too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434237,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434237,MDEyOklzc3VlQ29tbWVudDY5NjQzNDIzNw==,9599,simonw,2020-09-21T23:31:07Z,2020-09-21T23:31:57Z,OWNER,"Does it make sense to support the `pk=` argument for changing the primary key?
If the user requests a primary key that doesn't make sense I think an integrity error will be raised when the SQL is being executed, which should hopefully cancel the transaction and raise an error. Need to check that this is what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434097,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434097,MDEyOklzc3VlQ29tbWVudDY5NjQzNDA5Nw==,9599,simonw,2020-09-21T23:30:40Z,2020-09-21T23:30:40Z,OWNER,"Since I have a `column_order=None` argument already, maybe I can ignore the order of the columns in that first argument and use that instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433778,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696433778,MDEyOklzc3VlQ29tbWVudDY5NjQzMzc3OA==,9599,simonw,2020-09-21T23:29:39Z,2020-09-21T23:29:39Z,OWNER,"The `columns=` argument is optional - so you can do just a rename operation like so:
```
table.transform(rename={""age"": ""dog_age""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433542,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696433542,MDEyOklzc3VlQ29tbWVudDY5NjQzMzU0Mg==,9599,simonw,2020-09-21T23:28:58Z,2020-09-21T23:28:58Z,OWNER,"If you want to both change the type of a column AND rename it in the same operation, how would you do that? I think like this:
```python
table.transform({""age"": int}, rename={""age"": ""dog_age""})
```
So any rename logic is applied at the end, after the type transformation or re-ordering logic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696432690,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696432690,MDEyOklzc3VlQ29tbWVudDY5NjQzMjY5MA==,9599,simonw,2020-09-21T23:26:32Z,2020-09-21T23:27:38Z,OWNER,"To expand on what that first argument - the `columns` argument - does. Say you have a table like this:
```
id integer
name text
age text
```
Any columns omitted from the `columns=` argument are left alone - they have to be explicitly dropped using `drop=` if you want to drop them.
Any new columns are added (at the end of the table):
```
table.tranform({""size"": float})
```
Any columns that have their type changed will have their type changed:
```
table.tranform({""age"": int})
```
Should I also re-order columns if the order doesn't match? I think so. Open question as to what happens to columns that aren't mentioned at all in the dictionary though - what order should they go in?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696431058,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696431058,MDEyOklzc3VlQ29tbWVudDY5NjQzMTA1OA==,9599,simonw,2020-09-21T23:21:37Z,2020-09-21T23:21:37Z,OWNER,I may need to do something special for `rowid` tables to ensure that the `rowid` values in the transformed table match those from the old table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696430843,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696430843,MDEyOklzc3VlQ29tbWVudDY5NjQzMDg0Mw==,9599,simonw,2020-09-21T23:21:00Z,2020-09-21T23:21:00Z,OWNER,"For FTS tables associated with the table that is being transformed, should I automatically drop the old FTS table and recreate it against the new one or will it just magically continue to work after the table is renamed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423138,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696423138,MDEyOklzc3VlQ29tbWVudDY5NjQyMzEzOA==,9599,simonw,2020-09-21T22:59:17Z,2020-09-21T23:01:06Z,OWNER,I'm going to sketch out a prototype of this new API design in that branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423066,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696423066,MDEyOklzc3VlQ29tbWVudDY5NjQyMzA2Ng==,9599,simonw,2020-09-21T22:59:01Z,2020-09-21T22:59:01Z,OWNER,"I'm rethinking the API design now. Maybe it could look like this:
To change the type of the `author_id` column from `text` to `int`:
```python
books.transform({""author_id"": int})
```
This would leave the existing columns alone, but would change the type of this column.
To rename `author_id` to `author_identifier`:
```python
books.transform(rename={""author_id"": ""author_identifier""})
```
To drop a column:
```python
books.transform(drop=[""author_id""])
```
Since the parameters all operate on columns they don't need to be called `drop_column` and `rename_column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696421240,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696421240,MDEyOklzc3VlQ29tbWVudDY5NjQyMTI0MA==,9599,simonw,2020-09-21T22:53:48Z,2020-09-21T22:53:48Z,OWNER,"I've decided to call this `table.transform()` - I was over-thinking whether people would remember that `.transform()` actually transforms the table, but that's what documentation is for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/datasette/issues/972#issuecomment-696308847,https://api.github.com/repos/simonw/datasette/issues/972,696308847,MDEyOklzc3VlQ29tbWVudDY5NjMwODg0Nw==,9599,simonw,2020-09-21T19:01:25Z,2020-09-21T19:01:25Z,OWNER,I did a bunch of initial work for this in #427.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705840673,Support faceting against arbitrary SQL queries,
https://github.com/simonw/datasette/issues/971#issuecomment-696307922,https://api.github.com/repos/simonw/datasette/issues/971,696307922,MDEyOklzc3VlQ29tbWVudDY5NjMwNzkyMg==,9599,simonw,2020-09-21T18:59:52Z,2020-09-21T19:00:02Z,OWNER,"Given `dbstat` isn't as widely available as I thought I'm going to let people who want to use `dbstat` run their own `select * from dbstat` queries rather than bake support directly into Datasette.
The experience of exploring `dbstat` will improve if I land support for running facets against arbitrary custom SQL queries, which is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646
https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696304108,https://api.github.com/repos/simonw/datasette/issues/971,696304108,MDEyOklzc3VlQ29tbWVudDY5NjMwNDEwOA==,9599,simonw,2020-09-21T18:52:50Z,2020-09-21T18:52:50Z,OWNER,Looks like the `pysqlite3-binary` package doesn't support `dbstat` either.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696302868,https://api.github.com/repos/simonw/datasette/issues/971,696302868,MDEyOklzc3VlQ29tbWVudDY5NjMwMjg2OA==,9599,simonw,2020-09-21T18:50:40Z,2020-09-21T18:50:40Z,OWNER,Easiest way to get this may be to run `create view dbstat_view as select * from dbstat` on databases that support it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696302020,https://api.github.com/repos/simonw/datasette/issues/971,696302020,MDEyOklzc3VlQ29tbWVudDY5NjMwMjAyMA==,9599,simonw,2020-09-21T18:49:09Z,2020-09-21T18:49:09Z,OWNER,... made harder to work on because I apparently don't have the `DBSTAT_VTAB` module on macOS.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696298614,https://api.github.com/repos/simonw/datasette/issues/971,696298614,MDEyOklzc3VlQ29tbWVudDY5NjI5ODYxNA==,9599,simonw,2020-09-21T18:43:07Z,2020-09-21T18:43:07Z,OWNER,"Or, do this:
```sql
SELECT 1 FROM dbstat limit 1;
```
And see if it returns a ""table does not exist"" error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696297930,https://api.github.com/repos/simonw/datasette/issues/971,696297930,MDEyOklzc3VlQ29tbWVudDY5NjI5NzkzMA==,9599,simonw,2020-09-21T18:41:47Z,2020-09-21T18:41:47Z,OWNER,"https://www.sqlite.org/dbstat.html
> The DBSTAT virtual table is an eponymous virtual table, meaning that is not necessary to run CREATE VIRTUAL TABLE to create an instance of the dbstat virtual table before using it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696297601,https://api.github.com/repos/simonw/datasette/issues/971,696297601,MDEyOklzc3VlQ29tbWVudDY5NjI5NzYwMQ==,9599,simonw,2020-09-21T18:41:07Z,2020-09-21T18:41:07Z,OWNER,"How to detect it? Looks like it's visible in SQLite compile time options: https://latest.datasette.io/-/versions
```
""compile_options"": [
""COMPILER=gcc-8.3.0"",
""ENABLE_COLUMN_METADATA"",
""ENABLE_DBSTAT_VTAB"",
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/670#issuecomment-696163452,https://api.github.com/repos/simonw/datasette/issues/670,696163452,MDEyOklzc3VlQ29tbWVudDY5NjE2MzQ1Mg==,652285,snth,2020-09-21T14:46:10Z,2020-09-21T14:46:10Z,NONE,I'm currently using PostgREST to serve OpenAPI APIs off Postgresql databases. I would like to try out datasette once this becomes available on Postgres.,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL,
https://github.com/simonw/datasette/issues/970#issuecomment-695896557,https://api.github.com/repos/simonw/datasette/issues/970,695896557,MDEyOklzc3VlQ29tbWVudDY5NTg5NjU1Nw==,9599,simonw,2020-09-21T04:40:12Z,2020-09-21T04:40:12Z,OWNER,The Python standard library has a module for this: https://docs.python.org/3/library/webbrowser.html,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url",
https://github.com/simonw/datasette/issues/970#issuecomment-695895960,https://api.github.com/repos/simonw/datasette/issues/970,695895960,MDEyOklzc3VlQ29tbWVudDY5NTg5NTk2MA==,9599,simonw,2020-09-21T04:36:45Z,2020-09-21T04:36:45Z,OWNER,I like this. It could work with the `--root` option too and automatically sign you in as the root user.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url",
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879531,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695879531,MDEyOklzc3VlQ29tbWVudDY5NTg3OTUzMQ==,9599,simonw,2020-09-21T02:55:28Z,2020-09-21T02:55:54Z,MEMBER,"Actually for the tie-breaker it should be something like https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+where+%28created+%3E+%3Ap1+or+%28created+%3D+%3Ap1+and+%28%28pk+%3E+%3Ap0%29%29%29%29+order+by+created%2C+pk+limit+11&p0=10&p1=2019-01-16+08%3A00%3A00
```sql
where
(
created > :p1
or (
created = :p1
and ((pk > :p0))
)
)
```
But with `rowid` and `timestamp` in place of `pk` and `created`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879237,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695879237,MDEyOklzc3VlQ29tbWVudDY5NTg3OTIzNw==,9599,simonw,2020-09-21T02:53:29Z,2020-09-21T02:53:29Z,MEMBER,"If previous page ended at `2018-02-11T16:32:53+00:00`:
```sql
select
search_index.rowid,
search_index.type,
search_index.key,
search_index.title,
search_index.category,
search_index.timestamp,
search_index.search_1
from
search_index
where
date(""timestamp"") = '2018-02-11'
and timestamp < '2018-02-11T16:32:53+00:00'
order by
search_index.timestamp desc, rowid
limit 41
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695877627,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,695877627,MDEyOklzc3VlQ29tbWVudDY5NTg3NzYyNw==,9599,simonw,2020-09-21T02:42:29Z,2020-09-21T02:42:29Z,MEMBER,"Fun twist: assuming `timestamp` is always stored as UTC, I need the interface to be timezone aware so I can see e.g. everything from 4th July 2020 in the San Francisco timezone definition of 4th July 2020.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566,Timeline view,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695875274,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695875274,MDEyOklzc3VlQ29tbWVudDY5NTg3NTI3NA==,9599,simonw,2020-09-21T02:28:58Z,2020-09-21T02:28:58Z,MEMBER,Datasette's implementation is complex because it has to support compound primary keys: https://github.com/simonw/datasette/blob/a258339a935d8d29a95940ef1db01e98bb85ae63/datasette/utils/__init__.py#L88-L114 - but that's not something that's needed for dogsheep-beta.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856967,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856967,MDEyOklzc3VlQ29tbWVudDY5NTg1Njk2Nw==,9599,simonw,2020-09-21T00:26:59Z,2020-09-21T00:26:59Z,MEMBER,It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L8-L43,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856398,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856398,MDEyOklzc3VlQ29tbWVudDY5NTg1NjM5OA==,9599,simonw,2020-09-21T00:22:20Z,2020-09-21T00:22:20Z,MEMBER,I'm going to try for keyset pagination sorted by relevance just as a learning exercise.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855723,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695855723,MDEyOklzc3VlQ29tbWVudDY5NTg1NTcyMw==,9599,simonw,2020-09-21T00:16:52Z,2020-09-21T00:17:53Z,MEMBER,"It feels a bit weird to implement keyset pagination against results sorted by `rank` because the ranks could change substantially if the search index gets updated while the user is paginating.
I may just ignore that though. If you want reliable pagination you can get it by sorting by date. Maybe it doesn't even make sense to offer pagination if you sort by relevance?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695855646,MDEyOklzc3VlQ29tbWVudDY5NTg1NTY0Ng==,9599,simonw,2020-09-21T00:16:11Z,2020-09-21T00:16:11Z,MEMBER,"Should I do this with offset/limit or should I do proper keyset pagination?
I think keyset because then it will work well for the full search interface with no filters or search string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination,
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695851036,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,695851036,MDEyOklzc3VlQ29tbWVudDY5NTg1MTAzNg==,9599,simonw,2020-09-20T23:34:57Z,2020-09-20T23:34:57Z,MEMBER,Really basic starting point is to add facet by date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566,Timeline view,
https://github.com/simonw/sqlite-utils/issues/160#issuecomment-695839557,https://api.github.com/repos/simonw/sqlite-utils/issues/160,695839557,MDEyOklzc3VlQ29tbWVudDY5NTgzOTU1Nw==,9599,simonw,2020-09-20T21:37:03Z,2020-09-20T21:37:03Z,OWNER,"Should this support `ignore=True` as well? I'm tempted to skip that - I think `replace=True` is more useful because it implies ""ignore if the options are already the same, but replace if they are different"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705190723,"table.enable_fts(..., replace=True)",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-695698227,https://api.github.com/repos/simonw/sqlite-utils/issues/42,695698227,MDEyOklzc3VlQ29tbWVudDY5NTY5ODIyNw==,9599,simonw,2020-09-20T04:27:26Z,2020-09-20T04:28:26Z,OWNER,This is going to need #114 (the `transform_table()` method) in order to convert string columns into integer foreign key columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776,https://api.github.com/repos/simonw/sqlite-utils/issues/68,695695776,MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng==,9599,simonw,2020-09-20T04:25:47Z,2020-09-20T04:25:47Z,OWNER,This is a dupe of #130 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531583658,Add support for porter stemming in FTS,
https://github.com/simonw/datasette/issues/943#issuecomment-695133768,https://api.github.com/repos/simonw/datasette/issues/943,695133768,MDEyOklzc3VlQ29tbWVudDY5NTEzMzc2OA==,9599,simonw,2020-09-19T00:06:56Z,2020-09-19T00:07:35Z,OWNER,"[dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) could do with this too. It currently [makes a call](https://github.com/dogsheep/dogsheep-beta/blob/ab36101bdae69b11af7c6bd7edee838d052e6ecf/dogsheep_beta/__init__.py#L216-L225) to `TableView` in a similar way to `datasette-graphql` in order to calculate facets.
`dogsheep-beta` would benefit with a mechanism for changing the facet timeout setting during that call (as would `datasette-graphql`, see the [DatasetteSpecialConfig mechanism](https://github.com/simonw/datasette-graphql/blob/f9dc5c518b7cdc94b93873ef20069a7ea2882a95/datasette_graphql/utils.py#L516-L519) it uses).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/dogsheep/dogsheep-beta/issues/15#issuecomment-695124698,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15,695124698,MDEyOklzc3VlQ29tbWVudDY5NTEyNDY5OA==,9599,simonw,2020-09-18T23:17:38Z,2020-09-18T23:17:38Z,MEMBER,This can be part of the demo instance in #6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694136490,Add a bunch of config examples,
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-695113871,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,695113871,MDEyOklzc3VlQ29tbWVudDY5NTExMzg3MQ==,9599,simonw,2020-09-18T22:30:17Z,2020-09-18T22:30:17Z,MEMBER,"I think I know what's going on here:
https://github.com/dogsheep/dogsheep-beta/blob/0f1b951c5131d16f3c8559a8e4d79ed5c559e3cb/dogsheep_beta/__init__.py#L166-L171
This is a logic bug - the `compiled` variable could be the template from the previous loop!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695109140,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25,695109140,MDEyOklzc3VlQ29tbWVudDY5NTEwOTE0MA==,9599,simonw,2020-09-18T22:12:20Z,2020-09-18T22:12:20Z,MEMBER,Documented here: https://github.com/dogsheep/dogsheep-beta/blob/534fc9689227eba70e69a45da0cee5820bbda9e1/README.md#datasette-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",704685890,template_debug mechanism,
https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695108895,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25,695108895,MDEyOklzc3VlQ29tbWVudDY5NTEwODg5NQ==,9599,simonw,2020-09-18T22:11:32Z,2020-09-18T22:11:32Z,MEMBER,"I'm going to make this a new plugin configuration setting, `template_debug`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",704685890,template_debug mechanism,
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694557425,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694557425,MDEyOklzc3VlQ29tbWVudDY5NDU1NzQyNQ==,9599,simonw,2020-09-17T23:41:01Z,2020-09-17T23:41:01Z,MEMBER,I removed all of the `json.loads()` calls and I'm still getting that `Undefined` error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694554584,MDEyOklzc3VlQ29tbWVudDY5NDU1NDU4NA==,9599,simonw,2020-09-17T23:31:25Z,2020-09-17T23:31:25Z,MEMBER,"I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694553579,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694553579,MDEyOklzc3VlQ29tbWVudDY5NDU1MzU3OQ==,9599,simonw,2020-09-17T23:28:37Z,2020-09-17T23:28:37Z,MEMBER,"More investigation in pdb:
```
(dogsheep-beta) dogsheep-beta % datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb
> /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads()
-> raise TypeError(f'the JSON object must be str, bytes or bytearray, '
(Pdb) list
336 if s.startswith('\ufeff'):
337 raise JSONDecodeError(""Unexpected UTF-8 BOM (decode using utf-8-sig)"",
338 s, 0)
339 else:
340 if not isinstance(s, (bytes, bytearray)):
341 -> raise TypeError(f'the JSON object must be str, bytes or bytearray, '
342 f'not {s.__class__.__name__}')
343 s = s.decode(detect_encoding(s), 'surrogatepass')
344
345 if ""encoding"" in kw:
346 import warnings
(Pdb) bytes
(Pdb) locals()['s']
Undefined
(Pdb) type(locals()['s'])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552681,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694552681,MDEyOklzc3VlQ29tbWVudDY5NDU1MjY4MQ==,9599,simonw,2020-09-17T23:25:54Z,2020-09-17T23:25:54Z,MEMBER,"This is the template fragment it's rendering:
```html+jinja
{% if display.media_urls and json.loads(display.media_urls) %}
{% for url in json.loads(display.media_urls) %}
{% endfor %}
{% endif %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552393,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694552393,MDEyOklzc3VlQ29tbWVudDY5NDU1MjM5Mw==,9599,simonw,2020-09-17T23:25:01Z,2020-09-17T23:25:17Z,MEMBER,"Ran `locals()` In the debugger:
`{'range': , 'dict': , 'lipsum': , 'cycler': , 'joiner': , 'namespace': , 'rank': -9.383801886431414, 'rowid': 14297, 'type': 'twitter.db/tweets', 'key': '312658917933076480', 'title': 'Tweet by @chrisstreeter', 'category': 2, 'timestamp': '2013-03-15T20:17:49+00:00', 'search_1': '@simonw are you at pycon? Would love to meet you.', 'display': {'avatar_url': 'https://pbs.twimg.com/profile_images/806275088597204993/38yLHfJi_normal.jpg', 'user_name': 'Chris Streeter', 'screen_name': 'chrisstreeter', 'followers_count': 280, 'tweet_id': 312658917933076480, 'created_at': '2013-03-15T20:17:49+00:00', 'full_text': '@simonw are you at pycon? Would love to meet you.', 'media_urls_2': '[]', 'media_urls': '[]'}, 'json': }`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694551646,MDEyOklzc3VlQ29tbWVudDY5NDU1MTY0Ng==,9599,simonw,2020-09-17T23:22:48Z,2020-09-17T23:22:48Z,MEMBER,"Looks like its happening in a Jinja fragment template for one of the results:
```
/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/__init__.py(169)process_results()
-> output = compiled.render({**result, **{""json"": json}})
/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/asyncsupport.py(71)render()
-> return original_render(self, *args, **kwargs)
/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(1090)render()
-> self.environment.handle_exception()
/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(832)handle_exception()
-> reraise(*rewrite_traceback_stack(source=source))
/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/_compat.py(28)reraise()
-> raise value.with_traceback(tb)
(5)top-level template code()
> /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads()
-> raise TypeError(f'the JSON object must be str, bytes or bytearray, '
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551406,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694551406,MDEyOklzc3VlQ29tbWVudDY5NDU1MTQwNg==,9599,simonw,2020-09-17T23:22:07Z,2020-09-17T23:22:07Z,MEMBER,"Neat, I can debug this with the new `--pdb` option:
datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'",
https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-694548909,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,694548909,MDEyOklzc3VlQ29tbWVudDY5NDU0ODkwOQ==,9599,simonw,2020-09-17T23:15:09Z,2020-09-17T23:15:09Z,MEMBER,"I have sort by date now, #21.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566,Timeline view,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693794700,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693794700,MDEyOklzc3VlQ29tbWVudDY5Mzc5NDcwMA==,9599,simonw,2020-09-17T04:02:39Z,2020-09-17T04:02:39Z,MEMBER,It would be useful if you could pass an `--accept` option to this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693789129,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693789129,MDEyOklzc3VlQ29tbWVudDY5Mzc4OTEyOQ==,9599,simonw,2020-09-17T03:40:01Z,2020-09-17T03:40:01Z,MEMBER,"Bug with endpoints that return dictionaries rather than arrays:
```
github-to-sqlite get /users/simonw
[
""login"",
""id"",
""node_id"",
""avatar_url"",
""gravatar_id"",
""url"",
""html_url"",
""followers_url"",
""following_url"",
""gists_url"",
""starred_url"",
""subscriptions_url"",
""organizations_url"",
""repos_url"",
""events_url"",
""received_events_url"",
""type"",
""site_admin"",
""name"",
""company"",
""blog"",
""location"",
""email"",
""hireable"",
""bio"",
""twitter_username"",
""public_repos"",
""public_gists"",
""followers"",
""following"",
""created_at"",
""updated_at""
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788387,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693788387,MDEyOklzc3VlQ29tbWVudDY5Mzc4ODM4Nw==,9599,simonw,2020-09-17T03:36:47Z,2020-09-17T03:36:58Z,MEMBER,"Fun demo of the `--nl` option:
github-to-sqlite get /users/simonw/repos --paginate --nl | sqlite-utils insert simonw.db repos - --nl
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693788032,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693788032,MDEyOklzc3VlQ29tbWVudDY5Mzc4ODAzMg==,9599,simonw,2020-09-17T03:35:22Z,2020-09-17T03:35:22Z,MEMBER,Documentation: https://github.com/dogsheep/github-to-sqlite/blob/b02bf135485c0a7a3768868967f45a6b5e515289/README.md#making-authenticated-api-calls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693775622,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693775622,MDEyOklzc3VlQ29tbWVudDY5Mzc3NTYyMg==,9599,simonw,2020-09-17T02:48:34Z,2020-09-17T02:48:34Z,MEMBER,I'd like a `--paginate` option that does the same thing as https://github.com/simonw/paginate-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/simonw/datasette/issues/507#issuecomment-693775390,https://api.github.com/repos/simonw/datasette/issues/507,693775390,MDEyOklzc3VlQ29tbWVudDY5Mzc3NTM5MA==,9599,simonw,2020-09-17T02:47:35Z,2020-09-17T02:47:35Z,OWNER,"I have a pattern for creating screenshots using Puppeteer running in a GitHub Action now, see https://simonwillison.net/2020/Sep/3/weeknotes-airtable-screenshots-dogsheep/#weeknotes-2020-09-03-social-media-cards-tils","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455852801,Every datasette plugin on the ecosystem page should have a screenshot,
https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-693773191,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,693773191,MDEyOklzc3VlQ29tbWVudDY5Mzc3MzE5MQ==,9599,simonw,2020-09-17T02:39:26Z,2020-09-17T02:39:26Z,MEMBER,I'm going to start with `github-to-sqlite get` and `github-to-sqlite post` - I may add `put` and suchlike later on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls,
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693695177,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693695177,MDEyOklzc3VlQ29tbWVudDY5MzY5NTE3Nw==,9599,simonw,2020-09-16T22:17:53Z,2020-09-16T22:17:53Z,OWNER,@spdkils can you share a minimal code example that exhibits the behavior you're seeing?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693694968,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693694968,MDEyOklzc3VlQ29tbWVudDY5MzY5NDk2OA==,9599,simonw,2020-09-16T22:17:19Z,2020-09-16T22:17:19Z,OWNER,"That's strange... this test here doesn't manually commit a transaction and passes:
https://github.com/simonw/sqlite-utils/blob/7805d53bcf11199bd1f2b07e05ae90151f9d0eb0/tests/test_delete.py#L17-L23","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693694343,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693694343,MDEyOklzc3VlQ29tbWVudDY5MzY5NDM0Mw==,9599,simonw,2020-09-16T22:15:39Z,2020-09-16T22:15:39Z,OWNER,"Independent of the transaction changes in #121 I may be able to check `self.conn.in_transaction` to see if a transaction is active and, if one is NOT active, execute the delete inside of one. https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.in_transaction","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693589321,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693589321,MDEyOklzc3VlQ29tbWVudDY5MzU4OTMyMQ==,9599,simonw,2020-09-16T18:41:42Z,2020-09-16T18:41:42Z,OWNER,Yeah I'm going to class this as a bug - that's definitely confusing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693486183,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693486183,MDEyOklzc3VlQ29tbWVudDY5MzQ4NjE4Mw==,11712349,spdkils,2020-09-16T15:34:13Z,2020-09-16T15:34:13Z,NONE,"I appreciate the response, it's just unexpected.
If I insert, it commits, if I update it commits, if I upsert it commits... if I delete.. it doesn't???
Confused me...
I did just db commit it... But it's confusing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/pull/158#issuecomment-693199392,https://api.github.com/repos/simonw/sqlite-utils/issues/158,693199392,MDEyOklzc3VlQ29tbWVudDY5MzE5OTM5Mg==,9599,simonw,2020-09-16T06:21:29Z,2020-09-16T06:21:29Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697203800,Fix accidental mega long line in docs,
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693199049,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693199049,MDEyOklzc3VlQ29tbWVudDY5MzE5OTA0OQ==,9599,simonw,2020-09-16T06:20:26Z,2020-09-16T06:20:26Z,OWNER,"See #121 - I need to think harder about how this all interacts with transactions.
You can do this:
```python
with db.conn:
db[""mytable""].delete_where()
```
But that should be documented and maybe rethought.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/datasette/issues/943#issuecomment-693010291,https://api.github.com/repos/simonw/datasette/issues/943,693010291,MDEyOklzc3VlQ29tbWVudDY5MzAxMDI5MQ==,9599,simonw,2020-09-15T22:20:55Z,2020-09-15T22:20:55Z,OWNER,"Should I instantiate a single `Client` and reuse it for all internal requests, or can I instantiate a new `Client` for each request?
https://www.python-httpx.org/advanced/#why-use-a-client says that the main benefit of a Client instance is HTTP connection pooling - which isn't an issue for these internal requests since they won't be using the HTTP protocol at all, they'll be calling the ASGI application directly.
So I'm leaning towards instantiating a fresh client for every internal request. I'll run a microbenchmark to check that this doesn't have any unpleasant performance implications.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693008540,https://api.github.com/repos/simonw/datasette/issues/943,693008540,MDEyOklzc3VlQ29tbWVudDY5MzAwODU0MA==,9599,simonw,2020-09-15T22:16:07Z,2020-09-15T22:16:07Z,OWNER,"I think I can use `async with httpx.AsyncClient(base_url=""http://localhost/"") as client:` to ensure I don't need to use `http://localhost/` on every call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693007512,https://api.github.com/repos/simonw/datasette/issues/943,693007512,MDEyOklzc3VlQ29tbWVudDY5MzAwNzUxMg==,9599,simonw,2020-09-15T22:13:30Z,2020-09-15T22:13:30Z,OWNER,"I could solve streaming using something like this:
```python
async with datasette.stream(""GET"", ""/fixtures/compound_three_primary_keys.csv?_stream=on&_size=max"") as response:
async for chunk in response.aiter_bytes():
print(chunk)
```
Which would be a wrapper around `AsyncClient.stream(method, url, ...)` from https://www.python-httpx.org/async/#streaming-responses","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693005033,https://api.github.com/repos/simonw/datasette/issues/943,693005033,MDEyOklzc3VlQ29tbWVudDY5MzAwNTAzMw==,9599,simonw,2020-09-15T22:06:58Z,2020-09-15T22:10:58Z,OWNER,"What if `datasette.get()` was an alias for `httpx.get()`, pre-configured to route to the correct application? And with some sugar that added `http://localhost/` to the beginning of the path if it was missing?
This would make `httpx` a dependency of core Datasette, which I think is OK.
It would also solve the return type problem: I would return whatever `httpx` returns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004770,https://api.github.com/repos/simonw/datasette/issues/943,693004770,MDEyOklzc3VlQ29tbWVudDY5MzAwNDc3MA==,9599,simonw,2020-09-15T22:06:13Z,2020-09-15T22:06:13Z,OWNER,I'm tempted to create a `await datasette.request()` method which can take any HTTP verb - then have `datasette.get()` and `datasette.post()` as thin wrappers around it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004572,https://api.github.com/repos/simonw/datasette/issues/943,693004572,MDEyOklzc3VlQ29tbWVudDY5MzAwNDU3Mg==,9599,simonw,2020-09-15T22:05:39Z,2020-09-15T22:05:39Z,OWNER,"Maybe these methods become the way most Datasette tests are written, replacing the existing `TestClient` mechanism?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004296,https://api.github.com/repos/simonw/datasette/issues/943,693004296,MDEyOklzc3VlQ29tbWVudDY5MzAwNDI5Ng==,9599,simonw,2020-09-15T22:04:54Z,2020-09-15T22:04:54Z,OWNER,"So what should I do about streaming responses?
I could deliberately ignore them - through an exception if you attempt to run `await datasette.get(...)` against a streaming URL.
I could load the entire response into memory and return it as a wrapped object.
I could support some kind of asynchronous iterator mechanism. This would be pretty elegant if I could decide the right syntax for it - it would allow plugins to take advantage of other internal URLs that return streaming content without needing to load that content entirely into memory in order to process it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693003652,https://api.github.com/repos/simonw/datasette/issues/943,693003652,MDEyOklzc3VlQ29tbWVudDY5MzAwMzY1Mg==,9599,simonw,2020-09-15T22:03:08Z,2020-09-15T22:03:08Z,OWNER,"I'm not going to mess around with formats - you'll get back the exact response that a web client would receive.
Question: what should the response object look like? e.g. if you do:
response = await datasette.get(""/db/table.json"")
What should `response` be?
I could reuse the Datasette `Response` class from `datasette.utils.asgi`. This would work well for regular responses which just have a status code, some headers and a response body. It wouldn't be great for streaming responses though such as you get back from `?_stream=1` CSV exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/891#issuecomment-693001937,https://api.github.com/repos/simonw/datasette/issues/891,693001937,MDEyOklzc3VlQ29tbWVudDY5MzAwMTkzNw==,9599,simonw,2020-09-15T21:58:56Z,2020-09-15T21:58:56Z,OWNER,"Here's what that looks like:
```
Traceback (most recent call last):
File ""/Users/simon/Dropbox/Development/datasette/plugins/sql_error.py"", line 5, in oh_no_error
return 100 / 0
ZeroDivisionError: division by zero
ERROR: conn=, sql = 'select oh_no_error()', params = {}: user-defined function raised exception
INFO: 127.0.0.1:54066 - ""GET /data?sql=select+oh_no_error%28%29 HTTP/1.1"" 400 Bad Request
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-693000522,https://api.github.com/repos/simonw/datasette/issues/891,693000522,MDEyOklzc3VlQ29tbWVudDY5MzAwMDUyMg==,9599,simonw,2020-09-15T21:55:11Z,2020-09-15T21:55:11Z,OWNER,I'm going to turn this on. If people complain about it I can turn it off again (or make it a configuration setting).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-692999893,https://api.github.com/repos/simonw/datasette/issues/891,692999893,MDEyOklzc3VlQ29tbWVudDY5Mjk5OTg5Mw==,9599,simonw,2020-09-15T21:53:36Z,2020-09-15T21:53:36Z,OWNER,"Here's the commit (from 15 years ago) where `enable_callback_tracebacks` was first added: https://github.com/ghaering/pysqlite/commit/1e8bd36be93b7d7425910642b72e4152c77b0dfd
> - Exceptions in callbacks lead to the query being aborted now instead of silently leading to generating values.
> - Exceptions in callbacks can be echoed to stderr if you call the module level function enable_callback_tracebacks: enable_callback_tracebacks(1).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-692998061,https://api.github.com/repos/simonw/datasette/issues/891,692998061,MDEyOklzc3VlQ29tbWVudDY5Mjk5ODA2MQ==,9599,simonw,2020-09-15T21:49:03Z,2020-09-15T21:49:03Z,OWNER,"I've been trying to figure out why this is an optional setting that defaults to off.
I think it's because it writes directly to `stderr`, so the maintainers of `sqlite3` reasonably decided that people should be able to opt in to that rather than having weird stuff show up on `stderr` that they weren't expecting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-692968792,https://api.github.com/repos/simonw/datasette/issues/891,692968792,MDEyOklzc3VlQ29tbWVudDY5Mjk2ODc5Mg==,9599,simonw,2020-09-15T20:44:15Z,2020-09-15T20:44:15Z,OWNER,"https://github.com/peter-wangxu/persist-queue/issues/74 warns that this might not work with PyPy.
I could solve that with:
```python
if hasattr(sqlite3, ""enable_callback_tracebacks""):
sqlite3.enable_callback_tracebacks(True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/877#issuecomment-692967733,https://api.github.com/repos/simonw/datasette/issues/877,692967733,MDEyOklzc3VlQ29tbWVudDY5Mjk2NzczMw==,9599,simonw,2020-09-15T20:42:04Z,2020-09-15T20:42:04Z,OWNER,"I'm not going to drop CSRF protection - it's still needed for older browsers - but I have relaxed the circumstances under which it is applied. It only applies to requests that include cookies for example, so API clients that don't send cookies don't need to worry about it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/889#issuecomment-692967123,https://api.github.com/repos/simonw/datasette/issues/889,692967123,MDEyOklzc3VlQ29tbWVudDY5Mjk2NzEyMw==,9599,simonw,2020-09-15T20:40:52Z,2020-09-15T20:40:52Z,OWNER,Thanks - I've fixed this in `datasette-media` and the other plugins that use that hook now I think.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649907676,asgi_wrapper plugin hook is crashing at startup,
https://github.com/simonw/datasette/issues/888#issuecomment-692966625,https://api.github.com/repos/simonw/datasette/issues/888,692966625,MDEyOklzc3VlQ29tbWVudDY5Mjk2NjYyNQ==,9599,simonw,2020-09-15T20:39:49Z,2020-09-15T20:39:49Z,OWNER,"Thanks, I've fixed that now. It only affected the GitHub release notes - the ones at https://docs.datasette.io/en/stable/changelog.html#v0-45 had the correct links.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649702801,URLs in release notes point to 127.0.0.1,
https://github.com/simonw/datasette/issues/634#issuecomment-692965761,https://api.github.com/repos/simonw/datasette/issues/634,692965761,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTc2MQ==,9599,simonw,2020-09-15T20:37:58Z,2020-09-15T20:37:58Z,OWNER,I fixed this in 5e0b72247ecab4ce0fcec599b77a83d73a480872,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",522352520,Don't run tests twice when releasing a tag,
https://github.com/simonw/datasette/issues/849#issuecomment-692965391,https://api.github.com/repos/simonw/datasette/issues/849,692965391,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTM5MQ==,9599,simonw,2020-09-15T20:37:14Z,2020-09-15T20:37:14Z,OWNER,I've been running on `main` for a while now with no issues.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/956#issuecomment-692965022,https://api.github.com/repos/simonw/datasette/issues/956,692965022,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTAyMg==,9599,simonw,2020-09-15T20:36:34Z,2020-09-15T20:36:34Z,OWNER,https://hub.docker.com/r/datasetteproject/datasette/tags - 0.49.1 was successfully pushed to Docker Hub by https://github.com/simonw/datasette/runs/1119815175?check_suite_focus=true,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-692955850,https://api.github.com/repos/simonw/datasette/issues/956,692955850,MDEyOklzc3VlQ29tbWVudDY5Mjk1NTg1MA==,9599,simonw,2020-09-15T20:17:49Z,2020-09-15T20:17:49Z,OWNER,I think I've fixed this with recent changes I made as part of #941 - but I won't know until I release the next version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/946#issuecomment-692955379,https://api.github.com/repos/simonw/datasette/issues/946,692955379,MDEyOklzc3VlQ29tbWVudDY5Mjk1NTM3OQ==,9599,simonw,2020-09-15T20:16:50Z,2020-09-15T20:16:50Z,OWNER,Can't reproduce this bug now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",682184050,Exception in tracing code,
https://github.com/simonw/datasette/issues/492#issuecomment-692953174,https://api.github.com/repos/simonw/datasette/issues/492,692953174,MDEyOklzc3VlQ29tbWVudDY5Mjk1MzE3NA==,9599,simonw,2020-09-15T20:12:29Z,2020-09-15T20:12:29Z,OWNER,I fixed this in ea340cf320a2566d24517fb4a0c9852c5059e771 for #963 (a duplicate of this issue).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449854604,Facets not correctly persisted in hidden form fields,
https://github.com/simonw/datasette/issues/967#issuecomment-692951144,https://api.github.com/repos/simonw/datasette/issues/967,692951144,MDEyOklzc3VlQ29tbWVudDY5Mjk1MTE0NA==,9599,simonw,2020-09-15T20:08:12Z,2020-09-15T20:08:12Z,OWNER,I think the easiest fix is for me to ensure that calls to `__len__` on the `MagicParameters` class always return at least 1.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692946616,https://api.github.com/repos/simonw/datasette/issues/967,692946616,MDEyOklzc3VlQ29tbWVudDY5Mjk0NjYxNg==,9599,simonw,2020-09-15T19:59:21Z,2020-09-15T19:59:21Z,OWNER,"I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python.
Might be possible to do that using `ctypes` - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html
```python
param_count = lib.sqlite3_bind_parameter_count(self.statement)
for idx in range(1, param_count + 1):
param_name = lib.sqlite3_bind_parameter_name(self.statement,
idx)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692945504,https://api.github.com/repos/simonw/datasette/issues/967,692945504,MDEyOklzc3VlQ29tbWVudDY5Mjk0NTUwNA==,9599,simonw,2020-09-15T19:57:10Z,2020-09-15T19:57:10Z,OWNER,"So the problem actually occurs when the `MagicParameters` class wraps an empty dictionary.
Relevant code:
https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236
And:
https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383
I'm passing a special magic parameters dictionary for the Python `sqlite3` module to look up parameters in. When that dictionary is `{}` a `__len__` check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters.
I tracked down the relevant C code:
https://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237
```c
Py_BEGIN_ALLOW_THREADS
num_params_needed = sqlite3_bind_parameter_count(self->st);
Py_END_ALLOW_THREADS
if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) {
/* parameters passed as sequence */
if (PyTuple_CheckExact(parameters)) {
num_params = PyTuple_GET_SIZE(parameters);
} else if (PyList_CheckExact(parameters)) {
num_params = PyList_GET_SIZE(parameters);
} else {
num_params = PySequence_Size(parameters);
}
if (num_params != num_params_needed) {
PyErr_Format(pysqlite_ProgrammingError,
""Incorrect number of bindings supplied. The current ""
""statement uses %d, and there are %zd supplied."",
num_params_needed, num_params);
return;
}
```
It looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692940375,https://api.github.com/repos/simonw/datasette/issues/967,692940375,MDEyOklzc3VlQ29tbWVudDY5Mjk0MDM3NQ==,9599,simonw,2020-09-15T19:47:09Z,2020-09-15T19:47:09Z,OWNER,"Yes! The tests all pass if I update the test function to do this:
```python
response = magic_parameters_client.post(
""/data/runme_post{}"".format(qs),
{""ignore_me"": ""1""},
csrftoken_from=use_csrf or None,
allow_redirects=False,
)
```
So the bug only occurs if the POST body is completely empty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692938935,https://api.github.com/repos/simonw/datasette/issues/967,692938935,MDEyOklzc3VlQ29tbWVudDY5MjkzODkzNQ==,9599,simonw,2020-09-15T19:44:21Z,2020-09-15T19:44:41Z,OWNER,"While I'm running the above test, in the rounds that work the `receive()` awaitable returns `{'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}`
In the rounds that fails it returns `{'type': 'http.request'}`
So it looks like the `csrftoken_from=True` parameter may be helping just by ensuring the `body` key is present and not missing. I wonder if it would work if a body of `b''` was present there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692937150,https://api.github.com/repos/simonw/datasette/issues/967,692937150,MDEyOklzc3VlQ29tbWVudDY5MjkzNzE1MA==,9599,simonw,2020-09-15T19:42:57Z,2020-09-15T19:42:57Z,OWNER,"New (failing) test:
```python
@pytest.mark.parametrize(""use_csrf"", [True, False])
@pytest.mark.parametrize(""return_json"", [True, False])
def test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_json):
magic_parameters_client.ds._metadata[""databases""][""data""][""queries""][""runme_post""][
""sql""
] = ""insert into logs (line) values (:_header_host)""
qs = """"
if return_json:
qs = ""?_json=1""
response = magic_parameters_client.post(
""/data/runme_post{}"".format(qs),
{},
csrftoken_from=use_csrf or None,
allow_redirects=False,
)
if return_json:
assert response.status == 200
assert response.json[""ok""], response.json
else:
assert response.status == 302
messages = magic_parameters_client.ds.unsign(
response.cookies[""ds_messages""], ""messages""
)
assert [[""Query executed, 1 row affected"", 1]] == messages
post_actual = magic_parameters_client.get(
""/data/logs.json?_sort_desc=rowid&_shape=array""
).json[0][""line""]
assert post_actual == ""localhost""
```
It passes twice, fails twice - failures are for the ones where `use_csrf` is `False`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692927867,https://api.github.com/repos/simonw/datasette/issues/967,692927867,MDEyOklzc3VlQ29tbWVudDY5MjkyNzg2Nw==,9599,simonw,2020-09-15T19:25:23Z,2020-09-15T19:25:23Z,OWNER,Hunch: I think the `asgi-csrf` middleware may be consuming the request body and failing to restore it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692835066,https://api.github.com/repos/simonw/datasette/issues/967,692835066,MDEyOklzc3VlQ29tbWVudDY5MjgzNTA2Ng==,9599,simonw,2020-09-15T16:40:12Z,2020-09-15T16:40:12Z,OWNER,Is the bug here that magic parameters are incompatible with CSRF-exempt requests (e.g. request with no cookies)?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692834670,https://api.github.com/repos/simonw/datasette/issues/967,692834670,MDEyOklzc3VlQ29tbWVudDY5MjgzNDY3MA==,9599,simonw,2020-09-15T16:39:29Z,2020-09-15T16:39:29Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L222-L236
This issue may not be about `_json=1` interacting with magic parameters after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692834064,https://api.github.com/repos/simonw/datasette/issues/967,692834064,MDEyOklzc3VlQ29tbWVudDY5MjgzNDA2NA==,9599,simonw,2020-09-15T16:38:21Z,2020-09-15T16:38:21Z,OWNER,So the mystery here is why does omitting `csrftoken_from=True` break the `MagicParameters` mechanism?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692832113,https://api.github.com/repos/simonw/datasette/issues/967,692832113,MDEyOklzc3VlQ29tbWVudDY5MjgzMjExMw==,9599,simonw,2020-09-15T16:34:53Z,2020-09-15T16:37:43Z,OWNER,"This is so weird. In the test I wrote for this the following passed:
response = magic_parameters_client.post(""/data/runme_post?_json=1"", {}, csrftoken_from=True)
But without the `csrftoken_from=True` parameter it failed with the bindings error:
response = magic_parameters_client.post(""/data/runme_post?_json=1"", {})
Here's the test I wrote:
```python
def test_magic_parameters_json_body(magic_parameters_client):
magic_parameters_client.ds._metadata[""databases""][""data""][""queries""][""runme_post""][
""sql""
] = ""insert into logs (line) values (:_header_host)""
response = magic_parameters_client.post(""/data/runme_post?_json=1"", {}, csrftoken_from=True)
assert response.status == 200
assert response.json[""ok""], response.json
post_actual = magic_parameters_client.get(
""/data/logs.json?_sort_desc=rowid&_shape=array""
).json[0][""line""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/940#issuecomment-692340275,https://api.github.com/repos/simonw/datasette/issues/940,692340275,MDEyOklzc3VlQ29tbWVudDY5MjM0MDI3NQ==,9599,simonw,2020-09-14T22:09:35Z,2020-09-14T22:09:35Z,OWNER,I'm going to cross my fingers and hope that this works - I don't want to leave this issue open until Datasette 0.50.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692339645,https://api.github.com/repos/simonw/datasette/issues/940,692339645,MDEyOklzc3VlQ29tbWVudDY5MjMzOTY0NQ==,9599,simonw,2020-09-14T22:07:58Z,2020-09-14T22:07:58Z,OWNER,"I shipped the Docker build manually by running the following in a tmate session:
docker login
# Typed my username and password interactively
export REPO=datasetteproject/datasette
docker build -f Dockerfile -t $REPO:0.49 .
docker tag $REPO:0.49 $REPO:latest
docker push $REPO
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692337397,https://api.github.com/repos/simonw/datasette/issues/940,692337397,MDEyOklzc3VlQ29tbWVudDY5MjMzNzM5Nw==,9599,simonw,2020-09-14T22:01:56Z,2020-09-14T22:01:56Z,OWNER,"I'm going to switch to using this logic to decide if I should ship to Docker: https://github.community/t/release-prerelease-action-triggers/17275/2
if: ""!github.event.release.prerelease""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692336564,https://api.github.com/repos/simonw/datasette/issues/940,692336564,MDEyOklzc3VlQ29tbWVudDY5MjMzNjU2NA==,9599,simonw,2020-09-14T21:59:40Z,2020-09-14T21:59:40Z,OWNER,Using https://github.com/marketplace/actions/debugging-with-tmate to manually submit a new build from within an interactive GitHub Actions session.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692332430,https://api.github.com/repos/simonw/datasette/issues/940,692332430,MDEyOklzc3VlQ29tbWVudDY5MjMzMjQzMA==,9599,simonw,2020-09-14T21:48:59Z,2020-09-14T21:48:59Z,OWNER,"So now I've released Datasette 0.49 but failed to push a new Docker image. This is bad, and I need to fix it.
I'd like to push to Docker from GitHub Actions, so I think I'm going to create a one-off workflow task for doing that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692331919,https://api.github.com/repos/simonw/datasette/issues/940,692331919,MDEyOklzc3VlQ29tbWVudDY5MjMzMTkxOQ==,9599,simonw,2020-09-14T21:47:39Z,2020-09-14T21:47:39Z,OWNER,"I bet that's because the `github.ref` actually looks like this:
`${GITHUB_REF#refs/tags/}`
And the `refs/tags/` part has an `a` in it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692331349,https://api.github.com/repos/simonw/datasette/issues/940,692331349,MDEyOklzc3VlQ29tbWVudDY5MjMzMTM0OQ==,9599,simonw,2020-09-14T21:46:11Z,2020-09-14T21:46:11Z,OWNER,"Just release Datasette 0.49 - which shipped to PyPI just fine but skipped the Docker step for some reason!
https://github.com/simonw/datasette/runs/1114585275?check_suite_focus=true
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/880#issuecomment-692324230,https://api.github.com/repos/simonw/datasette/issues/880,692324230,MDEyOklzc3VlQ29tbWVudDY5MjMyNDIzMA==,9599,simonw,2020-09-14T21:28:15Z,2020-09-14T21:28:21Z,OWNER,Documentation here: https://docs.datasette.io/en/latest/sql_queries.html#json-api-for-writable-canned-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-692299770,https://api.github.com/repos/simonw/datasette/issues/880,692299770,MDEyOklzc3VlQ29tbWVudDY5MjI5OTc3MA==,9599,simonw,2020-09-14T20:36:40Z,2020-09-14T20:36:40Z,OWNER,"The JSON response will look like this:
```json
{
""ok"": true,
""message"": ""A message"",
""redirect"": ""/blah""
}
```
`""ok""` will be `true` if everything went right and `false` if there was an error.
The `""message""` and `""redirect""` will be whatever was configured using the on_success_message - the message shown `on_success_message`, `on_success_redirect`, `on_error_message` and `on_error_redirect` settings, see https://docs.datasette.io/en/stable/sql_queries.html#writable-canned-queries","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-692298011,https://api.github.com/repos/simonw/datasette/issues/880,692298011,MDEyOklzc3VlQ29tbWVudDY5MjI5ODAxMQ==,9599,simonw,2020-09-14T20:33:13Z,2020-09-14T20:33:13Z,OWNER,"I'm going to support several ways of indicating that you would like a JSON response instead of getting a HTTP redirect from your writable canned query submission:
- Use the `Accept: application/json` request header
- Include `?_json=1` in the request query string
- Include `""_json"": 1` in the form submission (or the JSON body submission)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-692272860,https://api.github.com/repos/simonw/datasette/issues/880,692272860,MDEyOklzc3VlQ29tbWVudDY5MjI3Mjg2MA==,9599,simonw,2020-09-14T19:43:47Z,2020-09-14T19:43:47Z,OWNER,"I'm going to add support for POST content that is sent as a JSON document, in addition to the existing support for key=value encoded POST bodies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-692271804,https://api.github.com/repos/simonw/datasette/issues/880,692271804,MDEyOklzc3VlQ29tbWVudDY5MjI3MTgwNA==,9599,simonw,2020-09-14T19:41:37Z,2020-09-14T19:41:37Z,OWNER,Relevant code section: https://github.com/simonw/datasette/blob/1552ac931e4d2cf516caac3ceeab4fd24da1510a/datasette/views/database.py#L209-L232,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/965#issuecomment-692244252,https://api.github.com/repos/simonw/datasette/issues/965,692244252,MDEyOklzc3VlQ29tbWVudDY5MjI0NDI1Mg==,9599,simonw,2020-09-14T18:49:48Z,2020-09-14T18:49:48Z,OWNER,Documented here: https://docs.datasette.io/en/latest/custom_templates.html#custom-error-pages,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",701294727,"Documentation for 404.html, 500.html templates",
https://github.com/simonw/datasette/issues/965#issuecomment-692231257,https://api.github.com/repos/simonw/datasette/issues/965,692231257,MDEyOklzc3VlQ29tbWVudDY5MjIzMTI1Nw==,9599,simonw,2020-09-14T18:25:04Z,2020-09-14T18:25:04Z,OWNER,In documenting this I realized that it's confusing that the default `500.html` template is often used for non-500 errors (404 for example). I think I'll rename that default template to `error.html` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",701294727,"Documentation for 404.html, 500.html templates",
https://github.com/simonw/datasette/issues/964#issuecomment-692212641,https://api.github.com/repos/simonw/datasette/issues/964,692212641,MDEyOklzc3VlQ29tbWVudDY5MjIxMjY0MQ==,9599,simonw,2020-09-14T17:49:44Z,2020-09-14T17:49:44Z,OWNER,Documentation: https://docs.datasette.io/en/latest/custom_templates.html#returning-404s,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",700728217,raise_404 mechanism for custom templates,
https://github.com/simonw/datasette/issues/965#issuecomment-692207341,https://api.github.com/repos/simonw/datasette/issues/965,692207341,MDEyOklzc3VlQ29tbWVudDY5MjIwNzM0MQ==,9599,simonw,2020-09-14T17:40:05Z,2020-09-14T17:40:05Z,OWNER,Also link to these from the docs added in #964.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",701294727,"Documentation for 404.html, 500.html templates",
https://github.com/simonw/datasette/issues/944#issuecomment-691788478,https://api.github.com/repos/simonw/datasette/issues/944,691788478,MDEyOklzc3VlQ29tbWVudDY5MTc4ODQ3OA==,9599,simonw,2020-09-14T03:21:45Z,2020-09-14T03:21:45Z,OWNER,Having tried this out I think it does need a `raise_404()` mechanism - which needs to be smart enough to trigger the default 404 handler without accidentally going into an infinite loop.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/880#issuecomment-691785692,https://api.github.com/repos/simonw/datasette/issues/880,691785692,MDEyOklzc3VlQ29tbWVudDY5MTc4NTY5Mg==,9599,simonw,2020-09-14T03:10:11Z,2020-09-14T03:10:11Z,OWNER,"Answer: no, it's [not safe](https://twitter.com/glenathan/status/1305081266065244162) to skip CSRF if there's an `Accept: application/json` header because of a nasty old `crossdomain.xml` Flash vulnerability: https://blog.appsecco.com/exploiting-csrf-on-json-endpoints-with-flash-and-redirects-681d4ad6b31b?gi=a5ee3d7a8235","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/940#issuecomment-691781345,https://api.github.com/repos/simonw/datasette/issues/940,691781345,MDEyOklzc3VlQ29tbWVudDY5MTc4MTM0NQ==,9599,simonw,2020-09-14T02:53:25Z,2020-09-14T02:53:49Z,OWNER,"That worked: https://github.com/simonw/datasette/runs/1110040212?check_suite_focus=true ran and deployed https://pypi.org/project/datasette/0.49a1/ to PyPI but it skipped the push to Docker step because there was an ""a"" in the tag.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-691779693,https://api.github.com/repos/simonw/datasette/issues/940,691779693,MDEyOklzc3VlQ29tbWVudDY5MTc3OTY5Mw==,9599,simonw,2020-09-14T02:46:39Z,2020-09-14T02:46:39Z,OWNER,I think those should be single quoted.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-691779510,https://api.github.com/repos/simonw/datasette/issues/940,691779510,MDEyOklzc3VlQ29tbWVudDY5MTc3OTUxMA==,9599,simonw,2020-09-14T02:45:53Z,2020-09-14T02:45:53Z,OWNER,This bit here: https://github.com/simonw/datasette/blob/c18117cf08ad67c704dab29e3cb3b88f1de4026b/.github/workflows/publish.yml#L58-L62,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-691779361,https://api.github.com/repos/simonw/datasette/issues/940,691779361,MDEyOklzc3VlQ29tbWVudDY5MTc3OTM2MQ==,9599,simonw,2020-09-14T02:45:04Z,2020-09-14T02:45:04Z,OWNER,"Package deploys are still broken, just got this error trying to ship 0.49a1: https://github.com/simonw/datasette/actions/runs/253099665
> The workflow is not valid. .github/workflows/publish.yml (Line: 61, Col: 9): Unexpected symbol: '""a""'. Located at position 24 within expression: !(contains(github.ref, ""a"") || contains(github.ref, ""b""))
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/944#issuecomment-691774262,https://api.github.com/repos/simonw/datasette/issues/944,691774262,MDEyOklzc3VlQ29tbWVudDY5MTc3NDI2Mg==,9599,simonw,2020-09-14T02:24:08Z,2020-09-14T02:24:08Z,OWNER,"Actually don't need `{{ raise_404(""Museum not found"") }}` because we already have `{{ custom_status(404) }}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/944#issuecomment-691769222,https://api.github.com/repos/simonw/datasette/issues/944,691769222,MDEyOklzc3VlQ29tbWVudDY5MTc2OTIyMg==,9599,simonw,2020-09-14T02:01:33Z,2020-09-14T02:01:33Z,OWNER,I'm going to cache the `list_templates()` result in memory. If you want to add a new template-defined route you will need to restart the server. I think that's acceptable.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/519#issuecomment-691566247,https://api.github.com/repos/simonw/datasette/issues/519,691566247,MDEyOklzc3VlQ29tbWVudDY5MTU2NjI0Nw==,9599,simonw,2020-09-12T22:48:53Z,2020-09-12T22:48:53Z,OWNER,"I think I've figured out what to do about stability of the HTML and the default templates with respect to semantic versioning.
I'm going to announce that the JSON API - including the variables made available to templates - should be considered stable according to semver. I will only break backwards compatibility at that level in a major version release.
The template HTML (and default CSS) will not be considered a stable interface. They won't change on bug fix releases but they may change (albeit described in the release notes) on minor version bumps.
Since the template inputs are stable, you can run your own copy of the previous version's templates if something breaks.
This means users (and plugin authors) who make changes to the default Datasette UI will have to test their changes against every minor release. I think that's OK.
If you write plugins that don't affect the Datasette HTML UI you will be able to expect stability across minor version releases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0,
https://github.com/simonw/datasette/issues/880#issuecomment-691558387,https://api.github.com/repos/simonw/datasette/issues/880,691558387,MDEyOklzc3VlQ29tbWVudDY5MTU1ODM4Nw==,9599,simonw,2020-09-12T22:04:48Z,2020-09-12T22:04:48Z,OWNER,"Is it safe to skip CSRF checks if the incoming request has `Accept: application/json` on it?
I'm not sure that matters since `asgi-csrf` already won't reject requests that either have no cookies or are using a `Authorization: Bearer ...` header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-691557675,https://api.github.com/repos/simonw/datasette/issues/880,691557675,MDEyOklzc3VlQ29tbWVudDY5MTU1NzY3NQ==,9599,simonw,2020-09-12T22:01:02Z,2020-09-12T22:01:11Z,OWNER,"Maybe POST to `.json` doesn't actually make sense. I could instead support `POST /db/queryname` with an optional mechanism for requesting that the response to that POST be in a JSON format.
Could be a `Accept: application/json` header with an option of including `""_accept"": ""json""` as a POST parameter instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-691557429,https://api.github.com/repos/simonw/datasette/issues/880,691557429,MDEyOklzc3VlQ29tbWVudDY5MTU1NzQyOQ==,9599,simonw,2020-09-12T21:59:39Z,2020-09-12T21:59:39Z,OWNER,"What should happen when something does a POST to an extension that was registered by a plugin, e.g. `POST /db/table.atom` ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/782#issuecomment-691554088,https://api.github.com/repos/simonw/datasette/issues/782,691554088,MDEyOklzc3VlQ29tbWVudDY5MTU1NDA4OA==,9599,simonw,2020-09-12T21:39:03Z,2020-09-12T21:39:03Z,OWNER,"Plan: release a new release of Datasette (probably 0.49) with the new JSON API design, but provide a plugin called something like `datasette-api-0-48` which runs as ASGI wrapping middleware and internally rewrites incoming requests to e.g. `/db/table.json` to behave if they have the `?_extra=` params on them necessary to produce the 0.48 version of the JSON.
Anyone who has built applications against 0.48 can install that plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/262#issuecomment-691526975,https://api.github.com/repos/simonw/datasette/issues/262,691526975,MDEyOklzc3VlQ29tbWVudDY5MTUyNjk3NQ==,9599,simonw,2020-09-12T18:22:44Z,2020-09-12T18:22:44Z,OWNER,Are there any interesting use-cases for a plugin hook that allows plugins to define their own `?_extra=` blocks?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/782#issuecomment-691526878,https://api.github.com/repos/simonw/datasette/issues/782,691526878,MDEyOklzc3VlQ29tbWVudDY5MTUyNjg3OA==,9599,simonw,2020-09-12T18:21:41Z,2020-09-12T18:22:20Z,OWNER,"Would it be so bad if the default format had a `""rows""` key containing the array of rows? Maybe it wouldn't. The reason I always use `?_shape=array` is because I want an array of objects, rather than an array of arrays that I have to match up again with their columns.
A default format that's an object rather than array also gives something for the `?_extra=` parameter to add its extras to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-691526762,https://api.github.com/repos/simonw/datasette/issues/782,691526762,MDEyOklzc3VlQ29tbWVudDY5MTUyNjc2Mg==,9599,simonw,2020-09-12T18:20:19Z,2020-09-12T18:20:19Z,OWNER,"I'd like to revisit the idea of using `?_extra=x` to opt-in to extra blocks of JSON, from #262","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/262#issuecomment-691526719,https://api.github.com/repos/simonw/datasette/issues/262,691526719,MDEyOklzc3VlQ29tbWVudDY5MTUyNjcxOQ==,9599,simonw,2020-09-12T18:19:50Z,2020-09-12T18:19:50Z,OWNER,"> Idea: `?_extra=sqllog` could output a lot of every individual SQL statement that was executed in order to generate the page - useful for seeing how foreign key expansion and faceting actually works.
I built a version of that a while ago as the `?_trace=1` argument.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/262#issuecomment-389702480,https://api.github.com/repos/simonw/datasette/issues/262,389702480,MDEyOklzc3VlQ29tbWVudDM4OTcwMjQ4MA==,9599,simonw,2018-05-17T00:00:39Z,2020-09-12T18:19:30Z,OWNER,Idea: `?_extra=sqllog` could output a lot of every individual SQL statement that was executed in order to generate the page - useful for seeing how foreign key expansion and faceting actually works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/680#issuecomment-691526635,https://api.github.com/repos/simonw/datasette/issues/680,691526635,MDEyOklzc3VlQ29tbWVudDY5MTUyNjYzNQ==,9599,simonw,2020-09-12T18:18:50Z,2020-09-12T18:18:50Z,OWNER,"I'm happy with the not-quite-automated way I'm doing this, so I'm going to close this issue.
That's documented here https://docs.datasette.io/en/0.48/contributing.html#release-process - I use https://euangoddard.github.io/clipboard2markdown/ to create the GitHub releases markdown version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",569275763,Release automation: automate the bit that posts the GitHub release,
https://github.com/simonw/datasette/issues/782#issuecomment-691526489,https://api.github.com/repos/simonw/datasette/issues/782,691526489,MDEyOklzc3VlQ29tbWVudDY5MTUyNjQ4OQ==,9599,simonw,2020-09-12T18:17:16Z,2020-09-12T18:17:16Z,OWNER,(I think I may have been over-thinking the details of this is for a couple of years now.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-691526416,https://api.github.com/repos/simonw/datasette/issues/782,691526416,MDEyOklzc3VlQ29tbWVudDY5MTUyNjQxNg==,9599,simonw,2020-09-12T18:16:36Z,2020-09-12T18:16:36Z,OWNER,I'm going to hack together a preview of this in a branch and deploy it somewhere so people can see what I've got planned. Much easier to evaluate a working prototype than static examples.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-691501132,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50,691501132,MDEyOklzc3VlQ29tbWVudDY5MTUwMTEzMg==,706257,bcongdon,2020-09-12T14:48:10Z,2020-09-12T14:48:10Z,NONE,"This seems to be an issue even with larger values of `--stop_after`:
```
$ twitter-to-sqlite favorites twitter.db --stop_after=2000
Importing favorites [####################################] 198
$
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",698791218,"favorites --stop_after=N stops after min(N, 200)",
https://github.com/simonw/datasette/issues/963#issuecomment-691379980,https://api.github.com/repos/simonw/datasette/issues/963,691379980,MDEyOklzc3VlQ29tbWVudDY5MTM3OTk4MA==,9599,simonw,2020-09-12T01:50:56Z,2020-09-12T01:50:56Z,OWNER,Good bug - looks like a problem with the hidden form fields.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",699947574,Currently selected array facets are not correctly persisted through hidden form fields,
https://github.com/simonw/datasette/issues/782#issuecomment-691323302,https://api.github.com/repos/simonw/datasette/issues/782,691323302,MDEyOklzc3VlQ29tbWVudDY5MTMyMzMwMg==,9599,simonw,2020-09-11T21:38:27Z,2020-09-11T21:40:04Z,OWNER,"Another idea: the default output could be the list of dicts:
```json
[
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""a"",
""content"": ""a-a-a""
},
...
]
```
BUT... I could include pagination information in the HTTP headers - as seen in the WordPress REST API or the GitHub API:
```
~ % curl -s -i 'https://api.github.com/repos/simonw/datasette/commits' | head -n 40
HTTP/1.1 200 OK
server: GitHub.com
date: Fri, 11 Sep 2020 21:37:46 GMT
content-type: application/json; charset=utf-8
status: 200 OK
cache-control: public, max-age=60, s-maxage=60
vary: Accept, Accept-Encoding, Accept, X-Requested-With
etag: W/""71c99379743513394e880c6306b66bf9""
last-modified: Fri, 11 Sep 2020 21:32:54 GMT
x-github-media-type: github.v3; format=json
link: ; rel=""next"", ; rel=""last""
access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
access-control-allow-origin: *
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'
X-Ratelimit-Limit: 60
X-Ratelimit-Remaining: 55
X-Ratelimit-Reset: 1599863850
X-Ratelimit-Used: 5
Accept-Ranges: bytes
Content-Length: 118240
X-GitHub-Request-Id: EC76:0EAD:313F40:5291A4:5F5BEE37
[
{
""sha"": ""d02f6151dae073135a22d0123e8abdc6cbef7c50"",
""node_id"": ""MDY6Q29tbWl0MTA3OTE0NDkzOmQwMmY2MTUxZGFlMDczMTM1YTIyZDAxMjNlOGFiZGM2Y2JlZjdjNTA="",
""commit"": {
```
Alternative shapes would provide the pagination information (and other extensions) in the JSON, e.g.:
`/squirrels/squirrels.json?_shape=paginated`
```json
{
""rows"": [
{
""pk1"": ""a"",
""pk2"": ""a"",
""pk3"": ""a"",
""content"": ""a-a-a""
}
],
""pagination"": {
""next"": ""234"",
""count"": 442
}
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/947#issuecomment-691318133,https://api.github.com/repos/simonw/datasette/issues/947,691318133,MDEyOklzc3VlQ29tbWVudDY5MTMxODEzMw==,9599,simonw,2020-09-11T21:23:40Z,2020-09-11T21:23:40Z,OWNER,"I'm going to use exit code 1 for any errors, be they 500 or 404.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684111953,datasette --get exit code should reflect HTTP errors,
https://github.com/simonw/datasette/issues/962#issuecomment-691250299,https://api.github.com/repos/simonw/datasette/issues/962,691250299,MDEyOklzc3VlQ29tbWVudDY5MTI1MDI5OQ==,9599,simonw,2020-09-11T18:33:50Z,2020-09-11T18:33:50Z,OWNER,Since this is purely a debugging option I'm going to allow myself not to write a unit test for it!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",699622046,datasette --pdb option for debugging errors,
https://github.com/dogsheep/twitter-to-sqlite/issues/50#issuecomment-690860653,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50,690860653,MDEyOklzc3VlQ29tbWVudDY5MDg2MDY1Mw==,370930,mikepqr,2020-09-11T04:04:08Z,2020-09-11T04:04:08Z,CONTRIBUTOR,"There's probably a nicer way of doing (hence this is a comment rather than a PR), but this appears to fix it:
```diff
--- a/twitter_to_sqlite/utils.py
+++ b/twitter_to_sqlite/utils.py
@@ -181,6 +181,7 @@ def fetch_timeline(
args[""tweet_mode""] = ""extended""
min_seen_id = None
num_rate_limit_errors = 0
+ seen_count = 0
while True:
if min_seen_id is not None:
args[""max_id""] = min_seen_id - 1
@@ -208,6 +209,7 @@ def fetch_timeline(
yield tweet
min_seen_id = min(t[""id""] for t in tweets)
max_seen_id = max(t[""id""] for t in tweets)
+ seen_count += len(tweets)
if last_since_id is not None:
max_seen_id = max((last_since_id, max_seen_id))
last_since_id = max_seen_id
@@ -217,7 +219,9 @@ def fetch_timeline(
replace=True,
)
if stop_after is not None:
- break
+ if seen_count >= stop_after:
+ break
+ args[""count""] = min(args[""count""], stop_after - seen_count)
time.sleep(sleep)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",698791218,"favorites --stop_after=N stops after min(N, 200)",
https://github.com/simonw/sqlite-utils/issues/157#issuecomment-689850509,https://api.github.com/repos/simonw/sqlite-utils/issues/157,689850509,MDEyOklzc3VlQ29tbWVudDY4OTg1MDUwOQ==,9599,simonw,2020-09-09T22:14:49Z,2020-09-09T22:14:49Z,OWNER,It will call this method: https://github.com/simonw/sqlite-utils/blob/367082e787101fb90901ef3214804ab23a92ce46/sqlite_utils/db.py#L405-L411,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697179806,sqlite-utils add-foreign-keys command,
https://github.com/simonw/sqlite-utils/issues/157#issuecomment-689850289,https://api.github.com/repos/simonw/sqlite-utils/issues/157,689850289,MDEyOklzc3VlQ29tbWVudDY4OTg1MDI4OQ==,9599,simonw,2020-09-09T22:14:19Z,2020-09-09T22:14:19Z,OWNER,"This can accept four arguments: table, column, other_table, other_column:
```
sqlite-utils add-foreign-keys calands.db \
units_with_maps ACCESS_TYP ACCESS_TYP id \
units_with_maps AGNCY_NAME AGNCY_NAME id \
units_with_maps AGNCY_LEV AGNCY_LEV id
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697179806,sqlite-utils add-foreign-keys command,
https://github.com/simonw/sqlite-utils/pull/156#issuecomment-689735140,https://api.github.com/repos/simonw/sqlite-utils/issues/156,689735140,MDEyOklzc3VlQ29tbWVudDY4OTczNTE0MA==,9599,simonw,2020-09-09T18:21:06Z,2020-09-09T18:21:06Z,OWNER,"Good spot, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697030843,Typos in tests,
https://github.com/simonw/datasette/issues/961#issuecomment-689635754,https://api.github.com/repos/simonw/datasette/issues/961,689635754,MDEyOklzc3VlQ29tbWVudDY4OTYzNTc1NA==,9599,simonw,2020-09-09T15:24:31Z,2020-09-09T15:24:31Z,OWNER,"I thought about checking that every database in the `databases:` section exists and ditto for `tables:` - but actually I think it's useful to be able to keep a `metadata.yml` around with configuration for databases or tables that aren't currently attached to Datasette.
I could treat those as warnings and output a warning to standard out when the server starts instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",696908389,Verification checks for metadata.json on startup,
https://github.com/simonw/datasette/issues/961#issuecomment-689635094,https://api.github.com/repos/simonw/datasette/issues/961,689635094,MDEyOklzc3VlQ29tbWVudDY4OTYzNTA5NA==,9599,simonw,2020-09-09T15:23:24Z,2020-09-09T15:23:24Z,OWNER,"Checks can include:
- `facets:` lists columns that exist
- `sort:` and `sort_desc:` columns
- `fts_table` and `fts_pk` are valid","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",696908389,Verification checks for metadata.json on startup,
https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-689226390,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17,689226390,MDEyOklzc3VlQ29tbWVudDY4OTIyNjM5MA==,9599,simonw,2020-09-09T00:36:07Z,2020-09-09T00:36:07Z,MEMBER,"Alternative names:
- type
- record_type
- doctype
I think `type` is right. It matches what Elasticsearch used to call their equivalent of this (before they removed the feature!). https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694500679,"Rename ""table"" to ""type""",
https://github.com/simonw/sqlite-utils/issues/145#issuecomment-689186423,https://api.github.com/repos/simonw/sqlite-utils/issues/145,689186423,MDEyOklzc3VlQ29tbWVudDY4OTE4NjQyMw==,9599,simonw,2020-09-08T23:21:23Z,2020-09-08T23:21:23Z,OWNER,Fixed in PR #146.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688659182,Bug when first record contains fewer columns than subsequent records,
https://github.com/simonw/sqlite-utils/pull/146#issuecomment-689185393,https://api.github.com/repos/simonw/sqlite-utils/issues/146,689185393,MDEyOklzc3VlQ29tbWVudDY4OTE4NTM5Mw==,9599,simonw,2020-09-08T23:17:42Z,2020-09-08T23:17:42Z,OWNER,"That seems like a reasonable approach to me, especially since this is going to be a pretty rare edge-case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns,
https://github.com/simonw/sqlite-utils/issues/155#issuecomment-689166404,https://api.github.com/repos/simonw/sqlite-utils/issues/155,689166404,MDEyOklzc3VlQ29tbWVudDY4OTE2NjQwNA==,9599,simonw,2020-09-08T22:20:03Z,2020-09-08T22:20:03Z,OWNER,"I'm going to update `sqlite-utils optimize` to also take an optional list of tables, for consistency.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",696045581,rebuild-fts command and table.rebuild_fts() method,
https://github.com/simonw/sqlite-utils/issues/153#issuecomment-689165985,https://api.github.com/repos/simonw/sqlite-utils/issues/153,689165985,MDEyOklzc3VlQ29tbWVudDY4OTE2NTk4NQ==,9599,simonw,2020-09-08T22:18:52Z,2020-09-08T22:18:52Z,OWNER,"I've reverted this change again, because it turns out using the `rebuild` FTS mechanism is a better way of repairing this issue - see #155.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695377804,table.optimize() should delete junk rows from *_fts_docsize,
https://github.com/simonw/sqlite-utils/issues/155#issuecomment-689163158,https://api.github.com/repos/simonw/sqlite-utils/issues/155,689163158,MDEyOklzc3VlQ29tbWVudDY4OTE2MzE1OA==,9599,simonw,2020-09-08T22:10:27Z,2020-09-08T22:10:27Z,OWNER,"For the command version:
sqlite-utils rebuild-fts mydb.db
This will rebuild all detected FTS tables. You can also specify one or more explicit tables:
sqlite-utils rebuild-fts mydb.db dogs
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",696045581,rebuild-fts command and table.rebuild_fts() method,
https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688626037,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19,688626037,MDEyOklzc3VlQ29tbWVudDY4ODYyNjAzNw==,9599,simonw,2020-09-08T05:27:07Z,2020-09-08T05:27:07Z,MEMBER,"A really clever way to do this would be with triggers. The indexer script would add triggers to each of the database tables that it is indexing - each in their own database.
Those triggers would then maintain a `_index_queue_` table. This table would record the primary key of rows that are added, modified or deleted. The indexer could then work by reading through the `_index_queue_` table, re-indexing (or deleting) just the primary keys listed there, and then emptying the queue once it has finished.
This would add a small amount of overhead to insert/update/delete queries run against the table. My hunch is that the overhead would be miniscule, but I could still allow people to opt-out for tables that are so high traffic that this would matter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695556681,Figure out incremental re-indexing,
https://github.com/dogsheep/dogsheep-beta/issues/19#issuecomment-688625430,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19,688625430,MDEyOklzc3VlQ29tbWVudDY4ODYyNTQzMA==,9599,simonw,2020-09-08T05:24:50Z,2020-09-08T05:24:50Z,MEMBER,"I thought about allowing tables to define a incremental indexing SQL query - maybe something that can return just records touched in the past hour, or records since a recorded ""last indexed record"" value.
The problem with this is deletes - if you delete a record, how does the indexer know to remove it? See #18 - that's already caused problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695556681,Figure out incremental re-indexing,
https://github.com/dogsheep/dogsheep-beta/issues/18#issuecomment-688623097,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18,688623097,MDEyOklzc3VlQ29tbWVudDY4ODYyMzA5Nw==,9599,simonw,2020-09-08T05:15:51Z,2020-09-08T05:15:51Z,MEMBER,"I'm inclined to go with the first, simpler option. I have longer term plans for efficient incremental index updates based on clever trickery with triggers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695553522,Deleted records stay in the search index,
https://github.com/dogsheep/dogsheep-beta/issues/18#issuecomment-688622995,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18,688622995,MDEyOklzc3VlQ29tbWVudDY4ODYyMjk5NQ==,9599,simonw,2020-09-08T05:15:21Z,2020-09-08T05:15:21Z,MEMBER,"Alternatively it could run as it does now but add a `DELETE FROM index1.search_index WHERE key not in (select key from ...)`.
I'm not sure which would be more efficient.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695553522,Deleted records stay in the search index,
https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688573964,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688573964,MDEyOklzc3VlQ29tbWVudDY4ODU3Mzk2NA==,96218,simonwiles,2020-09-08T01:55:07Z,2020-09-08T01:55:07Z,CONTRIBUTOR,"Okay, I've rewritten this PR to preserve the batching behaviour but still fix #145, and rebased the branch to account for the `db.execute()` api change. It's not terribly sophisticated -- if it attempts to insert a batch which has too many variables, the exception is caught, the batch is split in two and each half is inserted separately, and then it carries on as before with the same `batch_size`. In the edge case where this gets triggered, subsequent batches will all be inserted in two groups too if they continue to have the same number of columns (which is presumably reasonably likely). Do you reckon this is acceptable when set against the awkwardness of recalculating the `batch_size` on the fly?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns,
https://github.com/simonw/sqlite-utils/issues/154#issuecomment-688544156,https://api.github.com/repos/simonw/sqlite-utils/issues/154,688544156,MDEyOklzc3VlQ29tbWVudDY4ODU0NDE1Ng==,9599,simonw,2020-09-07T23:47:10Z,2020-09-07T23:47:10Z,OWNER,This is already covered in the tests though: https://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/tests/test_cli.py#L1300-L1328,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695441530,OperationalError: cannot change into wal mode from within a transaction,
https://github.com/simonw/sqlite-utils/issues/154#issuecomment-688543128,https://api.github.com/repos/simonw/sqlite-utils/issues/154,688543128,MDEyOklzc3VlQ29tbWVudDY4ODU0MzEyOA==,9599,simonw,2020-09-07T23:43:10Z,2020-09-07T23:43:10Z,OWNER,"Running this against the same file works:
```
$ sqlite3 beta.db
SQLite version 3.31.1 2020-01-27 19:55:54
Enter "".help"" for usage hints.
sqlite> PRAGMA journal_mode=wal;
wal
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695441530,OperationalError: cannot change into wal mode from within a transaction,
https://github.com/simonw/sqlite-utils/issues/152#issuecomment-688500704,https://api.github.com/repos/simonw/sqlite-utils/issues/152,688500704,MDEyOklzc3VlQ29tbWVudDY4ODUwMDcwNA==,9599,simonw,2020-09-07T20:28:45Z,2020-09-07T21:17:48Z,OWNER,"The principle reason to turn these on - at least so far - is that without it weird things happen where FTS tables (in particular `*_fts_docsize`) grow without limit over time, because calls to `INSERT OR REPLACE` against the parent table cause additional rows to be inserted into `*_fts_docsize` even if the row was replaced rather than being inserted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695376054,Turn on recursive_triggers by default,
https://github.com/simonw/sqlite-utils/issues/153#issuecomment-688511161,https://api.github.com/repos/simonw/sqlite-utils/issues/153,688511161,MDEyOklzc3VlQ29tbWVudDY4ODUxMTE2MQ==,9599,simonw,2020-09-07T21:07:20Z,2020-09-07T21:07:29Z,OWNER,"FTS4 uses a different column name here: https://datasette-sqlite-fts4.datasette.io/24ways-fts4/articles_fts_docsize
```
CREATE TABLE 'articles_fts_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695377804,table.optimize() should delete junk rows from *_fts_docsize,
https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688508510,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688508510,MDEyOklzc3VlQ29tbWVudDY4ODUwODUxMA==,9599,simonw,2020-09-07T20:56:03Z,2020-09-07T20:56:24Z,OWNER,"The problem with this approach is that it requires us to consume the entire iterator before we can start inserting rows into the table - here on line 1052:
https://github.com/simonw/sqlite-utils/blob/bb131793feac16bc7181ab997568f941b0220ef2/sqlite_utils/db.py#L1047-L1054
I designed the `.insert_all()` to avoid doing this, because I want to be able to pass it an iterator (or more likely a generator) that could produce potentially millions of records. Doing things one batch of 100 records at a time means that the Python process doesn't need to pull millions of records into memory at once.
`db-to-sqlite` is one example of a tool that uses that characteristic, in https://github.com/simonw/db-to-sqlite/blob/63e4ee972f292de13bb11767c0fb64b35339d954/db_to_sqlite/cli.py#L94-L106
So we need to solve this issue without consuming the entire iterator with a `records = list(records)` call.
I think one way to do this is to execute each chunk one at a time and watch out for an exception that indicates that we sent too many parameters - then adjust the chunk size down and try again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns,
https://github.com/simonw/sqlite-utils/issues/153#issuecomment-688506015,https://api.github.com/repos/simonw/sqlite-utils/issues/153,688506015,MDEyOklzc3VlQ29tbWVudDY4ODUwNjAxNQ==,9599,simonw,2020-09-07T20:46:58Z,2020-09-07T20:46:58Z,OWNER,Writing a test for this will be a tiny bit tricky. I think I'll use a test that replicates the bug in #149.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695377804,table.optimize() should delete junk rows from *_fts_docsize,
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688501064,MDEyOklzc3VlQ29tbWVudDY4ODUwMTA2NA==,9599,simonw,2020-09-07T20:30:15Z,2020-09-07T20:30:38Z,OWNER,"The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL:
```sql
DELETE FROM [licenses_fts_docsize] WHERE id NOT IN (
SELECT rowid FROM [licenses_fts]);
```
I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/152#issuecomment-688500294,https://api.github.com/repos/simonw/sqlite-utils/issues/152,688500294,MDEyOklzc3VlQ29tbWVudDY4ODUwMDI5NA==,9599,simonw,2020-09-07T20:27:07Z,2020-09-07T20:27:07Z,OWNER,I'm going to make this an argument to the `Database()` class constructor which defaults to `True`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695376054,Turn on recursive_triggers by default,
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688499924,MDEyOklzc3VlQ29tbWVudDY4ODQ5OTkyNA==,9599,simonw,2020-09-07T20:25:40Z,2020-09-07T20:25:50Z,OWNER,"https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:
> Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.
So I think the fix is to turn on `recursive_triggers` globally by default for `sqlite-utils`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499650,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688499650,MDEyOklzc3VlQ29tbWVudDY4ODQ5OTY1MA==,9599,simonw,2020-09-07T20:24:35Z,2020-09-07T20:24:35Z,OWNER,"This replicates the problem:
```
(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
{""table"": ""licenses"", ""count"": 7},
{""table"": ""licenses_fts_data"", ""count"": 35},
{""table"": ""licenses_fts_idx"", ""count"": 16},
{""table"": ""licenses_fts_docsize"", ""count"": 9151},
{""table"": ""licenses_fts_config"", ""count"": 1},
{""table"": ""licenses_fts"", ""count"": 7},
(github-to-sqlite) /tmp % github-to-sqlite repos github.db dogsheep
(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
{""table"": ""licenses"", ""count"": 7},
{""table"": ""licenses_fts_data"", ""count"": 45},
{""table"": ""licenses_fts_idx"", ""count"": 26},
{""table"": ""licenses_fts_docsize"", ""count"": 9161},
{""table"": ""licenses_fts_config"", ""count"": 1},
{""table"": ""licenses_fts"", ""count"": 7},
```
Note how the number of rows in `licenses_fts_docsize` goes from 9151 to 9161.
The number went up by ten. I used tracing from #151 to show that the following SQL executed ten times:
```
INSERT OR REPLACE INTO [licenses] ([key], [name], [node_id], [spdx_id], [url]) VALUES
(?, ?, ?, ?, ?);
```
Then I tried executing `PRAGMA recursive_triggers=on;` at the start of the script. This fixed the problem - running the script did not increase the number of rows in `licenses_fts_docsize`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482355,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688482355,MDEyOklzc3VlQ29tbWVudDY4ODQ4MjM1NQ==,9599,simonw,2020-09-07T19:22:51Z,2020-09-07T19:22:51Z,OWNER,"And the SQLite documentation says:
> When the REPLACE conflict resolution strategy deletes rows in order to satisfy a constraint, [delete triggers](https://www.sqlite.org/lang_createtrigger.html) fire if and only if [recursive triggers](https://www.sqlite.org/pragma.html#pragma_recursive_triggers) are enabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482055,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688482055,MDEyOklzc3VlQ29tbWVudDY4ODQ4MjA1NQ==,9599,simonw,2020-09-07T19:21:42Z,2020-09-07T19:21:42Z,OWNER,"Using `replace=True` there executes `INSERT OR REPLACE` - and Dan Kennedy (SQLite maintainer) on the SQLite forums said this:
> Are you using ""REPLACE INTO"", or ""UPDATE OR REPLACE"" on the ""licenses"" table without having first executed ""PRAGMA recursive_triggers = 1""? The docs note that delete triggers will not be fired in this case, which would explain things. Second paragraph under ""REPLACE"" here:
>
> https://www.sqlite.org/lang_conflict.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688481374,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688481374,MDEyOklzc3VlQ29tbWVudDY4ODQ4MTM3NA==,9599,simonw,2020-09-07T19:19:08Z,2020-09-07T19:19:08Z,OWNER,"reading through the code for `github-to-sqlite repos` - one of the things it does is calls `save_license` for each repo:
https://github.com/dogsheep/github-to-sqlite/blob/39b2234253096bd579feed4e25104698b8ccd2ba/github_to_sqlite/utils.py#L259-L262
```python
def save_license(db, license):
if license is None:
return None
return db[""licenses""].insert(license, pk=""key"", replace=True).last_pk
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688481317,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688481317,MDEyOklzc3VlQ29tbWVudDY4ODQ4MTMxNw==,96218,simonwiles,2020-09-07T19:18:55Z,2020-09-07T19:18:55Z,CONTRIBUTOR,"Just force-pushed to update d042f9c with more formatting changes to satisfy `black==20.8b1` and pass the GitHub Actions ""Test"" workflow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns,
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688480665,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688480665,MDEyOklzc3VlQ29tbWVudDY4ODQ4MDY2NQ==,9599,simonw,2020-09-07T19:16:20Z,2020-09-07T19:16:20Z,OWNER,"Aha! I have managed to replicate the bug:
```
(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
{""table"": ""licenses"", ""count"": 7},
{""table"": ""licenses_fts_data"", ""count"": 35},
{""table"": ""licenses_fts_idx"", ""count"": 16},
{""table"": ""licenses_fts_docsize"", ""count"": 9151},
{""table"": ""licenses_fts_config"", ""count"": 1},
{""table"": ""licenses_fts"", ""count"": 7},
(github-to-sqlite) /tmp % github-to-sqlite repos github.db dogsheep
(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
{""table"": ""licenses"", ""count"": 7},
{""table"": ""licenses_fts_data"", ""count"": 45},
{""table"": ""licenses_fts_idx"", ""count"": 26},
{""table"": ""licenses_fts_docsize"", ""count"": 9161},
{""table"": ""licenses_fts_config"", ""count"": 1},
{""table"": ""licenses_fts"", ""count"": 7},
```
Note that the number of records in `licenses_fts_docsize` went from 9151 to 9161.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688479163,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688479163,MDEyOklzc3VlQ29tbWVudDY4ODQ3OTE2Mw==,96218,simonwiles,2020-09-07T19:10:33Z,2020-09-07T19:11:57Z,CONTRIBUTOR,"@simonw -- I've gone ahead updated the documentation to reflect the changes introduced in this PR. IMO it's ready to merge now.
In writing the documentation changes, I begin to wonder about the value and role of `batch_size` at all, tbh. May I assume it was originally intended to prevent using the entire row set to determine columns and column types, and that this was a performance consideration? If so, this PR entirely undermines its purpose. I've been passing in excess of 500,000 rows at a time to `insert_all()` with these changes and although I'm sure the performance difference is measurable it's not really noticeable; given #145, I don't know that any performance advantages outweigh the problems doing it this way removes. What do you think about just dropping the argument and defaulting to the maximum `batch_size` permissible given `SQLITE_MAX_VARS`? Are there other reasons one might want to restrict `batch_size` that I've overlooked? I could open a new issue to discuss/implement this.
Of course the documentation will need to change again too if/when something is done about #147.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns,
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688464181,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688464181,MDEyOklzc3VlQ29tbWVudDY4ODQ2NDE4MQ==,9599,simonw,2020-09-07T18:19:54Z,2020-09-07T18:19:54Z,OWNER,"Even though that table doesn't declare an integer primary key it does have a `rowid` column: https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+%5Bkey%5D%2C+name%2C+spdx_id%2C+url%2C+node_id+from+licenses+order+by+%5Bkey%5D+limit+101
| rowid | key | name | spdx_id | url | node_id |
| --- | --- | --- | --- | --- | --- |
| 9150 | apache-2.0 | Apache License 2.0 | Apache-2.0 | | MDc6TGljZW5zZTI= |
| 112 | bsd-3-clause | BSD 3-Clause ""New"" or ""Revised"" License | BSD-3-Clause | | MDc6TGljZW5zZTU= |
https://www.sqlite.org/rowidtable.html explains has this clue:
> If the rowid is not aliased by INTEGER PRIMARY KEY then it is not persistent and might change. In particular the VACUUM command will change rowids for tables that do not declare an INTEGER PRIMARY KEY. Therefore, applications should not normally access the rowid directly, but instead use an INTEGER PRIMARY KEY. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460865,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688460865,MDEyOklzc3VlQ29tbWVudDY4ODQ2MDg2NQ==,9599,simonw,2020-09-07T18:07:14Z,2020-09-07T18:07:14Z,OWNER,"Another likely culprit: `licenses` has a text primary key, so it's not using `rowid`:
```sql
CREATE TABLE [licenses] (
[key] TEXT PRIMARY KEY,
[name] TEXT,
[spdx_id] TEXT,
[url] TEXT,
[node_id] TEXT
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460729,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688460729,MDEyOklzc3VlQ29tbWVudDY4ODQ2MDcyOQ==,9599,simonw,2020-09-07T18:06:44Z,2020-09-07T18:06:44Z,OWNER,First posted on SQLite forum here but I'm pretty sure this is a bug in how `sqlite-utils` created those tables: https://sqlite.org/forum/forumpost/51aada1b45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/148#issuecomment-688434226,https://api.github.com/repos/simonw/sqlite-utils/issues/148,688434226,MDEyOklzc3VlQ29tbWVudDY4ODQzNDIyNg==,9599,simonw,2020-09-07T16:50:33Z,2020-09-07T16:50:33Z,OWNER,"This may be as easy as applying `textwrap.dedent()` to this: https://github.com/simonw/sqlite-utils/blob/0e62744da9a429093e3409575c1f881376b0361f/sqlite_utils/db.py#L778-L787
I could apply that to a few other queries in that code as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695276328,More attractive indentation of created FTS table schema,
https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-687880459,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17,687880459,MDEyOklzc3VlQ29tbWVudDY4Nzg4MDQ1OQ==,9599,simonw,2020-09-06T19:36:32Z,2020-09-06T19:36:32Z,MEMBER,At some point I may even want to support search types which are indexed from (and inflated from) more than one database file. I'm going to ignore that for the moment though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694500679,"Rename ""table"" to ""type""",
https://github.com/dogsheep/dogsheep-beta/issues/13#issuecomment-686774592,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13,686774592,MDEyOklzc3VlQ29tbWVudDY4Njc3NDU5Mg==,9599,simonw,2020-09-03T21:30:21Z,2020-09-03T21:30:21Z,MEMBER,"This is partially supported: the custom search SQL we run doesn't escape them, but the `?_search` used to calculate facet counts does. So this is a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",692386625,Support advanced FTS queries,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686767208,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686767208,MDEyOklzc3VlQ29tbWVudDY4Njc2NzIwOA==,9599,simonw,2020-09-03T21:12:14Z,2020-09-03T21:12:14Z,MEMBER,Documentation: https://github.com/dogsheep/dogsheep-beta/blob/0.4/README.md#custom-results-display,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-686689612,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,686689612,MDEyOklzc3VlQ29tbWVudDY4NjY4OTYxMg==,9599,simonw,2020-09-03T18:44:20Z,2020-09-03T18:44:20Z,MEMBER,Facets are now displayed but selecting them doesn't work yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340,"Datasette plugin to provide custom page for running faceted, ranked searches",
https://github.com/dogsheep/dogsheep-beta/issues/5#issuecomment-686689366,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5,686689366,MDEyOklzc3VlQ29tbWVudDY4NjY4OTM2Ng==,9599,simonw,2020-09-03T18:43:50Z,2020-09-03T18:43:50Z,MEMBER,No longer needed thanks to #9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689847361,Add a context column that's not searchable,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686689122,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686689122,MDEyOklzc3VlQ29tbWVudDY4NjY4OTEyMg==,9599,simonw,2020-09-03T18:43:20Z,2020-09-03T18:43:20Z,MEMBER,Needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686688963,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686688963,MDEyOklzc3VlQ29tbWVudDY4NjY4ODk2Mw==,9599,simonw,2020-09-03T18:42:59Z,2020-09-03T18:42:59Z,MEMBER,I'm pleased with how this works now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/11#issuecomment-686618669,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/11,686618669,MDEyOklzc3VlQ29tbWVudDY4NjYxODY2OQ==,9599,simonw,2020-09-03T16:47:34Z,2020-09-03T16:53:25Z,MEMBER,I think a `is_public` integer column which defaults to 0 would be good here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",692125110,Public / Private mechanism,
https://github.com/dogsheep/dogsheep-beta/issues/10#issuecomment-686238498,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/10,686238498,MDEyOklzc3VlQ29tbWVudDY4NjIzODQ5OA==,9599,simonw,2020-09-03T04:05:05Z,2020-09-03T04:05:05Z,MEMBER,Since the first two categories are `created` and `saved` this one should be called `received`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691557547,Category 3: received,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686163754,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686163754,MDEyOklzc3VlQ29tbWVudDY4NjE2Mzc1NA==,9599,simonw,2020-09-03T00:46:21Z,2020-09-03T00:46:21Z,MEMBER,"Challenge: the `dogsheep-beta.yml` configuration file that is passed to the `dogsheep-beta index` command needs to also be made available to Datasette itself, so that it can read the configuration.
Let's say it can either be duplicated in the `plugins` configuration block of the `metadata.yml` OR you can do this in `metadata.yml`:
```yaml
plugins:
dogsheep-beta:
config_file: dogsheep-beta.yml
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686158454,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686158454,MDEyOklzc3VlQ29tbWVudDY4NjE1ODQ1NA==,9599,simonw,2020-09-03T00:32:42Z,2020-09-03T00:32:42Z,MEMBER,"If this turns out to be too inefficient I could add a `display` text column to the `search_index` table which is designed to be populated with arbitrary JSON by the indexing query, which can then be used to render the template fragment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154627,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686154627,MDEyOklzc3VlQ29tbWVudDY4NjE1NDYyNw==,9599,simonw,2020-09-03T00:19:22Z,2020-09-03T00:19:22Z,MEMBER,If this performs well enough (100 displayed items will be 100 extra `display_sql` calls) then I'll go with this as the design for the feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154486,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686154486,MDEyOklzc3VlQ29tbWVudDY4NjE1NDQ4Ng==,9599,simonw,2020-09-03T00:18:54Z,2020-09-03T00:18:54Z,MEMBER,"`display_sql` could be optional. If it's not defined, a `row` object is passed to the template which is the row that's stored in `search_index`. If `display_sql` IS defined then it's executed and the result is made available as a `display` object in addition to the `row` object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686153967,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686153967,MDEyOklzc3VlQ29tbWVudDY4NjE1Mzk2Nw==,9599,simonw,2020-09-03T00:17:16Z,2020-09-03T00:17:55Z,MEMBER,"Maybe I can take advantage of https://sqlite.org/np1queryprob.html here - I could define a SQL query for fetching the ""display"" version of each item, and include a Jinja template fragment in the configuration as well. Maybe something like this:
```yaml
photos.db:
photos_with_apple_metadata:
sql: |-
select
sha256 as key,
'Photo in ' || coalesce(place_city, 'unknown') as title,
(
select
group_concat(normalized_string, ' ')
from
labels
where
labels.uuid = photos_with_apple_metadata.uuid
) as search_1,
date as timestamp,
1 as category
from
photos_with_apple_metadata
display_sql: |-
select
sha256, place_city, date
from photos_with_apple_metadata
where sha256 = :key
display: |-
Taken in {{ display.place_city }} on {{ display.date }}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965,Mechanism for defining custom display of results,
https://github.com/simonw/datasette/pull/952#issuecomment-686061028,https://api.github.com/repos/simonw/datasette/issues/952,686061028,MDEyOklzc3VlQ29tbWVudDY4NjA2MTAyOA==,27856297,dependabot-preview[bot],2020-09-02T22:26:14Z,2020-09-02T22:26:14Z,CONTRIBUTOR,"Looks like black is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687245650,"Update black requirement from ~=19.10b0 to >=19.10,<21.0",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685970384,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685970384,MDEyOklzc3VlQ29tbWVudDY4NTk3MDM4NA==,9599,simonw,2020-09-02T20:11:41Z,2020-09-02T20:11:59Z,MEMBER,"Default categories:
- 1 = created
- 2 = saved","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966707,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685966707,MDEyOklzc3VlQ29tbWVudDY4NTk2NjcwNw==,9599,simonw,2020-09-02T20:04:08Z,2020-09-02T20:04:08Z,MEMBER,I'll make `category` a foreign key to a `categories` table so Datasette can automatically show the `name` column.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966361,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685966361,MDEyOklzc3VlQ29tbWVudDY4NTk2NjM2MQ==,9599,simonw,2020-09-02T20:03:29Z,2020-09-02T20:03:41Z,MEMBER,"I'm going to implement the first version of this as an indexed integer `category` column which has 1 for ""about me"" and 2 for ""liked by me"" - and space for other category numerals in the future, albeit a row can only belong to one category.
I'll think about a full tagging system separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685965516,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685965516,MDEyOklzc3VlQ29tbWVudDY4NTk2NTUxNg==,9599,simonw,2020-09-02T20:01:54Z,2020-09-02T20:01:54Z,MEMBER,"Relevant post: https://sqlite.org/forum/forumpost/9f06fedaa5 - drh says:
> Indexes are one-to-one. There is one entry in the index for each row in the table.
>
> You are asking for an index that is many-to-one - multiple index entries for each table row.
>
> A Full-Text Index is basically a many-to-one index. So if all of your array entries really are words, you could probably get this to work using a Full-Text Index.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685962280,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685962280,MDEyOklzc3VlQ29tbWVudDY4NTk2MjI4MA==,9599,simonw,2020-09-02T19:55:26Z,2020-09-02T19:59:58Z,MEMBER,"Relevant: https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/
SQLite supports bitwise operators Binary AND (&) and Binary OR (|) - I could try those. Not sure how they interact with indexes though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-685961809,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,685961809,MDEyOklzc3VlQ29tbWVudDY4NTk2MTgwOQ==,9599,simonw,2020-09-02T19:54:24Z,2020-09-02T19:54:24Z,MEMBER,"This should implement search highlighting too, as seen on https://til.simonwillison.net/til/search?q=cloud
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340,"Datasette plugin to provide custom page for running faceted, ranked searches",
https://github.com/dogsheep/dogsheep-beta/issues/8#issuecomment-685960072,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8,685960072,MDEyOklzc3VlQ29tbWVudDY4NTk2MDA3Mg==,9599,simonw,2020-09-02T19:50:47Z,2020-09-02T19:50:47Z,MEMBER,"This doesn't actually help, because the Datasette table view page doesn't then support adding the `where search_index_fts match :query` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691369691,Create a view for running faceted searches,
https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685895540,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685895540,MDEyOklzc3VlQ29tbWVudDY4NTg5NTU0MA==,9599,simonw,2020-09-02T17:46:44Z,2020-09-02T17:46:44Z,MEMBER,"Some opet questions about this:
- Should I restrict to two exclusive categories here, or should I have a generic category mechanism that can be expanded to more than two?
- Should an item be able to exist in more than one category? Do I want to be able to mark an indexed item as both by-me and liked-by-me for example? This question is more interesting if the number of categories is greater than two.
- How should this be modeled? Single column, multiple boolean columns, JSON array, m2m against separate table?
- What's the best way to make this performant","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198,"Mechanism for differentiating between ""by me"" and ""liked by me""",
https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685121074,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2,685121074,MDEyOklzc3VlQ29tbWVudDY4NTEyMTA3NA==,9599,simonw,2020-09-01T20:42:00Z,2020-09-01T20:42:00Z,MEMBER,Documentation at the bottom of the Usage section here: https://github.com/dogsheep/dogsheep-beta/blob/0.2/README.md#usage,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689809225,Apply porter stemming,
https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685115519,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2,685115519,MDEyOklzc3VlQ29tbWVudDY4NTExNTUxOQ==,9599,simonw,2020-09-01T20:31:57Z,2020-09-01T20:31:57Z,MEMBER,"Actually this doesn't work: you can't turn on stemming for specific tables, because all of the content goes into a single `search_index` table which is configured the same way.
So stemming needs to be a global option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689809225,Apply porter stemming,
https://github.com/dogsheep/pocket-to-sqlite/issues/5#issuecomment-684425714,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5,684425714,MDEyOklzc3VlQ29tbWVudDY4NDQyNTcxNA==,9599,simonw,2020-09-01T06:18:32Z,2020-09-01T06:18:32Z,MEMBER,"Good suggestion, I'll setup a demo somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629473827,Set up a demo,
https://github.com/dogsheep/pocket-to-sqlite/issues/3#issuecomment-684424396,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3,684424396,MDEyOklzc3VlQ29tbWVudDY4NDQyNDM5Ng==,9599,simonw,2020-09-01T06:17:45Z,2020-09-01T06:17:45Z,MEMBER,It looks like I could ignore the `image` column and synthesize a unique key from the data in the `images` column using `$item_id/$image_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503243784,Extract images into separate tables,
https://github.com/dogsheep/dogsheep-beta/issues/4#issuecomment-684395444,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4,684395444,MDEyOklzc3VlQ29tbWVudDY4NDM5NTQ0NA==,9599,simonw,2020-09-01T06:00:03Z,2020-09-01T06:00:03Z,MEMBER,I ran `sqlite-utils optimize beta.db` against my test DB and the size reduced from 183M to 176M - and a 450ms search ran in 359ms. So not a huge improvement but still worthwhile.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689839399,Optimize the FTS table,
https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-684250044,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,684250044,MDEyOklzc3VlQ29tbWVudDY4NDI1MDA0NA==,9599,simonw,2020-09-01T05:01:09Z,2020-09-01T05:01:23Z,MEMBER,Maybe this starts out as a custom templated canned query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340,"Datasette plugin to provide custom page for running faceted, ranked searches",
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-683528149,https://api.github.com/repos/simonw/sqlite-utils/issues/147,683528149,MDEyOklzc3VlQ29tbWVudDY4MzUyODE0OQ==,9599,simonw,2020-08-31T03:17:26Z,2020-08-31T03:17:26Z,OWNER,"+1 to making this something that users can customize. An optional argument to the `Database` constructor would be a neat way to do this.
I think there's a terrifying way that we could find this value... we could perform a binary search for it! Open up a memory connection and try running different bulk inserts against it and catch the exceptions - then adjust and try again.
My hunch is that we could perform just 2 or 3 probes (maybe against carefully selected values) to find the highest value that works. If this process took less than a few ms to run I'd be happy to do it automatically when the class is instantiated (and let users disable that automatic proving by setting a value using the constructor argument).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/datasette/issues/948#issuecomment-683448569,https://api.github.com/repos/simonw/datasette/issues/948,683448569,MDEyOklzc3VlQ29tbWVudDY4MzQ0ODU2OQ==,9599,simonw,2020-08-30T17:39:09Z,2020-08-30T18:34:34Z,OWNER,"So the steps needed are:
- Download and extract latest CodeMirror zip file
- Rename `lib/codemirror.js` to `codemirror-5.57.0.js`
- Rename `lib/codemirror.css` to `codemirror-5.57.0.css`
- Rename `mode/sql/sql.js` to `codemirror-5.57.0-sql.js`
- Edit both JS files to make the top comment a `/* */` block
- Minify JavaScript files like this:
- `npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'`
- `npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js --comments '/LICENSE/'`
- Check that the LICENSE comment did indeed survive minification
- Minify CSS file like this:
- `npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css`
- Edit the `_codemirror.html` template to reference the new files
- `git rm` the old files, `git add` the new files","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-683452613,https://api.github.com/repos/simonw/datasette/issues/948,683452613,MDEyOklzc3VlQ29tbWVudDY4MzQ1MjYxMw==,9599,simonw,2020-08-30T18:16:28Z,2020-08-30T18:16:28Z,OWNER,I added documentation on how to upgrade CodeMirror for the future here: https://docs.datasette.io/en/latest/contributing.html#upgrading-codemirror,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/655#issuecomment-683449837,https://api.github.com/repos/simonw/datasette/issues/655,683449837,MDEyOklzc3VlQ29tbWVudDY4MzQ0OTgzNw==,9599,simonw,2020-08-30T17:51:38Z,2020-08-30T17:51:38Z,OWNER,I think was fixed by #948,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542553350,Copy and paste doesn't work reliably on iPhone for SQL editor,
https://github.com/simonw/datasette/issues/948#issuecomment-683449804,https://api.github.com/repos/simonw/datasette/issues/948,683449804,MDEyOklzc3VlQ29tbWVudDY4MzQ0OTgwNA==,9599,simonw,2020-08-30T17:51:18Z,2020-08-30T17:51:18Z,OWNER,Copy and paste on mobile safari seems to work now. #655 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-683448635,https://api.github.com/repos/simonw/datasette/issues/948,683448635,MDEyOklzc3VlQ29tbWVudDY4MzQ0ODYzNQ==,9599,simonw,2020-08-30T17:39:54Z,2020-08-30T17:39:54Z,OWNER,I'll wait for this to deploy to https://latest.datasette.io/ and then test it in various desktop and mobile browsers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-683445704,https://api.github.com/repos/simonw/datasette/issues/948,683445704,MDEyOklzc3VlQ29tbWVudDY4MzQ0NTcwNA==,9599,simonw,2020-08-30T17:11:58Z,2020-08-30T17:33:30Z,OWNER,"One catch: this stripped the license information from the top of the JS.
I fixed this by editing the license to be a single `/* ... */` block comment instead of multiple `//` lines and running this:
npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-683445114,https://api.github.com/repos/simonw/datasette/issues/948,683445114,MDEyOklzc3VlQ29tbWVudDY4MzQ0NTExNA==,9599,simonw,2020-08-30T17:06:39Z,2020-08-30T17:06:39Z,OWNER,"Minifying using `npx`:
```
npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js
npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js
npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/sqlite-utils/issues/145#issuecomment-683382252,https://api.github.com/repos/simonw/sqlite-utils/issues/145,683382252,MDEyOklzc3VlQ29tbWVudDY4MzM4MjI1Mg==,96218,simonwiles,2020-08-30T06:27:25Z,2020-08-30T06:27:52Z,CONTRIBUTOR,"Note: had to adjust the test above because trying to exhaust a `SQLITE_MAX_VARIABLE_NUMBER` of 250000 in 99 records requires 2526 columns, and trips the ` ""Rows can have a maximum of {} columns"".format(SQLITE_MAX_VARS)` check even before it trips the default `SQLITE_MAX_COLUMN` value (2000).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688659182,Bug when first record contains fewer columns than subsequent records,
https://github.com/simonw/datasette/issues/957#issuecomment-683357092,https://api.github.com/repos/simonw/datasette/issues/957,683357092,MDEyOklzc3VlQ29tbWVudDY4MzM1NzA5Mg==,9599,simonw,2020-08-30T00:15:51Z,2020-08-30T00:16:02Z,OWNER,"Weirdly even removing this single `datasette` import from `utils/asgi.py` didn't fix the circular import:
https://github.com/simonw/datasette/blob/44cf424a94a85b74552075272660bb96a7432661/datasette/utils/asgi.py#L1-L3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/957#issuecomment-683356440,https://api.github.com/repos/simonw/datasette/issues/957,683356440,MDEyOklzc3VlQ29tbWVudDY4MzM1NjQ0MA==,9599,simonw,2020-08-30T00:08:18Z,2020-08-30T00:10:26Z,OWNER,"Annoyingly this seems to be the line that causes the circular import:
```python
from .utils.asgi import Forbidden, NotFound, Response
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/957#issuecomment-683355993,https://api.github.com/repos/simonw/datasette/issues/957,683355993,MDEyOklzc3VlQ29tbWVudDY4MzM1NTk5Mw==,9599,simonw,2020-08-30T00:02:11Z,2020-08-30T00:04:18Z,OWNER,"I tried doing this and got this error:
```
(datasette) datasette % pytest
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
plugins: asyncio-0.14.0, timeout-1.4.2
collected 1 item / 23 errors
=========================================================================== ERRORS ===========================================================================
_____________________________________________________________ ERROR collecting tests/test_api.py _____________________________________________________________
ImportError while importing test module '/Users/simon/Dropbox/Development/datasette/tests/test_api.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_api.py:1: in
from datasette.plugins import DEFAULT_PLUGINS
datasette/__init__.py:2: in
from .app import Datasette
datasette/app.py:30: in
from .views.base import DatasetteError, ureg
datasette/views/base.py:12: in
from datasette.plugins import pm
datasette/plugins.py:26: in
mod = importlib.import_module(plugin)
/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
datasette/publish/heroku.py:2: in
from datasette import hookimpl
E ImportError: cannot import name 'hookimpl' from partially initialized module 'datasette' (most likely due to a circular import) (/Users/simon/Dropbox/Development/datasette/datasette/__init__.py)
```
That's with `datasette/__init__.py` looking like this:
```python
from datasette.version import __version_info__, __version__ # noqa
from .app import Datasette
from .utils.asgi import Forbidden, NotFound, Response
from .utils import actor_matches_allow, QueryInterrupted
from .hookspecs import hookimpl # noqa
from .hookspecs import hookspec # noqa
__all__ = [
""actor_matches_allow"",
""hookimpl"",
""hookspec"",
""QueryInterrupted"",
""Forbidden"",
""NotFound"",
""Response"",
""Datasette"",
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/957#issuecomment-683355598,https://api.github.com/repos/simonw/datasette/issues/957,683355598,MDEyOklzc3VlQ29tbWVudDY4MzM1NTU5OA==,9599,simonw,2020-08-29T23:55:10Z,2020-08-29T23:55:34Z,OWNER,"Of these I think I'm going to promote the following to being importable directly `from datasette`:
- `from datasette.app import Datasette`
- `from datasette.utils import QueryInterrupted`
- `from datasette.utils.asgi import Response, Forbidden, NotFound`
- `from datasette.utils import actor_matches_allow`
All of the rest are infrequently used enough (or clearly named enough) that I'm happy to leave them as-is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/957#issuecomment-683355508,https://api.github.com/repos/simonw/datasette/issues/957,683355508,MDEyOklzc3VlQ29tbWVudDY4MzM1NTUwOA==,9599,simonw,2020-08-29T23:54:01Z,2020-08-29T23:54:01Z,OWNER,"Reviewing https://github.com/search?q=user%3Asimonw+%22from+datasette%22&type=Code I spotted these others:
```python
# Various:
from datasette.utils import path_with_replaced_args
from datasette.plugins import pm
from datasette.utils import QueryInterrupted
from datasette.utils.asgi import Response, Forbidden, NotFound
# datasette-publish-vercel:
from datasette.publish.common import (
add_common_publish_arguments_and_options,
fail_if_publish_binary_not_installed
)
from datasette.utils import temporary_docker_directory
# datasette-insert
from datasette.utils import actor_matches_allow, sqlite3
# obsolete: russian-ira-facebook-ads-datasette
from datasette.utils import TableFilter
# simonw/museums
from datasette.utils.asgi import asgi_send
# datasette-media
from datasette.utils.asgi import Response, asgi_send_file
# datasette/tests/plugins/my_plugin.py
from datasette.facets import Facet
# datasette-graphql
from datasette.views.table import TableView
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/956#issuecomment-683214102,https://api.github.com/repos/simonw/datasette/issues/956,683214102,MDEyOklzc3VlQ29tbWVudDY4MzIxNDEwMg==,9599,simonw,2020-08-29T01:32:21Z,2020-08-29T01:32:21Z,OWNER,Maybe the bug here is the double colon?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-683213973,https://api.github.com/repos/simonw/datasette/issues/956,683213973,MDEyOklzc3VlQ29tbWVudDY4MzIxMzk3Mw==,9599,simonw,2020-08-29T01:31:39Z,2020-08-29T01:31:39Z,OWNER,"Here's how the old Travis mechanism worked: https://github.com/simonw/datasette/blob/52eabb019d4051084b21524bd0fd9c2731126985/.travis.yml#L41-L47
So I was assuming that the eqivalent of `$REPO:$TRAVIS_TAG` in GitHub Actions is `$REPO::${GITHUB_REF#refs/tags/}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-683212960,https://api.github.com/repos/simonw/datasette/issues/956,683212960,MDEyOklzc3VlQ29tbWVudDY4MzIxMjk2MA==,9599,simonw,2020-08-29T01:25:34Z,2020-08-29T01:25:34Z,OWNER,So I guess this bit is wrong: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L71-L73,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-683212421,https://api.github.com/repos/simonw/datasette/issues/956,683212421,MDEyOklzc3VlQ29tbWVudDY4MzIxMjQyMQ==,9599,simonw,2020-08-29T01:22:23Z,2020-08-29T01:22:23Z,OWNER,"Here's the error message again:
> invalid argument `""***/datasette::0.49a0""` for `""-t, --tag""` flag: invalid reference format","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-683212246,https://api.github.com/repos/simonw/datasette/issues/956,683212246,MDEyOklzc3VlQ29tbWVudDY4MzIxMjI0Ng==,9599,simonw,2020-08-29T01:21:26Z,2020-08-29T01:21:26Z,OWNER,I added this but I have no idea if I got it right or not: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L58-L63,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/955#issuecomment-683189334,https://api.github.com/repos/simonw/datasette/issues/955,683189334,MDEyOklzc3VlQ29tbWVudDY4MzE4OTMzNA==,9599,simonw,2020-08-28T23:30:48Z,2020-08-28T23:30:48Z,OWNER,Also https://github.com/simonw/datasette-copyable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687711713,Release updated datasette-atom and datasette-ics,
https://github.com/simonw/datasette/issues/955#issuecomment-683185861,https://api.github.com/repos/simonw/datasette/issues/955,683185861,MDEyOklzc3VlQ29tbWVudDY4MzE4NTg2MQ==,9599,simonw,2020-08-28T23:17:09Z,2020-08-28T23:17:09Z,OWNER,I released 0.49a0 which means I can update the main branches of those two plugins - I'll push a release of them once 0.49 is fully shipped.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687711713,Release updated datasette-atom and datasette-ics,
https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683180581,https://api.github.com/repos/simonw/sqlite-utils/issues/144,683180581,MDEyOklzc3VlQ29tbWVudDY4MzE4MDU4MQ==,9599,simonw,2020-08-28T22:57:04Z,2020-08-28T22:57:04Z,OWNER,"That worked! https://github.com/simonw/sqlite-utils/runs/1043640785?check_suite_focus=true
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688395275,Run some tests against numpy,
https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683179678,https://api.github.com/repos/simonw/sqlite-utils/issues/144,683179678,MDEyOklzc3VlQ29tbWVudDY4MzE3OTY3OA==,9599,simonw,2020-08-28T22:53:17Z,2020-08-28T22:53:17Z,OWNER,I'm going to try doing this as a GitHub Actions test matrix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688395275,Run some tests against numpy,
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-683178570,https://api.github.com/repos/simonw/sqlite-utils/issues/139,683178570,MDEyOklzc3VlQ29tbWVudDY4MzE3ODU3MA==,9599,simonw,2020-08-28T22:48:51Z,2020-08-28T22:48:51Z,OWNER,"Thanks @simonwiles, this is now released in 2.16.1: https://sqlite-utils.readthedocs.io/en/stable/changelog.html","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/143#issuecomment-683175491,https://api.github.com/repos/simonw/sqlite-utils/issues/143,683175491,MDEyOklzc3VlQ29tbWVudDY4MzE3NTQ5MQ==,9599,simonw,2020-08-28T22:37:15Z,2020-08-28T22:37:15Z,OWNER,"I'm going to start running black exclusively in the GitHub Actions workflow, rather than having it run by the unit tests themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688389933,Move to GitHub Actions CI,
https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683173375,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683173375,MDEyOklzc3VlQ29tbWVudDY4MzE3MzM3NQ==,9599,simonw,2020-08-28T22:29:02Z,2020-08-28T22:29:02Z,OWNER,Yeah I think that failure is actually because there's a brand new release of Black out and it subtly changes some of the formatting rules. I'll merge this and then run Black against the entire codebase.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172829,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683172829,MDEyOklzc3VlQ29tbWVudDY4MzE3MjgyOQ==,9599,simonw,2020-08-28T22:27:05Z,2020-08-28T22:27:05Z,OWNER,"Looks like it failed the ""black"" formatting test - possibly because there's a new release if black out. I'm going to merge despite that failure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172082,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683172082,MDEyOklzc3VlQ29tbWVudDY4MzE3MjA4Mg==,9599,simonw,2020-08-28T22:24:25Z,2020-08-28T22:24:25Z,OWNER,Thanks very much!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/119#issuecomment-683146200,https://api.github.com/repos/simonw/sqlite-utils/issues/119,683146200,MDEyOklzc3VlQ29tbWVudDY4MzE0NjIwMA==,9599,simonw,2020-08-28T21:05:37Z,2020-08-28T21:05:37Z,OWNER,Maybe use `transform_table()` in #114 for this? Would be less efficient as it would copy the whole table but it would reduce library complexity a bit.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770,Ability to remove a foreign key,
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682815377,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682815377,MDEyOklzc3VlQ29tbWVudDY4MjgxNTM3Nw==,96218,simonwiles,2020-08-28T16:14:58Z,2020-08-28T16:14:58Z,CONTRIBUTOR,"Thanks! And yeah, I had updating the docs on my list too :) Will try to get to it this afternoon (budgeting time is fraught with uncertainty at the moment!).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682771226,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682771226,MDEyOklzc3VlQ29tbWVudDY4Mjc3MTIyNg==,9599,simonw,2020-08-28T15:57:42Z,2020-08-28T15:57:42Z,OWNER,"That pull request should update this section of the documentation too:
> If you have more than one record to insert, the insert_all() method is a much more efficient way of inserting them. Just like insert() it will automatically detect the columns that should be created, but it will inspect the first batch of 100 items to help decide what those column types should be.
https://github.com/simonw/sqlite-utils/blob/ea87c2b943fdd162c42a900ac0aea5ecc2f4b9d9/docs/python-api.rst#L393","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682762911,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682762911,MDEyOklzc3VlQ29tbWVudDY4Mjc2MjkxMQ==,9599,simonw,2020-08-28T15:54:57Z,2020-08-28T15:55:20Z,OWNER,"Here's a suggested test update:
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index a8791c3..12fa2f2 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1074,6 +1074,13 @@ class Table(Queryable):
all_columns = list(sorted(all_columns))
if hash_id:
all_columns.insert(0, hash_id)
+ else:
+ all_columns += [
+ column
+ for record in chunk
+ for column in record
+ if column not in all_columns
+ ]
validate_column_names(all_columns)
first = False
# values is the list of insert data that is passed to the
diff --git a/tests/test_create.py b/tests/test_create.py
index a84eb8d..3a7fafc 100644
--- a/tests/test_create.py
+++ b/tests/test_create.py
@@ -707,13 +707,15 @@ def test_insert_thousands_using_generator(fresh_db):
assert 10000 == fresh_db[""test""].count
-def test_insert_thousands_ignores_extra_columns_after_first_100(fresh_db):
+def test_insert_thousands_adds_extra_columns_after_first_100(fresh_db):
+ # https://github.com/simonw/sqlite-utils/issues/139
fresh_db[""test""].insert_all(
[{""i"": i, ""word"": ""word_{}"".format(i)} for i in range(100)]
- + [{""i"": 101, ""extra"": ""This extra column should cause an exception""}]
+ + [{""i"": 101, ""extra"": ""Should trigger ALTER""}],
+ alter=True,
)
rows = fresh_db.execute_returning_dicts(""select * from test where i = 101"")
- assert [{""i"": 101, ""word"": None}] == rows
+ assert [{""i"": 101, ""word"": None, ""extra"": ""Should trigger ALTER""}] == rows
def test_insert_ignore(fresh_db):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/datasette/issues/954#issuecomment-682312736,https://api.github.com/repos/simonw/datasette/issues/954,682312736,MDEyOklzc3VlQ29tbWVudDY4MjMxMjczNg==,9599,simonw,2020-08-28T04:05:01Z,2020-08-28T04:05:10Z,OWNER,> It can also return a dictionary with the following keys. This format is **deprecated** as-of Datasette 0.49 and will be removed by Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687694947,Remove old register_output_renderer dict mechanism in Datasette 1.0,
https://github.com/simonw/datasette/issues/953#issuecomment-682312494,https://api.github.com/repos/simonw/datasette/issues/953,682312494,MDEyOklzc3VlQ29tbWVudDY4MjMxMjQ5NA==,9599,simonw,2020-08-28T04:03:56Z,2020-08-28T04:03:56Z,OWNER,"Documentation says that the old dictionary mechanism will be deprecated by 1.0:
https://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687681018,register_output_renderer render function should be able to return a Response,
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682285212,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682285212,MDEyOklzc3VlQ29tbWVudDY4MjI4NTIxMg==,9599,simonw,2020-08-28T02:12:51Z,2020-08-28T02:12:51Z,OWNER,"I'd be happy to accept a PR for this, provided it included updated unit tests that illustrate it working. I think this is a really good improvement.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682284908,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682284908,MDEyOklzc3VlQ29tbWVudDY4MjI4NDkwOA==,9599,simonw,2020-08-28T02:11:40Z,2020-08-28T02:11:40Z,OWNER,"This is deliberate behaviour, but I'm not at all attached to it - you're right in pointing out that it's actually pretty unexpected.
I'd be happy to change this behaviour so if you pass `alter=True` and then use `.insert_all()` on more than 100 rows it works as you would expect, instead of silently ignoring new columns past the first 100 rows. I don't expect that anyone would be depending on the current behaviour (ignore new columns after the first 100) such that this should be considered a backwards incompatible change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682182178,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682182178,MDEyOklzc3VlQ29tbWVudDY4MjE4MjE3OA==,96218,simonwiles,2020-08-27T20:46:18Z,2020-08-27T20:46:18Z,CONTRIBUTOR,"> I tried changing the batch_size argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.
So the reason for this is that the `batch_size` for import is limited (of necessity) here: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L1048
With regard to the issue of ignoring columns, however, I made a fork and hacked a temporary fix that looks like this:
https://github.com/simonwiles/sqlite-utils/commit/3901f43c6a712a1a3efc340b5b8d8fd0cbe8ee63
It doesn't seem to affect performance enormously (but I've not tested it thoroughly), and it now does what I need (and would expect, tbh), but it now fails the test here:
https://github.com/simonw/sqlite-utils/blob/main/tests/test_create.py#L710-L716
The existence of this test suggests that `insert_all()` is behaving as intended, of course. It seems odd to me that this would be a desirable default behaviour (let alone the only behaviour), and its not very prominently flagged-up, either.
@simonw is this something you'd be willing to look at a PR for? I assume you wouldn't want to change the default behaviour at this point, but perhaps an option could be provided, or at least a bit more of a warning in the docs. Are there oversights in the implementation that I've made?
Would be grateful for your thoughts! Thanks!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/datasette/issues/950#issuecomment-680374196,https://api.github.com/repos/simonw/datasette/issues/950,680374196,MDEyOklzc3VlQ29tbWVudDY4MDM3NDE5Ng==,9599,simonw,2020-08-26T00:43:50Z,2020-08-26T00:43:50Z,OWNER,"The problem with the term ""private"" is that it could be confused with the concept of databases that aren't visible to the public due to the permissions system - the ones that are displayed with the padlock icon e.g. on https://datasette-auth-passwords-demo.datasette.io/
So I think ""secret"" is a better term for these.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/950#issuecomment-680264202,https://api.github.com/repos/simonw/datasette/issues/950,680264202,MDEyOklzc3VlQ29tbWVudDY4MDI2NDIwMg==,9599,simonw,2020-08-25T20:53:13Z,2020-08-25T20:53:13Z,OWNER,Forcing people to spell out `datasette github.db --private private.db` isn't terrible though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/950#issuecomment-680263999,https://api.github.com/repos/simonw/datasette/issues/950,680263999,MDEyOklzc3VlQ29tbWVudDY4MDI2Mzk5OQ==,9599,simonw,2020-08-25T20:52:47Z,2020-08-25T20:52:47Z,OWNER,"Naming challenge: secret databases or private databases?
I prefer private. But `datasette -p` is already taken by `--port`. `datasette -s` is currently available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/950#issuecomment-680263427,https://api.github.com/repos/simonw/datasette/issues/950,680263427,MDEyOklzc3VlQ29tbWVudDY4MDI2MzQyNw==,9599,simonw,2020-08-25T20:51:30Z,2020-08-25T20:52:13Z,OWNER,"`datasette-graphql` currently dispatches requests through the `TableView` class, so if that couldn't access private databases then it would not be able to either. See also the concept for `datasette.get(...)` as an internal API in #943 - that might need to have a mechanism for also being able to query private databases, maybe `datasette.get(path, allow_private=True)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/949#issuecomment-679367931,https://api.github.com/repos/simonw/datasette/issues/949,679367931,MDEyOklzc3VlQ29tbWVudDY3OTM2NzkzMQ==,9599,simonw,2020-08-24T21:09:50Z,2020-08-24T21:09:50Z,OWNER,"I'm attracted to this because of how good GraphiQL is for auto-completing queries. But I realize there's a problem here: GraphQL is designed to be autocomplete-friendly, but SQL is not. If you type `select ` and it doesn't know what's going in the `from` clause it can't give you good column autocomplete, for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints,
https://github.com/simonw/datasette/issues/949#issuecomment-679363710,https://api.github.com/repos/simonw/datasette/issues/949,679363710,MDEyOklzc3VlQ29tbWVudDY3OTM2MzcxMA==,9599,simonw,2020-08-24T21:00:43Z,2020-08-24T21:00:43Z,OWNER,"I think this requires three extra files from https://github.com/codemirror/CodeMirror/tree/5.57.0/addon/hint
- `show-hint.css`
- `show-hint.js`
- `sql-hint.js`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints,
https://github.com/simonw/datasette/issues/948#issuecomment-679355426,https://api.github.com/repos/simonw/datasette/issues/948,679355426,MDEyOklzc3VlQ29tbWVudDY3OTM1NTQyNg==,9599,simonw,2020-08-24T20:43:07Z,2020-08-24T20:43:07Z,OWNER,"It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-679333717,https://api.github.com/repos/simonw/datasette/issues/948,679333717,MDEyOklzc3VlQ29tbWVudDY3OTMzMzcxNw==,9599,simonw,2020-08-24T19:55:59Z,2020-08-24T19:55:59Z,OWNER,CodeMirror 6 is in pre-release at the moment and is a complete rewrite. I'll stick with the 5.x series for now. https://github.com/codemirror/codemirror.next/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/sqlite-utils/issues/138#issuecomment-678732667,https://api.github.com/repos/simonw/sqlite-utils/issues/138,678732667,MDEyOklzc3VlQ29tbWVudDY3ODczMjY2Nw==,9599,simonw,2020-08-23T05:46:10Z,2020-08-23T05:46:10Z,OWNER,"Actually the `TEXT` column thing wasn't a `sqlite-utils` issue, it was unique to how `shapefile-to-spatialite` was creating the table when using the SpatiaLite extension.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684118950,extracts= doesn't configure foreign keys,
https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678508056,https://api.github.com/repos/simonw/sqlite-utils/issues/136,678508056,MDEyOklzc3VlQ29tbWVudDY3ODUwODA1Ng==,9599,simonw,2020-08-21T21:13:41Z,2020-08-21T21:13:41Z,OWNER,"The `--spatialite` option should be available for other useful commands too, refs #137.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642,--load-extension=spatialite shortcut option,
https://github.com/simonw/sqlite-utils/issues/137#issuecomment-678507502,https://api.github.com/repos/simonw/sqlite-utils/issues/137,678507502,MDEyOklzc3VlQ29tbWVudDY3ODUwNzUwMg==,9599,simonw,2020-08-21T21:13:19Z,2020-08-21T21:13:19Z,OWNER,Adding `--spatialite` too would be great for usability: #136,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683830416,--load-extension for other sqlite-utils commands,
https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678497497,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678497497,MDEyOklzc3VlQ29tbWVudDY3ODQ5NzQ5Nw==,9599,simonw,2020-08-21T21:06:26Z,2020-08-21T21:06:26Z,OWNER,"Ended up needing two skipIfs:
https://github.com/simonw/sqlite-utils/blob/7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d/tests/test_cli.py#L888-L893","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172,--load-extension option for sqlite-utils query,
https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678480969,https://api.github.com/repos/simonw/sqlite-utils/issues/136,678480969,MDEyOklzc3VlQ29tbWVudDY3ODQ4MDk2OQ==,9599,simonw,2020-08-21T20:33:45Z,2020-08-21T20:33:45Z,OWNER,"I think this should initialize SpatiaLite against the current database if it has not been initialized already.
Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126
```python
def init_spatialite(db, lib):
db.conn.enable_load_extension(True)
db.conn.load_extension(lib)
# Initialize SpatiaLite if not yet initialized
if ""spatial_ref_sys"" in db.table_names():
return
db.conn.execute(""select InitSpatialMetadata(1)"")
def ensure_table_has_geometry(db, table, table_srid):
if ""geometry"" not in db[table].columns_dict:
db.conn.execute(
""SELECT AddGeometryColumn(?, 'geometry', ?, 'GEOMETRY', 2);"",
[table, table_srid],
)
```
Not sure if I should add a utility function or CLI command for that `ensure_table_has_geometry` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642,--load-extension=spatialite shortcut option,
https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678479741,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678479741,MDEyOklzc3VlQ29tbWVudDY3ODQ3OTc0MQ==,9599,simonw,2020-08-21T20:30:37Z,2020-08-21T20:30:37Z,OWNER,Docs: https://github.com/simonw/sqlite-utils/blob/bf4c6b7c82fab6b2400e48424f8dac1ae2f0a2dc/docs/python-api.rst#finding-spatialite,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434,Code for finding SpatiaLite in the usual locations,
https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678476842,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678476842,MDEyOklzc3VlQ29tbWVudDY3ODQ3Njg0Mg==,9599,simonw,2020-08-21T20:23:13Z,2020-08-21T20:23:13Z,OWNER,I'm going to start with just the first two - I'm not convinced I understand the `.so.5` variants.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434,Code for finding SpatiaLite in the usual locations,
https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678476338,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678476338,MDEyOklzc3VlQ29tbWVudDY3ODQ3NjMzOA==,9599,simonw,2020-08-21T20:22:02Z,2020-08-21T20:22:02Z,OWNER,I think that adds it as `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172,--load-extension option for sqlite-utils query,
https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678475578,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678475578,MDEyOklzc3VlQ29tbWVudDY3ODQ3NTU3OA==,9599,simonw,2020-08-21T20:20:05Z,2020-08-21T20:20:05Z,OWNER,"https://github.com/simonw/cryptozoology/blob/2ad69168f3b78ebd90a2cbeea8136c9115e2a9b7/build_cryptids_database.py#L16-L22
```python
try_these = (
""mod_spatialite"",
""/usr/local/lib/mod_spatialite.dylib"",
""/usr/lib/x86_64-linux-gnu/mod_spatialite.so"",
""/usr/lib/x86_64-linux-gnu/libspatialite.so.5"",
""/usr/lib/x86_64-linux-gnu/libspatialite.so.7"",
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434,Code for finding SpatiaLite in the usual locations,
https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474928,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678474928,MDEyOklzc3VlQ29tbWVudDY3ODQ3NDkyOA==,9599,simonw,2020-08-21T20:18:33Z,2020-08-21T20:18:33Z,OWNER,"This should get me SpatiaLite in the GitHub Actions Ubuntu:
```
apt install libsqlite3-mod-spatialite
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172,--load-extension option for sqlite-utils query,
https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474018,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678474018,MDEyOklzc3VlQ29tbWVudDY3ODQ3NDAxOA==,9599,simonw,2020-08-21T20:16:20Z,2020-08-21T20:16:20Z,OWNER,"Trickiest part of this is how to write a test for it.
I'll do a `pytest.skipIf` that only executes the test if SpatiaLite is available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172,--load-extension option for sqlite-utils query,
https://github.com/simonw/datasette/issues/945#issuecomment-676556377,https://api.github.com/repos/simonw/datasette/issues/945,676556377,MDEyOklzc3VlQ29tbWVudDY3NjU1NjM3Nw==,9599,simonw,2020-08-19T17:21:16Z,2020-08-19T17:21:16Z,OWNER,Documented here: https://docs.datasette.io/en/latest/plugins.html#installing-plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",682005535,datasette install -U for upgrading packages,
https://github.com/simonw/datasette/issues/943#issuecomment-675889865,https://api.github.com/repos/simonw/datasette/issues/943,675889865,MDEyOklzc3VlQ29tbWVudDY3NTg4OTg2NQ==,9599,simonw,2020-08-19T06:57:00Z,2020-08-19T06:57:00Z,OWNER,Maybe `.get` vs `.get_html`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675889551,https://api.github.com/repos/simonw/datasette/issues/943,675889551,MDEyOklzc3VlQ29tbWVudDY3NTg4OTU1MQ==,9599,simonw,2020-08-19T06:56:06Z,2020-08-19T06:56:17Z,OWNER,"I'm leaning towards defaulting to JSON as the requested format - you can pass `format=""html""` if you want HTML.
But weird that it's different from the web UI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675884980,https://api.github.com/repos/simonw/datasette/issues/943,675884980,MDEyOklzc3VlQ29tbWVudDY3NTg4NDk4MA==,9599,simonw,2020-08-19T06:44:26Z,2020-08-19T06:44:26Z,OWNER,"Need to decide what to do about JSON responses.
When called from a template it's likely the intent will be to further loop through the JSON data returned. It would be annoying to have to run `json.loads` here.
Maybe a `.get_json()` method then? Or even return a response that has `.json()` and `.text` similar to `httpx` - or just return an `httpx` response.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/944#issuecomment-675830678,https://api.github.com/repos/simonw/datasette/issues/944,675830678,MDEyOklzc3VlQ29tbWVudDY3NTgzMDY3OA==,9599,simonw,2020-08-19T03:30:10Z,2020-08-19T03:30:10Z,OWNER,"These templates will need a way to raise a 404 - so that if the template itself is deciding if the page exists (for example using `datasette-template-sql` or the proposed `datasette.get()` method from #943 or the `graphql()` template function in https://github.com/simonw/datasette-graphql/issues/50) it can return a regular 404 page.
This can imitate the `custom_redirect()` function from https://docs.datasette.io/en/stable/custom_templates.html#custom-redirects:
```html+jinja
{{ custom_redirect(""https://github.com/simonw/datasette"", 301) }}
```
It could be as simple as this:
```
{{ raise_404(""Museum not found"") }}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/944#issuecomment-675829942,https://api.github.com/repos/simonw/datasette/issues/944,675829942,MDEyOklzc3VlQ29tbWVudDY3NTgyOTk0Mg==,9599,simonw,2020-08-19T03:27:25Z,2020-08-19T03:27:25Z,OWNER,"I created a template file called `templates/pages/museums/{slug}.html` and used the debugger to see if Jinja could see it. This worked:
```
(Pdb) self.ds.jinja_env.list_templates()
['500.html', '_codemirror.html', '_codemirror_foot.html', '_description_source_license.html', '_footer.html',
'_table.html', 'allow_debug.html', 'base.html', 'database.html', 'default:500.html', 'default:_codemirror.html',
'default:_codemirror_foot.html', 'default:_description_source_license.html', 'default:_footer.html',
'default:_table.html', 'default:allow_debug.html', 'default:base.html', 'default:database.html',
'default:index.html', 'default:logout.html', 'default:messages_debug.html', 'default:patterns.html',
'default:permissions_debug.html', 'default:query.html', 'default:row.html', 'default:show_json.html',
'default:table.html', 'forbidden.html', 'index.html', 'logout.html', 'messages_debug.html',
'pages/about.html', 'pages/museums/{slug}.html', 'patterns.html', 'permissions_debug.html',
'query.html', 'row.html', 'show_json.html', 'table.html']
```
The `pages/museums/{slug}.html` template is in that list.
Here's the implementation of that `list_templates()` method - it does some filesystem walking so it may be a bit expensive to run it on every request: https://github.com/pallets/jinja/blob/ca8b0b0287e320fe1f4a74f36910ef7ae3303d99/src/jinja2/loaders.py#L197-L212
But caching it would be pretty easy - either until the server is restarted or as an in-memory cache for a few seconds.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/943#issuecomment-675788203,https://api.github.com/repos/simonw/datasette/issues/943,675788203,MDEyOklzc3VlQ29tbWVudDY3NTc4ODIwMw==,9599,simonw,2020-08-19T00:46:08Z,2020-08-19T00:46:23Z,OWNER,Also fun: the inevitable plugin that exposes this to the template language - so Datasette templates can stitch together data from multiple other internal API calls. Fun way to take advantage of `async` support in Jinja.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675787416,https://api.github.com/repos/simonw/datasette/issues/943,675787416,MDEyOklzc3VlQ29tbWVudDY3NTc4NzQxNg==,9599,simonw,2020-08-19T00:42:38Z,2020-08-19T00:42:38Z,OWNER,"I just realised that this mechanism is kind of like being able to use microservices - make API calls within your application - except that everything runs in the same process against SQLite databases so calls will be _lightning fast_.
It also means that a plugin can add a new internal API to Datasette that's accessible to other plugins by registering a new route with `register_routes`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675753114,https://api.github.com/repos/simonw/datasette/issues/943,675753114,MDEyOklzc3VlQ29tbWVudDY3NTc1MzExNA==,9599,simonw,2020-08-18T22:34:55Z,2020-08-18T22:34:55Z,OWNER,"Maybe allow this:
response = await datasette.get(""/{database}/{table}.json"", database=database, table=table)
This could cause problems if users ever need to pass literal `{` in their paths. Maybe allow this too:
response = await datasette.get(""/{database}/{table}.json"", interpolate=False)
Not convinced this is useful - it's a bit unintuitive.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675752436,https://api.github.com/repos/simonw/datasette/issues/943,675752436,MDEyOklzc3VlQ29tbWVudDY3NTc1MjQzNg==,9599,simonw,2020-08-18T22:32:44Z,2020-08-18T22:32:44Z,OWNER,"One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted.
If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values.
Here's how the `row.html` table handles that right now: https://github.com/simonw/datasette/blob/b21ed237ab940768574c834aa5a7130724bd3a2d/datasette/templates/row.html#L19-L23
It would be an improvement to have this logic abstracted out somewhere and documented so plugins can use it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675751719,https://api.github.com/repos/simonw/datasette/issues/943,675751719,MDEyOklzc3VlQ29tbWVudDY3NTc1MTcxOQ==,9599,simonw,2020-08-18T22:30:27Z,2020-08-18T22:30:27Z,OWNER,"Right now calling `datasette.app()` instantiates an ASGI application - complete with a bunch of routes and wrappers - and returns that application object. Calling it twice instantiates another ASGI application.
I think a single `Datasette` instance should only ever create a single ASGI app - so the `.app()` method should cache the ASGI app that it returns the first time and return the same application again on future calls.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/915#issuecomment-675751136,https://api.github.com/repos/simonw/datasette/issues/915,675751136,MDEyOklzc3VlQ29tbWVudDY3NTc1MTEzNg==,9599,simonw,2020-08-18T22:28:36Z,2020-08-18T22:28:36Z,OWNER,I'm closing this in favour of an internal requests mechanism in #943.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671763164,Refactor TableView class so things like datasette-graphql can reuse the logic,
https://github.com/simonw/datasette/issues/943#issuecomment-675750845,https://api.github.com/repos/simonw/datasette/issues/943,675750845,MDEyOklzc3VlQ29tbWVudDY3NTc1MDg0NQ==,9599,simonw,2020-08-18T22:27:43Z,2020-08-18T22:27:43Z,OWNER,"What about authentication checks etc? Won't they run twice?
I think that's OK too, in fact it's desirable: think of the case of `datasette-graphql` where a bunch of different TableView calls are being made as part of the same GraphQL queries. Having those calls take advantage of finely grained per-table authentication and permission checks seems like a good feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675750382,https://api.github.com/repos/simonw/datasette/issues/943,675750382,MDEyOklzc3VlQ29tbWVudDY3NTc1MDM4Mg==,9599,simonw,2020-08-18T22:26:15Z,2020-08-18T22:26:15Z,OWNER,"Should internal requests executed in this way be handled by plugins that used the `asgi_wrapper()` hook?
Hard to be sure one way or the other. I'm worried about logging middleware triggering twice - but actually anyone doing serious logging of their Datasette instance is probably doing it in a different layer (uvicorn logs or nginx proxy or whatever) so they wouldn't be affected. There aren't any ASGI logging middlewares out there that I've seen.
Also: if you run into a situation where your stuff is breaking because `datasette.get()` is calling ASGI middleware twice you can fix it by running your ASGI middleware outside of the `asgi_wrapper` plugin hook mechanism.
So I think it DOES execute `asgi_wrapper()` middleware.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675749319,https://api.github.com/repos/simonw/datasette/issues/943,675749319,MDEyOklzc3VlQ29tbWVudDY3NTc0OTMxOQ==,9599,simonw,2020-08-18T22:23:01Z,2020-08-18T22:23:01Z,OWNER,"Actually no - `requests.get()` and `httpx.get()` prove that having a `.get()` method for an HTTP-related API isn't confusing to people at all.
`datasette.get()` it is.
(I'll probably add `datasette.post()` in the future too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675749076,https://api.github.com/repos/simonw/datasette/issues/943,675749076,MDEyOklzc3VlQ29tbWVudDY3NTc0OTA3Ng==,9599,simonw,2020-08-18T22:22:21Z,2020-08-18T22:22:21Z,OWNER,"Alternative name possibilities:
- `datasette.http_get(...)` - slightly misleading since it's not going over the HTTP protocol
- `datasette.internal_get(...)` - the `internal_` might suggest its not an API for external use, which isn't true - it's for plugins
- `datasette.get(...)` - clashes with `dict.get()` but I'm not at all sure that's a good reason not to use it","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675748573,https://api.github.com/repos/simonw/datasette/issues/943,675748573,MDEyOklzc3VlQ29tbWVudDY3NTc0ODU3Mw==,9599,simonw,2020-08-18T22:20:52Z,2020-08-18T22:20:52Z,OWNER,"Should it default to treating things as if they had the `.json` extension? There are use-cases for the non-JSON method, such as https://github.com/natbat/tidepools_near_me/commit/ec102c6da5a5d86f17628740d90b6365b671b5e1
I think I'm OK with people having to add `.json` to their internal calls. Maybe they could use `format=""json""`) as an optional parameter which would automatically handle the very weird edge-cases where you need to use `?_format=json` instead of `.json` (due to table names existing with a `.json` suffix).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675747878,https://api.github.com/repos/simonw/datasette/issues/943,675747878,MDEyOklzc3VlQ29tbWVudDY3NTc0Nzg3OA==,9599,simonw,2020-08-18T22:18:46Z,2020-08-18T22:19:12Z,OWNER,"Could be as simple as `response = await datasette.get(""/path/blah"")` - which could also be re-used by the implementation of the `datasette --get /` CLI option introduced in #927.
Bit weird calling it `.get()` since that clashes with Python's dictionary `.get()` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/915#issuecomment-675746544,https://api.github.com/repos/simonw/datasette/issues/915,675746544,MDEyOklzc3VlQ29tbWVudDY3NTc0NjU0NA==,9599,simonw,2020-08-18T22:14:41Z,2020-08-18T22:14:41Z,OWNER,"I'm actually pretty happy with how `datasette-graphql` works now - maybe the trick here is to redesign the JSON format in #782 such that it can be used as a documented interface by things like `datasette-graphql` and then ensure Datasette has a documented mechanism for dispatching internal requests.
I just did a horrible hack here that simulates an internal request, so supporting them as a feature would definitely make sense: https://github.com/natbat/tidepools_near_me/commit/ec102c6da5a5d86f17628740d90b6365b671b5e1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671763164,Refactor TableView class so things like datasette-graphql can reuse the logic,
https://github.com/simonw/datasette/issues/268#issuecomment-675725464,https://api.github.com/repos/simonw/datasette/issues/268,675725464,MDEyOklzc3VlQ29tbWVudDY3NTcyNTQ2NA==,9599,simonw,2020-08-18T21:18:07Z,2020-08-18T21:18:35Z,OWNER,"I want this on the table page - but that means that the table page will need to run a slightly more complex query since it needs access to a `rank` column to sort by - which it gets from running a join.
BUT... that join needs to be constructed in a way that keeps existing filters, `?_where=` clauses etc intact.
Here's a prototype using SQLite CTEs: https://register-of-members-interests.datasettes.com/regmem?sql=with+original+as+%28select+rowid%2C+*+from+items%29%0D%0Aselect%0D%0A++original.*%2C%0D%0A++items_fts.rank+as+items_fts_rank%0D%0Afrom%0D%0A++original+join+items_fts+on+original.rowid+%3D+items_fts.rowid%0D%0Awhere%0D%0A++items_fts+match+escape_fts%28%3Asearch%29%0D%0Aorder+by+items_fts_rank+desc+limit+10&search=hotel
```sql
with original as (
select
rowid,
*
from
items
)
select
original.*,
items_fts.rank as items_fts_rank
from
original
join items_fts on original.rowid = items_fts.rowid
where
items_fts match escape_fts(:search)
order by
items_fts_rank desc
limit
10
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/942#issuecomment-675720040,https://api.github.com/repos/simonw/datasette/issues/942,675720040,MDEyOklzc3VlQ29tbWVudDY3NTcyMDA0MA==,9599,simonw,2020-08-18T21:05:24Z,2020-08-18T21:05:24Z,OWNER,"Is `columns` the right key for this in the table metadata block? I might want to use that for initial values for `?_col=` in #615.
Alternative names:
- `column_descriptions`
- `column_info`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-675718593,https://api.github.com/repos/simonw/datasette/issues/942,675718593,MDEyOklzc3VlQ29tbWVudDY3NTcxODU5Mw==,9599,simonw,2020-08-18T21:02:11Z,2020-08-18T21:02:24Z,OWNER,"Easiest solution: if you provide column metadata it gets displayed above the table, something like on https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act
HTML `title=` tooltips are also added to the table headers, which won't be visible on touch devices but that's OK because the information is visible on the page already.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-675715472,https://api.github.com/repos/simonw/datasette/issues/942,675715472,MDEyOklzc3VlQ29tbWVudDY3NTcxNTQ3Mg==,9599,simonw,2020-08-18T20:55:02Z,2020-08-18T20:55:02Z,OWNER,"Could display these as tooltips on icons something like this (from the experimental `datasette-inspect-columns` plugin):
This would need to take accessibility into account, and would need a different display for the mobile web layout. Need to consider how it will interact with the column menu suggested in #690.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/873#issuecomment-675610275,https://api.github.com/repos/simonw/datasette/issues/873,675610275,MDEyOklzc3VlQ29tbWVudDY3NTYxMDI3NQ==,9599,simonw,2020-08-18T17:24:05Z,2020-08-18T17:26:10Z,OWNER,"Maybe I can do this with ASGI after all. Here's the output of `/-/asgi-scope` with `datasette-debug-asgi` installed:
```
{'asgi': {'spec_version': '2.1', 'version': '3.0'},
'client': ('127.0.0.1', 62035),
'headers': [(b'host', b'127.0.0.1:62029'),
(b'user-agent',
b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:79.0) Gecko'
b'/20100101 Firefox/79.0'),
(b'accept',
b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
b'webp,*/*;q=0.8'),
(b'accept-language', b'en-US,en;q=0.5'),
(b'accept-encoding', b'gzip, deflate'),
(b'dnt', b'1'),
(b'connection', b'keep-alive'),
(b'upgrade-insecure-requests', b'1'),
(b'cache-control', b'max-age=0')],
'http_version': '1.1',
'method': 'GET',
'path': '/-/asgi-scope',
'query_string': b'',
'raw_path': b'/-/asgi-scope',
'root_path': '',
'scheme': 'http',
'server': ('127.0.0.1', 62029),
'type': 'http'}
```
That `'server': ('127.0.0.1', 62029)` bit has the correct port. Question is, can I access that programmatically on server startup?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-675609109,https://api.github.com/repos/simonw/datasette/issues/873,675609109,MDEyOklzc3VlQ29tbWVudDY3NTYwOTEwOQ==,9599,simonw,2020-08-18T17:21:51Z,2020-08-18T17:21:51Z,OWNER,Asked about this on the encode gitter here: https://gitter.im/encode/community?at=5f3c0dcaa8c17801765940c0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/940#issuecomment-675538586,https://api.github.com/repos/simonw/datasette/issues/940,675538586,MDEyOklzc3VlQ29tbWVudDY3NTUzODU4Ng==,9599,simonw,2020-08-18T15:11:36Z,2020-08-18T15:11:36Z,OWNER,I tested this new publish pattern (running the tests in parallel before the deploy step) on `github-to-sqlite` - skipping the Docker step - and it worked: https://github.com/dogsheep/github-to-sqlite/actions/runs/213809864,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/dogsheep/github-to-sqlite/issues/47#issuecomment-675523053,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/47,675523053,MDEyOklzc3VlQ29tbWVudDY3NTUyMzA1Mw==,9599,simonw,2020-08-18T14:45:53Z,2020-08-18T14:45:53Z,MEMBER,"```
% github-to-sqlite emojis emojis.db --fetch
[########----------------------------] 397/1682 23% 00:03:43
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681086659,emojis command,
https://github.com/dogsheep/github-to-sqlite/issues/39#issuecomment-675509550,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/39,675509550,MDEyOklzc3VlQ29tbWVudDY3NTUwOTU1MA==,9599,simonw,2020-08-18T14:23:56Z,2020-08-18T14:23:56Z,MEMBER,I think this is fixed: https://github-to-sqlite.dogsheep.net/github/issues?_facet=repo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",613777056,issues foreign key to repo isn't working,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-675259273,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,675259273,MDEyOklzc3VlQ29tbWVudDY3NTI1OTI3Mw==,9599,simonw,2020-08-18T05:28:32Z,2020-08-18T05:28:32Z,MEMBER,"Oh that's interesting - i didn't realize ""reviews"" were a separate concept. I'd definitely accept a pull request adding those!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/simonw/datasette/issues/940#issuecomment-675253373,https://api.github.com/repos/simonw/datasette/issues/940,675253373,MDEyOklzc3VlQ29tbWVudDY3NTI1MzM3Mw==,9599,simonw,2020-08-18T05:10:17Z,2020-08-18T05:10:17Z,OWNER,I'll close this after the next release successfully goes out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-675251613,https://api.github.com/repos/simonw/datasette/issues/940,675251613,MDEyOklzc3VlQ29tbWVudDY3NTI1MTYxMw==,9599,simonw,2020-08-18T05:05:15Z,2020-08-18T05:05:15Z,OWNER,I think this is ready. I'll only know for sure the first time I push a release through it though!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-674590583,https://api.github.com/repos/simonw/datasette/issues/940,674590583,MDEyOklzc3VlQ29tbWVudDY3NDU5MDU4Mw==,9599,simonw,2020-08-16T23:15:51Z,2020-08-18T05:04:43Z,OWNER,This example of jobs depending on each other and sharing data via artifacts looks relevant: https://docs.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts#passing-data-between-jobs-in-a-workflow,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/pull/941#issuecomment-674566290,https://api.github.com/repos/simonw/datasette/issues/941,674566290,MDEyOklzc3VlQ29tbWVudDY3NDU2NjI5MA==,22429695,codecov[bot],2020-08-16T19:18:43Z,2020-08-18T05:04:31Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=h1) Report
> Merging [#941](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/52eabb019d4051084b21524bd0fd9c2731126985&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/941/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #941 +/- ##
=======================================
Coverage 84.10% 84.10%
=======================================
Files 28 28
Lines 3788 3788
=======================================
Hits 3186 3186
Misses 602 602
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=footer). Last update [52eabb0...f5a72e1](https://codecov.io/gh/simonw/datasette/pull/941?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679809281,"Run CI on GitHub Actions, not Travis",
https://github.com/simonw/datasette/issues/940#issuecomment-675250280,https://api.github.com/repos/simonw/datasette/issues/940,675250280,MDEyOklzc3VlQ29tbWVudDY3NTI1MDI4MA==,9599,simonw,2020-08-18T05:01:34Z,2020-08-18T05:01:42Z,OWNER,I think `${GITHUB_REF#refs/tags/}` is the equivalent of `$TRAVIS_TAG`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-674589472,https://api.github.com/repos/simonw/datasette/issues/940,674589472,MDEyOklzc3VlQ29tbWVudDY3NDU4OTQ3Mg==,9599,simonw,2020-08-16T23:05:57Z,2020-08-16T23:05:57Z,OWNER,When I figure this out I'll update the https://github.com/simonw/datasette-plugin/blob/main/datasette-%7B%7Bcookiecutter.hyphenated%7D%7D/.github/workflows/publish.yml default workflow to do this - right now it runs the tests once on just a single version of Python as part of the package deploy to PyPI step.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-674589321,https://api.github.com/repos/simonw/datasette/issues/940,674589321,MDEyOklzc3VlQ29tbWVudDY3NDU4OTMyMQ==,9599,simonw,2020-08-16T23:04:34Z,2020-08-16T23:04:34Z,OWNER,"https://docs.github.com/en/actions/getting-started-with-github-actions/core-concepts-for-github-actions#job
> A set of steps that execute on the same runner. You can define the dependency rules for how jobs run in a workflow file. Jobs can run at the same time in parallel or run sequentially depending on the status of a previous job. For example, a workflow can have two sequential jobs that build and test code, where the test job is dependent on the status of the build job. If the build job fails, the test job will not run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-674589035,https://api.github.com/repos/simonw/datasette/issues/940,674589035,MDEyOklzc3VlQ29tbWVudDY3NDU4OTAzNQ==,9599,simonw,2020-08-16T23:02:23Z,2020-08-16T23:02:23Z,OWNER,"I'd like to set these up as different workflows that depend on each other, if that's possible.
I want to start three test runs in parallel (on three different Python versions), then if all three pass kick off the PyPI push (without running more tests), then if that passes do the Docker build and push.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/914#issuecomment-674578388,https://api.github.com/repos/simonw/datasette/issues/914,674578388,MDEyOklzc3VlQ29tbWVudDY3NDU3ODM4OA==,9599,simonw,2020-08-16T21:10:27Z,2020-08-16T21:10:27Z,OWNER,"Demo of the fix: https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array&_nl=on
```
{""rowid"": 2, ""data"": {""$base64"": true, ""encoded"": ""FRwDx60F/g==""}}
{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671056788,"""Object of type bytes is not JSON serializable"" for _nl=on",
https://github.com/simonw/datasette/issues/940#issuecomment-674566618,https://api.github.com/repos/simonw/datasette/issues/940,674566618,MDEyOklzc3VlQ29tbWVudDY3NDU2NjYxOA==,9599,simonw,2020-08-16T19:20:58Z,2020-08-16T19:20:58Z,OWNER,I need to figure out how to build and push the Docker image on releases. Here's the Travis code for that: https://github.com/simonw/datasette/blob/52eabb019d4051084b21524bd0fd9c2731126985/.travis.yml#L38-L47,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/938#issuecomment-674558631,https://api.github.com/repos/simonw/datasette/issues/938,674558631,MDEyOklzc3VlQ29tbWVudDY3NDU1ODYzMQ==,9599,simonw,2020-08-16T18:10:23Z,2020-08-16T18:10:23Z,OWNER,Documentation: https://github.com/simonw/datasette/blob/3a4c8ed36aa97211e46849d32a09f2f386f342dd/docs/plugin_hooks.rst#extra-template-vars-template-database-table-columns-view-name-request-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679700269,Pass columns to extra CSS/JS/etc plugin hooks,
https://github.com/simonw/datasette/issues/938#issuecomment-674551826,https://api.github.com/repos/simonw/datasette/issues/938,674551826,MDEyOklzc3VlQ29tbWVudDY3NDU1MTgyNg==,9599,simonw,2020-08-16T17:07:57Z,2020-08-16T17:07:57Z,OWNER,extra_tenplate_vars should he documented first and should be the only one to document the arguments and the async return feature. Others should refer back to it rather than duplicating that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679700269,Pass columns to extra CSS/JS/etc plugin hooks,
https://github.com/simonw/datasette/issues/939#issuecomment-674548163,https://api.github.com/repos/simonw/datasette/issues/939,674548163,MDEyOklzc3VlQ29tbWVudDY3NDU0ODE2Mw==,9599,simonw,2020-08-16T16:34:30Z,2020-08-16T16:34:30Z,OWNER,Docs also need to note that `request` may be `None` for all of these hooks.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674547811,https://api.github.com/repos/simonw/datasette/issues/939,674547811,MDEyOklzc3VlQ29tbWVudDY3NDU0NzgxMQ==,9599,simonw,2020-08-16T16:31:20Z,2020-08-16T16:31:20Z,OWNER,And the docs need to be updated too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674547788,https://api.github.com/repos/simonw/datasette/issues/939,674547788,MDEyOklzc3VlQ29tbWVudDY3NDU0Nzc4OA==,9599,simonw,2020-08-16T16:31:08Z,2020-08-16T16:31:08Z,OWNER,Also: they should all be able to return `async def` inner functions. At the moment only `extra_template_vars` supports that pattern.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674545058,https://api.github.com/repos/simonw/datasette/issues/939,674545058,MDEyOklzc3VlQ29tbWVudDY3NDU0NTA1OA==,9599,simonw,2020-08-16T16:07:20Z,2020-08-16T16:07:20Z,OWNER,"I'm going to implement this first as a single commit, then implement `columns` from #938 as a separate change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674544973,https://api.github.com/repos/simonw/datasette/issues/939,674544973,MDEyOklzc3VlQ29tbWVudDY3NDU0NDk3Mw==,9599,simonw,2020-08-16T16:06:34Z,2020-08-16T16:06:34Z,OWNER,"They should also be next to each other in the documentation - right now they're a bit scattered in terms of page order:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674544875,https://api.github.com/repos/simonw/datasette/issues/939,674544875,MDEyOklzc3VlQ29tbWVudDY3NDU0NDg3NQ==,9599,simonw,2020-08-16T16:05:36Z,2020-08-16T16:05:36Z,OWNER,"They should all take:
template, database, table, view_name, request, datasette, columns
`columns` is new, see #938.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/938#issuecomment-674544691,https://api.github.com/repos/simonw/datasette/issues/938,674544691,MDEyOklzc3VlQ29tbWVudDY3NDU0NDY5MQ==,9599,simonw,2020-08-16T16:03:52Z,2020-08-16T16:03:52Z,OWNER,"Four plugin hooks need this extra `columns` argument:
- `extra_css_urls(template, database, table, datasette)`
- `extra_js_urls(template, database, table, datasette)`
- `extra_body_script(template, database, table, view_name, datasette)`
- `extra_template_vars(template, database, table, view_name, request, datasette)`
It's weird that these take different arguments. I should fix that too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679700269,Pass columns to extra CSS/JS/etc plugin hooks,
https://github.com/simonw/datasette/pull/936#issuecomment-674453772,https://api.github.com/repos/simonw/datasette/issues/936,674453772,MDEyOklzc3VlQ29tbWVudDY3NDQ1Mzc3Mg==,22429695,codecov[bot],2020-08-15T22:35:29Z,2020-08-15T22:35:29Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=h1) Report
> Merging [#936](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/936/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #936 +/- ##
==========================================
+ Coverage 84.02% 84.04% +0.02%
==========================================
Files 28 28
Lines 3774 3774
==========================================
+ Hits 3171 3172 +1
+ Misses 603 602 -1
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/936/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.18% <0.00%> (+0.18%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=footer). Last update [13b3b51...94a68b9](https://codecov.io/gh/simonw/datasette/pull/936?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679650632,Don't hang in db.execute_write_fn() if connection fails,
https://github.com/simonw/datasette/pull/936#issuecomment-674453318,https://api.github.com/repos/simonw/datasette/issues/936,674453318,MDEyOklzc3VlQ29tbWVudDY3NDQ1MzMxOA==,9599,simonw,2020-08-15T22:29:15Z,2020-08-15T22:32:58Z,OWNER,"Test failure:
```
tests/test_canned_queries.py F
>>> captured stdout >>>
__enter__
>>> traceback >>>
canned_write_client =
def test_insert(canned_write_client):
response = canned_write_client.post(
""/data/add_name"",
{""name"": ""Hello""},
allow_redirects=False,
csrftoken_from=True,
cookies={""foo"": ""bar""},
)
assert 302 == response.status
> assert ""/data/add_name?success"" == response.headers[""Location""]
E AssertionError: assert '/data/add_name?success' == '/data/add_name'
E - /data/add_name
E + /data/add_name?success
E ? ++++++++
/Users/simon/Dropbox/Development/datasette/tests/test_canned_queries.py:66: AssertionError
```
No idea why this change would affect that test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679650632,Don't hang in db.execute_write_fn() if connection fails,
https://github.com/simonw/datasette/issues/935#issuecomment-674451012,https://api.github.com/repos/simonw/datasette/issues/935,674451012,MDEyOklzc3VlQ29tbWVudDY3NDQ1MTAxMg==,9599,simonw,2020-08-15T21:56:13Z,2020-08-15T21:56:13Z,OWNER,"This implementation seems to fix it, need to work out how to test though.
```diff
diff --git a/datasette/database.py b/datasette/database.py
index ffa7a79..7ba1456 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -89,14 +89,22 @@ class Database:
def _execute_writes(self):
# Infinite looping thread that protects the single write connection
# to this database
- conn = self.connect(write=True)
+ conn_exception = None
+ conn = None
+ try:
+ conn = self.connect(write=True)
+ except Exception as e:
+ conn_exception = e
while True:
task = self._write_queue.get()
- try:
- result = task.fn(conn)
- except Exception as e:
- print(e)
- result = e
+ if conn_exception is not None:
+ result = conn_exception
+ else:
+ try:
+ result = task.fn(conn)
+ except Exception as e:
+ print(e)
+ result = e
task.reply_queue.sync_q.put(result)
async def execute_fn(self, fn):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710,"db.execute_write_fn(create_tables, block=True) hangs a thread if connection fails",
https://github.com/simonw/datasette/issues/935#issuecomment-674450652,https://api.github.com/repos/simonw/datasette/issues/935,674450652,MDEyOklzc3VlQ29tbWVudDY3NDQ1MDY1Mg==,9599,simonw,2020-08-15T21:51:22Z,2020-08-15T21:51:22Z,OWNER,"The easiest way to recreate this is to attempt a write against an immutable database, which triggers this assertion error:
https://github.com/simonw/datasette/blob/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8/datasette/database.py#L47-L55","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710,"db.execute_write_fn(create_tables, block=True) hangs a thread if connection fails",
https://github.com/simonw/datasette/issues/935#issuecomment-674450607,https://api.github.com/repos/simonw/datasette/issues/935,674450607,MDEyOklzc3VlQ29tbWVudDY3NDQ1MDYwNw==,9599,simonw,2020-08-15T21:50:41Z,2020-08-15T21:50:41Z,OWNER,"The bug is here:
https://github.com/simonw/datasette/blob/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8/datasette/database.py#L89-L100
If `conn = self.connect(write=True)` raises an exception the entire server hangs, like this:
```
% datasette -i fixtures.db --get /
Exception in thread Thread-1:
Traceback (most recent call last):
File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py"", line 932, in _bootstrap_inner
self.run()
File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py"", line 870, in run
self._target(*self._args, **self._kwargs)
File ""/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py"", line 92, in _execute_writes
conn = self.connect(write=True)
File ""/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py"", line 55, in connect
assert not (write and not self.is_mutable)
AssertionError
... server hangs here ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710,"db.execute_write_fn(create_tables, block=True) hangs a thread if connection fails",
https://github.com/simonw/datasette/issues/932#issuecomment-674144798,https://api.github.com/repos/simonw/datasette/issues/932,674144798,MDEyOklzc3VlQ29tbWVudDY3NDE0NDc5OA==,9599,simonw,2020-08-14T16:02:24Z,2020-08-14T16:02:24Z,OWNER,"Things to go in here:
- What is Datasette?
- A *database* contains *tables* full of *records*. A table has *rows* and *columns*.
- Understanding faceting
- How to use the filter interface
- How to export data
- How to link to data
- How to run SQL","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation,
https://github.com/simonw/datasette/issues/932#issuecomment-673735299,https://api.github.com/repos/simonw/datasette/issues/932,673735299,MDEyOklzc3VlQ29tbWVudDY3MzczNTI5OQ==,9599,simonw,2020-08-13T22:10:40Z,2020-08-13T22:11:06Z,OWNER,"Idea: plugins can provide their own user-facing documentation. Datasette can like to eg `datasette.io/help?plugins=datasette-vega,datasette-cluster-map` to get the user manual with extra sections for those plugins.
Or... link to `?url=datasette-url` and the documentation site can hit `/-/plugins.json` to figure out what extra manual sections to display!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation,
https://github.com/simonw/datasette/issues/932#issuecomment-673734387,https://api.github.com/repos/simonw/datasette/issues/932,673734387,MDEyOklzc3VlQ29tbWVudDY3MzczNDM4Nw==,9599,simonw,2020-08-13T22:08:06Z,2020-08-13T22:08:06Z,OWNER,One challenge: how does this interact with plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation,
https://github.com/simonw/datasette/issues/932#issuecomment-673733904,https://api.github.com/repos/simonw/datasette/issues/932,673733904,MDEyOklzc3VlQ29tbWVudDY3MzczMzkwNA==,9599,simonw,2020-08-13T22:06:50Z,2020-08-13T22:06:50Z,OWNER,Title: **Using Datasette**. `using.rst`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation,
https://github.com/simonw/datasette/issues/931#issuecomment-673123213,https://api.github.com/repos/simonw/datasette/issues/931,673123213,MDEyOklzc3VlQ29tbWVudDY3MzEyMzIxMw==,9599,simonw,2020-08-12T21:36:20Z,2020-08-12T21:36:20Z,OWNER,That worked: https://hub.docker.com/r/datasetteproject/datasette/tags,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673104851,https://api.github.com/repos/simonw/datasette/issues/931,673104851,MDEyOklzc3VlQ29tbWVudDY3MzEwNDg1MQ==,9599,simonw,2020-08-12T20:52:08Z,2020-08-12T20:52:08Z,OWNER,"```
docker run -p 8001:8001 -v `pwd`:/mnt \
datasette-updated-spatialite \
datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db --load-extension=/usr/local/lib/mod_spatialite.so
```
This seems to work `/-/versions` reports the SpatiaLite versions as expected:
```
""extensions"": {
""json1"": null,
""spatialite"": ""4.4.0-RC0""
},
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673088110,https://api.github.com/repos/simonw/datasette/issues/931,673088110,MDEyOklzc3VlQ29tbWVudDY3MzA4ODExMA==,9599,simonw,2020-08-12T20:15:28Z,2020-08-12T20:15:28Z,OWNER,"I changed the Dockerfile and built it on my laptop using:
docker build -f Dockerfile -t datasette-updated-spatialite .
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673074297,https://api.github.com/repos/simonw/datasette/issues/931,673074297,MDEyOklzc3VlQ29tbWVudDY3MzA3NDI5Nw==,9599,simonw,2020-08-12T19:46:29Z,2020-08-12T19:46:29Z,OWNER,"Looks like the old files have moved to:
- http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/
- http://www.gaia-gis.it/gaia-sins/libspatialite-sources/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673070308,https://api.github.com/repos/simonw/datasette/issues/931,673070308,MDEyOklzc3VlQ29tbWVudDY3MzA3MDMwOA==,9599,simonw,2020-08-12T19:37:55Z,2020-08-12T19:37:55Z,OWNER,"Project news here: https://groups.google.com/g/spatialite-users/c/8AG3hQzXDmo
> FreeXL
> ==============================
> the current stable version is now 1.0.6
> (supporting few minor bug fixes and refreshed
> build scripts)
> DONE: no further development is expected in
> the near future.
>
> http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz
> http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.zip
>
>
> ReadOSM
> ===============================
> the current stable version is now 1.1.0a
> (supporting few minor bug fixes and refreshed
> build scripts)
> DONE: no further development is expected in
> the near future.
>
> http://www.gaia-gis.it/gaia-sins/readosm-1.1.0a.tar.gz
> http://www.gaia-gis.it/gaia-sins/readosm-1.1.0a.zip
>
>
> VirtualPG
> ===============================
> the current stanle version is now 2.0.0
> (supporting few minor bug fixes and refreshed
> build scripts)
> DONE: no further development is expected in
> the near future.
>
> http://www.gaia-gis.it/gaia-sins/virtualpg-2.0.0.tar.gz
> http://www.gaia-gis.it/gaia-sins/virtualpg-2.0.0.zip
>
>
> libspatialite
> ===============================
> 5.0.0 Release Candidate 1 (RC1) is now ready
> to become a stable release.
> I'll simply wait for more or less a week so to
> allow for a reasonable community testing; if
> no critical issue will be reported in the meanwhile
> I'll go ASAP to release 5.0.0 ""stable""
>
> http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.0-RC1.tar.gz
> http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.0-RC1.zip
>
>
> librasterlite2
> ================================
> still under active development.
> a very relevant milestone has been achieved,
> now the DB layout required for fully integrating
> libspatialte and librasterlie2 is definetely
> consolidated, no further changes are expected.
>
> http://www.gaia-gis.it/gaia-sins/librasterlite2-sources/librasterlite2-1.1.0-beta1.tar.gz
> http://www.gaia-gis.it/gaia-sins/librasterlite2-sources/librasterlite2-1.1.0-beta1.zip
>
> more development activity is still required for
> fully implementing SQL-driven graphic rendering
> of both Vector and Raster Coverages based on
> standard SLD/SE styles.
>
> a reasonable estimate is two man-month (and
> not necessarily one development month equals
> one calendar month)
>
>
> spatialite_gui
> =================================
> the GUI tool is expected to take full profit from
> the advanced features of RasterLite2 so to
> become a complete self-contained GIS viewer
> with map editing capabilities.
>
> consequently it wiil surely be the last
> member of the SpatiaLite family to be
> released in a final stable form.
> several transient development versions
> will be possibly released so to closely
> follow the evolution of RasterLite2.
>
> http://www.gaia-gis.it/gaia-sins/spatialite-gui-sources/spatialite_gui-2.1.0-beta1.tar.gz
> http://www.gaia-gis.it/gaia-sins/spatialite-gui-sources/spatialite_gui-2.1.0-beta1.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673068919,https://api.github.com/repos/simonw/datasette/issues/931,673068919,MDEyOklzc3VlQ29tbWVudDY3MzA2ODkxOQ==,9599,simonw,2020-08-12T19:34:57Z,2020-08-12T19:34:57Z,OWNER,"Looks like SpatiaLite released new versions: https://www.gaia-gis.it/fossil/freexl/index says ""Sources: current version is 1.0.6 (released on 2020-08-02)"" and links to http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673068327,https://api.github.com/repos/simonw/datasette/issues/931,673068327,MDEyOklzc3VlQ29tbWVudDY3MzA2ODMyNw==,9599,simonw,2020-08-12T19:33:42Z,2020-08-12T19:33:42Z,OWNER,"https://hub.docker.com/r/datasetteproject/datasette/tags shows this:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/sqlite-utils/issues/133#issuecomment-672997703,https://api.github.com/repos/simonw/sqlite-utils/issues/133,672997703,MDEyOklzc3VlQ29tbWVudDY3Mjk5NzcwMw==,9599,simonw,2020-08-12T17:05:06Z,2020-08-12T17:05:06Z,OWNER,Released.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677839979,Release a sdist to PyPI,
https://github.com/simonw/datasette/issues/930#issuecomment-672550662,https://api.github.com/repos/simonw/datasette/issues/930,672550662,MDEyOklzc3VlQ29tbWVudDY3MjU1MDY2Mg==,9599,simonw,2020-08-12T03:30:59Z,2020-08-12T03:30:59Z,OWNER,https://datasette.readthedocs.io/en/stable/changelog.html#v0-47-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672519787,https://api.github.com/repos/simonw/datasette/issues/930,672519787,MDEyOklzc3VlQ29tbWVudDY3MjUxOTc4Nw==,9599,simonw,2020-08-12T02:52:46Z,2020-08-12T02:52:46Z,OWNER,"Homebrew install is now fixed, using a temporary URL while waiting for Travis to ship the new release to PyPI: https://github.com/simonw/homebrew-datasette/issues/6","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672488293,https://api.github.com/repos/simonw/datasette/issues/930,672488293,MDEyOklzc3VlQ29tbWVudDY3MjQ4ODI5Mw==,9599,simonw,2020-08-12T02:35:34Z,2020-08-12T02:35:34Z,OWNER,"OK, this has fixed it - I used `sdist` to create the `.tar.gz` and then `pip installed` it into a new environment, and everything worked just fine. Going to ship this as Datasette 0.47.1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672480811,https://api.github.com/repos/simonw/datasette/issues/930,672480811,MDEyOklzc3VlQ29tbWVudDY3MjQ4MDgxMQ==,9599,simonw,2020-08-12T02:31:38Z,2020-08-12T02:31:38Z,OWNER,The root cause appears to be that the `templates/` are missing from the `sdist` `.tar.gz`: https://github.com/simonw/homebrew-datasette/issues/6#issuecomment-672477352,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672472518,https://api.github.com/repos/simonw/datasette/issues/930,672472518,MDEyOklzc3VlQ29tbWVudDY3MjQ3MjUxOA==,9599,simonw,2020-08-12T02:24:49Z,2020-08-12T02:24:49Z,OWNER,I checked and `datasette publish` deploys a 0.47 version that works fine too: https://datasette-0-47-j7hipcg4aq-uc.a.run.app/-/versions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672471431,https://api.github.com/repos/simonw/datasette/issues/930,672471431,MDEyOklzc3VlQ29tbWVudDY3MjQ3MTQzMQ==,9599,simonw,2020-08-12T02:21:19Z,2020-08-12T02:21:19Z,OWNER,"Correction: pip installed Datasette works fine, it's just the Homebrew release that is broken.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/926#issuecomment-672393737,https://api.github.com/repos/simonw/datasette/issues/926,672393737,MDEyOklzc3VlQ29tbWVudDY3MjM5MzczNw==,9599,simonw,2020-08-12T00:26:17Z,2020-08-12T00:26:17Z,OWNER,"```
$ datasette --get '/:memory:.json?sql=select+sqlite_version()' | jq .
{
""database"": "":memory:"",
""query_name"": null,
""rows"": [
[
""3.32.3""
]
],
""truncated"": false,
""columns"": [
""sqlite_version()""
],
""query"": {
""sql"": ""select sqlite_version()"",
""params"": {}
},
""private"": false,
""allow_execute_sql"": true,
""query_ms"": 1.165628433227539
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677250834,"datasette fixtures.db --get ""/fixtures.json""",
https://github.com/simonw/datasette/pull/927#issuecomment-672375214,https://api.github.com/repos/simonw/datasette/issues/927,672375214,MDEyOklzc3VlQ29tbWVudDY3MjM3NTIxNA==,22429695,codecov[bot],2020-08-12T00:01:23Z,2020-08-12T00:23:45Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=h1) Report
> Merging [#927](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/83eda049af3f38d4289118d3576f96b2535084b1&el=desc) will **increase** coverage by `0.39%`.
> The diff coverage is `100.00%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/927/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## main #927 +/- ##
==========================================
+ Coverage 83.62% 84.02% +0.39%
==========================================
Files 27 28 +1
Lines 3682 3774 +92
==========================================
+ Hits 3079 3171 +92
Misses 603 603
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/927/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `75.13% <100.00%> (+1.00%)` | :arrow_up: |
| [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/927/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `100.00% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=footer). Last update [83eda04...2111da0](https://codecov.io/gh/simonw/datasette/pull/927?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/pull/927#issuecomment-672391299,https://api.github.com/repos/simonw/datasette/issues/927,672391299,MDEyOklzc3VlQ29tbWVudDY3MjM5MTI5OQ==,9599,simonw,2020-08-12T00:19:20Z,2020-08-12T00:19:20Z,OWNER,Docs: https://github.com/simonw/datasette/blob/2111da01a03cfc62303b6a4b59ea9f96d22c0f78/docs/getting_started.rst#datasette---get,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/pull/927#issuecomment-672382108,https://api.github.com/repos/simonw/datasette/issues/927,672382108,MDEyOklzc3VlQ29tbWVudDY3MjM4MjEwOA==,9599,simonw,2020-08-12T00:09:18Z,2020-08-12T00:09:18Z,OWNER,Documentation can go here: https://datasette.readthedocs.io/en/latest/getting_started.html#datasette-serve-options,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/issues/928#issuecomment-672373061,https://api.github.com/repos/simonw/datasette/issues/928,672373061,MDEyOklzc3VlQ29tbWVudDY3MjM3MzA2MQ==,9599,simonw,2020-08-11T23:56:19Z,2020-08-11T23:56:19Z,OWNER,"New implementation of the `install` command:
https://github.com/simonw/datasette/blob/afdeda8216d4d3027f87583ccdbef17ad85022ef/datasette/cli.py#L235-L240","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618,Test failures caused by failed attempts to mock pip,
https://github.com/simonw/datasette/issues/928#issuecomment-672372465,https://api.github.com/repos/simonw/datasette/issues/928,672372465,MDEyOklzc3VlQ29tbWVudDY3MjM3MjQ2NQ==,9599,simonw,2020-08-11T23:54:28Z,2020-08-11T23:54:28Z,OWNER,"While debugging this I found a useful clue in https://github.com/pypa/pip/blob/e060970d51c5946beac8447eb95585d83019582d/src/pip/_internal/cli/main.py#L23-L47
```
# Do not import and use main() directly! Using it directly is actively
# discouraged by pip's maintainers. The name, location and behavior of
# this function is subject to change, so calling it directly is not
# portable across different pip versions.
# In addition, running pip in-process is unsupported and unsafe. This is
# elaborated in detail at
# https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program.
# That document also provides suggestions that should work for nearly
# all users that are considering importing and using main() directly.
# However, we know that certain users will still want to invoke pip
# in-process. If you understand and accept the implications of using pip
# in an unsupported manner, the best approach is to use runpy to avoid
# depending on the exact location of this entry point.
# The following example shows how to use runpy to invoke pip in that
# case:
#
# sys.argv = [""pip"", your, args, here]
# runpy.run_module(""pip"", run_name=""__main__"")
#
# Note that this will exit the process after running, unlike a direct
# call to main. As it is not safe to do any processing after calling
# main, this should not be an issue in practice.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618,Test failures caused by failed attempts to mock pip,
https://github.com/simonw/datasette/issues/928#issuecomment-672372197,https://api.github.com/repos/simonw/datasette/issues/928,672372197,MDEyOklzc3VlQ29tbWVudDY3MjM3MjE5Nw==,9599,simonw,2020-08-11T23:53:38Z,2020-08-11T23:53:38Z,OWNER,Caused by the tests for #925,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618,Test failures caused by failed attempts to mock pip,
https://github.com/simonw/datasette/pull/927#issuecomment-672357176,https://api.github.com/repos/simonw/datasette/issues/927,672357176,MDEyOklzc3VlQ29tbWVudDY3MjM1NzE3Ng==,9599,simonw,2020-08-11T23:32:08Z,2020-08-11T23:33:09Z,OWNER,Needs documentation and tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/pull/927#issuecomment-672357902,https://api.github.com/repos/simonw/datasette/issues/927,672357902,MDEyOklzc3VlQ29tbWVudDY3MjM1NzkwMg==,9599,simonw,2020-08-11T23:32:39Z,2020-08-11T23:32:39Z,OWNER,"It works:
```
$ datasette --get '/:memory:.json?sql=select * from sqlite_master' | jq .
{
""database"": "":memory:"",
""query_name"": null,
""rows"": [],
""truncated"": false,
""columns"": [
""type"",
""name"",
""tbl_name"",
""rootpage"",
""sql""
],
""query"": {
""sql"": ""select * from sqlite_master"",
""params"": {}
},
""private"": false,
""allow_execute_sql"": true,
""query_ms"": 0.8032321929931641
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/issues/926#issuecomment-672338113,https://api.github.com/repos/simonw/datasette/issues/926,672338113,MDEyOklzc3VlQ29tbWVudDY3MjMzODExMw==,9599,simonw,2020-08-11T22:57:28Z,2020-08-11T22:57:28Z,OWNER,"I partly want this so I can easily implement a better `test` method for the Homebrew package. The test I have right now looks like this:
https://github.com/simonw/homebrew-datasette/blob/8aa30aa183158051a987a7e3f50e7e3ee05d8ee9/Formula/datasette.rb#L125-L127
```
test do
system bin/""datasette"", ""--help""
end
```
The Homebrew docs at https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula say:
> We want tests that don't require any user input and test the basic functionality of the application. For example `foo build-foo input.foo` is a good test and (despite their widespread use) `foo --version` and `foo --help` are bad tests. However, a bad test is better than no test at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677250834,"datasette fixtures.db --get ""/fixtures.json""",
https://github.com/simonw/datasette/issues/923#issuecomment-672336720,https://api.github.com/repos/simonw/datasette/issues/923,672336720,MDEyOklzc3VlQ29tbWVudDY3MjMzNjcyMA==,9599,simonw,2020-08-11T22:53:07Z,2020-08-11T22:53:07Z,OWNER,https://github.com/simonw/datasette/blob/5126ecb1267ed3850bf3b0ab270accd031a02e79/docs/installation.rst#using-homebrew,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/923#issuecomment-672329101,https://api.github.com/repos/simonw/datasette/issues/923,672329101,MDEyOklzc3VlQ29tbWVudDY3MjMyOTEwMQ==,9599,simonw,2020-08-11T22:35:13Z,2020-08-11T22:35:13Z,OWNER,I added the `datasette install name-of-plugin` command in #925 mainly to simplify the process of installing plugins if Datasette itself was installed using homebrew.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/925#issuecomment-672328807,https://api.github.com/repos/simonw/datasette/issues/925,672328807,MDEyOklzc3VlQ29tbWVudDY3MjMyODgwNw==,9599,simonw,2020-08-11T22:34:37Z,2020-08-11T22:34:37Z,OWNER,"This will simplify the instructions for installing plugins with Datasette install via homebrew, refs #923","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912,"""datasette install"" and ""datasette uninstall"" commands",
https://github.com/simonw/datasette/issues/925#issuecomment-672328436,https://api.github.com/repos/simonw/datasette/issues/925,672328436,MDEyOklzc3VlQ29tbWVudDY3MjMyODQzNg==,9599,simonw,2020-08-11T22:33:32Z,2020-08-11T22:33:42Z,OWNER,"```
$ datasette install --help
Usage: datasette install [OPTIONS] PACKAGES...
Install Python packages - e.g. Datasette plugins - into the same
environment as Datasette
Options:
--help Show this message and exit.
$ datasette uninstall --help
Usage: datasette uninstall [OPTIONS] PACKAGES...
Uninstall Python packages (e.g. plugins) from the Datasette environment
Options:
-y, --yes Don't ask for confirmation
--help Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912,"""datasette install"" and ""datasette uninstall"" commands",
https://github.com/simonw/datasette/issues/925#issuecomment-672304650,https://api.github.com/repos/simonw/datasette/issues/925,672304650,MDEyOklzc3VlQ29tbWVudDY3MjMwNDY1MA==,9599,simonw,2020-08-11T22:04:48Z,2020-08-11T22:04:48Z,OWNER,Prototyped in this thread: https://github.com/simonw/datasette/issues/335#issuecomment-671005731,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912,"""datasette install"" and ""datasette uninstall"" commands",
https://github.com/simonw/datasette/issues/923#issuecomment-672288845,https://api.github.com/repos/simonw/datasette/issues/923,672288845,MDEyOklzc3VlQ29tbWVudDY3MjI4ODg0NQ==,9599,simonw,2020-08-11T21:28:17Z,2020-08-11T21:28:17Z,OWNER,"Here's a pattern for installing plugins:
```
$ datasette plugins
[]
$ /usr/local/opt/datasette/libexec/bin/pip install datasette-vega
Collecting datasette-vega
Using cached datasette_vega-0.6.2-py3-none-any.whl (1.8 MB)
Requirement already satisfied: datasette in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette-vega) (0.46)
Requirement already satisfied: click~=7.1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (7.1.2)
Requirement already satisfied: click-default-group~=1.2.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.2.2)
Requirement already satisfied: Jinja2<2.12.0,>=2.10.3 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (2.11.2)
Requirement already satisfied: hupper~=1.9 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.10.2)
Requirement already satisfied: pint~=0.9 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.14)
Requirement already satisfied: pluggy~=0.13.0 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.13.1)
Requirement already satisfied: uvicorn~=0.11 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.11.8)
Requirement already satisfied: aiofiles<0.6,>=0.4 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.5.0)
Requirement already satisfied: janus<0.6,>=0.4 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.5.0)
Requirement already satisfied: asgi-csrf>=0.6 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.6.1)
Requirement already satisfied: PyYAML~=5.3 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (5.3.1)
Requirement already satisfied: mergedeep<1.4.0,>=1.1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.3.0)
Requirement already satisfied: itsdangerous~=1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.1.0)
Requirement already satisfied: python-baseconv==1.2.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.2.2)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from Jinja2<2.12.0,>=2.10.3->datasette->datasette-vega) (1.1.1)
Requirement already satisfied: setuptools in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from pint~=0.9->datasette->datasette-vega) (49.3.1)
Requirement already satisfied: packaging in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from pint~=0.9->datasette->datasette-vega) (20.4)
Requirement already satisfied: h11<0.10,>=0.8 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.9.0)
Requirement already satisfied: websockets==8.* in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (8.1)
Requirement already satisfied: httptools==0.1.* in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.1.1)
Requirement already satisfied: uvloop>=0.14.0 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.14.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from packaging->pint~=0.9->datasette->datasette-vega) (2.4.7)
Requirement already satisfied: six in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from packaging->pint~=0.9->datasette->datasette-vega) (1.15.0)
Installing collected packages: datasette-vega
Successfully installed datasette-vega-0.6.2
$ datasette plugins
[
{
""name"": ""datasette-vega"",
""static"": true,
""templates"": false,
""version"": ""0.6.2"",
""hooks"": [
""extra_css_urls"",
""extra_js_urls""
]
}
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/923#issuecomment-672287754,https://api.github.com/repos/simonw/datasette/issues/923,672287754,MDEyOklzc3VlQ29tbWVudDY3MjI4Nzc1NA==,9599,simonw,2020-08-11T21:25:33Z,2020-08-11T21:25:33Z,OWNER,.. and confirm if `brew tap ...` is even needed if you run `brew install simonw/datasette/datasette`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/923#issuecomment-672089281,https://api.github.com/repos/simonw/datasette/issues/923,672089281,MDEyOklzc3VlQ29tbWVudDY3MjA4OTI4MQ==,9599,simonw,2020-08-11T16:54:50Z,2020-08-11T16:54:50Z,OWNER,Also need to talk about how you install plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/335#issuecomment-672088880,https://api.github.com/repos/simonw/datasette/issues/335,672088880,MDEyOklzc3VlQ29tbWVudDY3MjA4ODg4MA==,9599,simonw,2020-08-11T16:54:06Z,2020-08-11T16:54:06Z,OWNER,"It works!
```
$ brew tap simonw/datasette
$ brew install simonw/datasette/datasette
$ datasette --version
datasette, version 0.46
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-671733187,https://api.github.com/repos/simonw/datasette/issues/335,671733187,MDEyOklzc3VlQ29tbWVudDY3MTczMzE4Nw==,9599,simonw,2020-08-11T05:25:23Z,2020-08-11T05:25:23Z,OWNER,I got this almost working in `simonw/homebrew-datasette` - see https://github.com/simonw/homebrew-datasette/issues/2 for the last missing detail.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671151461,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671151461,MDEyOklzc3VlQ29tbWVudDY3MTE1MTQ2MQ==,9599,simonw,2020-08-10T03:54:06Z,2020-08-10T03:54:06Z,OWNER,For the moment I'll build it without the `--retry` mode.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671151170,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671151170,MDEyOklzc3VlQ29tbWVudDY3MTE1MTE3MA==,9599,simonw,2020-08-10T03:52:02Z,2020-08-10T03:52:02Z,OWNER,"I'm having trouble figuring out how to write a test that locks a SQLite database (so I can test that `--retry` actually works). I tried this recipe but it didn't seem to prevent another process from running `pragma journal_mode='wal';` against that database:
```python
import time
import sys
import sqlite3
filename = sys.argv[-1]
db = sqlite3.connect(filename)
with db:
db.execute(""create table if not exists counter(id integer primary key, counter text)"")
time.sleep(100)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671147344,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671147344,MDEyOklzc3VlQ29tbWVudDY3MTE0NzM0NA==,9599,simonw,2020-08-10T03:29:00Z,2020-08-10T03:29:00Z,OWNER,"The CLI options should take multiple database files:
$ sqlite-utils enable-wal *.db
It's possible for this to fail if the DB is locked. How about a `--retry` option that causes it to retry a bunch of times if that happens?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671147148,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671147148,MDEyOklzc3VlQ29tbWVudDY3MTE0NzE0OA==,9599,simonw,2020-08-10T03:27:50Z,2020-08-10T03:27:50Z,OWNER,"https://www.sqlite.org/pragma.html#pragma_journal_mode lists six modes: DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF
I'm only going to implement utilities for DELETE (wal-off) and WAL (wal-on) - the other modes look like they're for specialist purposes that I don't need to support.
If it turns out I do need them I can add those to `sqlite-utils` later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671146948,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671146948,MDEyOklzc3VlQ29tbWVudDY3MTE0Njk0OA==,9599,simonw,2020-08-10T03:26:51Z,2020-08-10T03:26:51Z,OWNER,"For the CLI:
$ sqlite-utils enable-wal github.db
$ sqlite-utils disable-wal github.db
For the Python library:
```python
import sqlite_utils
db = sqlite_utils.Database(""github.db"")
db.enable_wal()
db.disable_wal()
mode = db.journal_mode # ""wal"" or ""delete"" or others
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-671088832,https://api.github.com/repos/simonw/sqlite-utils/issues/131,671088832,MDEyOklzc3VlQ29tbWVudDY3MTA4ODgzMg==,9599,simonw,2020-08-09T19:00:41Z,2020-08-09T19:00:41Z,OWNER,"Should be consistent with the `create-table` command as much as possible:
```
$ sqlite-utils create-table mydb.db mytable \
id integer \
name text \
age integer \
is_good integer \
--not-null name \
--not-null age \
--default is_good 1 \
--pk=id
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/datasette/issues/335#issuecomment-671077168,https://api.github.com/repos/simonw/datasette/issues/335,671077168,MDEyOklzc3VlQ29tbWVudDY3MTA3NzE2OA==,9599,simonw,2020-08-09T17:10:15Z,2020-08-09T18:13:39Z,OWNER,"Here's the issue that explains that warning: https://github.com/pypa/pip/issues/5599
This should fix it (risky):
from pip._internal.cli.main import main","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-671076975,https://api.github.com/repos/simonw/datasette/issues/335,671076975,MDEyOklzc3VlQ29tbWVudDY3MTA3Njk3NQ==,9599,simonw,2020-08-09T17:08:34Z,2020-08-09T17:09:21Z,OWNER,"Quick prototype of `datasette install`:
```diff
diff --git a/datasette/cli.py b/datasette/cli.py
index 287195a..95b6eb7 100644
--- a/datasette/cli.py
+++ b/datasette/cli.py
@@ -231,6 +231,18 @@ def package(
call(args)
+@cli.command()
+@click.argument(""packages"", nargs=-1, required=True)
+def install(packages):
+ ""Install Python packages - e.g. Datasette plugins - into the same environment as Datasett""
+ import pip
+
+ try:
+ pip.main([""install""] + list(packages))
+ except SystemExit as e:
+ pass
+
+
@cli.command()
@click.argument(""files"", type=click.Path(exists=True), nargs=-1)
@click.option(
```
```
$ datasette install
Usage: datasette install [OPTIONS] PACKAGES...
Try 'datasette install --help' for help.
Error: Missing argument 'PACKAGES...'.
$ datasette install datasette-vega
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Collecting datasette-vega
Using cached datasette_vega-0.6.2-py3-none-any.whl (1.8 MB)
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-671005731,https://api.github.com/repos/simonw/datasette/issues/335,671005731,MDEyOklzc3VlQ29tbWVudDY3MTAwNTczMQ==,9599,simonw,2020-08-09T04:44:13Z,2020-08-09T17:04:21Z,OWNER,"Telling people how to figure out that `pip` location is going to be pretty unpleasant.
How about instead providing a `datasette plugins --install=datasette-graphql` command?
Or `datasette install datasette-vega`
It would run `pip install` in the same virtualenv as Datasette itself.
http://jelly.codes/articles/python-pip-module/ shows how to do this:
```python
import pip
try:
pip.main([""install"", ""plumbum""])
except SystemExit as e:
pass
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/918#issuecomment-671075764,https://api.github.com/repos/simonw/datasette/issues/918,671075764,MDEyOklzc3VlQ29tbWVudDY3MTA3NTc2NA==,9599,simonw,2020-08-09T16:56:48Z,2020-08-09T16:56:48Z,OWNER,GitHub security advisory: https://github.com/simonw/datasette/security/advisories/GHSA-q6j3-c4wc-63vw,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951,Security issue: read-only canned queries leak CSRF token in URL,
https://github.com/simonw/datasette/issues/915#issuecomment-671073223,https://api.github.com/repos/simonw/datasette/issues/915,671073223,MDEyOklzc3VlQ29tbWVudDY3MTA3MzIyMw==,9599,simonw,2020-08-09T16:35:20Z,2020-08-09T16:36:10Z,OWNER,"`datasette-graphql` uses the logic from `TableView` right now. It wasn't too unpleasant, but I do worry about the two of them being coupled together in this way.
https://github.com/simonw/datasette-graphql/blob/cc65ec294b0bf8e26213fc68bb5487066de9caab/datasette_graphql/utils.py#L412-L417
```python
request = Request.fake(path_with_query_string)
view = TableView(DatasetteSpecialConfig(datasette))
data, _, _ = await view.data(
request, database=database_name, hash=None, table=table_name, _next=after
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671763164,Refactor TableView class so things like datasette-graphql can reuse the logic,
https://github.com/simonw/datasette/issues/919#issuecomment-671072223,https://api.github.com/repos/simonw/datasette/issues/919,671072223,MDEyOklzc3VlQ29tbWVudDY3MTA3MjIyMw==,9599,simonw,2020-08-09T16:26:17Z,2020-08-09T16:26:17Z,OWNER,Should be released in a couple of minutes: https://travis-ci.org/github/simonw/datasette/builds/716328883,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",675727366,"Travis should not build the master branch, only the main branch",
https://github.com/simonw/datasette/issues/919#issuecomment-671072084,https://api.github.com/repos/simonw/datasette/issues/919,671072084,MDEyOklzc3VlQ29tbWVudDY3MTA3MjA4NA==,4312421,stonebig,2020-08-09T16:25:01Z,2020-08-09T16:26:03Z,NONE,don't forget the pypi wheel (still on datasette-0.45),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675727366,"Travis should not build the master branch, only the main branch",
https://github.com/simonw/datasette/issues/918#issuecomment-671071710,https://api.github.com/repos/simonw/datasette/issues/918,671071710,MDEyOklzc3VlQ29tbWVudDY3MTA3MTcxMA==,9599,simonw,2020-08-09T16:21:41Z,2020-08-09T16:21:41Z,OWNER,Submitting the form on https://latest.datasette.io/fixtures/neighborhood_search demonstrates that this is fixed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951,Security issue: read-only canned queries leak CSRF token in URL,
https://github.com/simonw/datasette/issues/919#issuecomment-671071461,https://api.github.com/repos/simonw/datasette/issues/919,671071461,MDEyOklzc3VlQ29tbWVudDY3MTA3MTQ2MQ==,9599,simonw,2020-08-09T16:19:37Z,2020-08-09T16:19:37Z,OWNER,That appears to have worked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675727366,"Travis should not build the master branch, only the main branch",
https://github.com/simonw/datasette/issues/918#issuecomment-671070528,https://api.github.com/repos/simonw/datasette/issues/918,671070528,MDEyOklzc3VlQ29tbWVudDY3MTA3MDUyOA==,9599,simonw,2020-08-09T16:12:16Z,2020-08-09T16:12:16Z,OWNER,"It's worth noting that in order to exploit this issue the following would all need to be true:
- A user is running a copy of Datasette protected by a cookie-based authentication plugin AND configured with at least one writable canned query
- An attacker is in control of a URL that could concievably be returned on a page that is displayed as the result of submitting a read-only canned query
- An authenticated user of that Datasette instance, who is running a browser that doesn't support the `SameSite=lax` cookie parameter (which is [widely supported](https://caniuse.com/#feat=same-site-cookie-attribute) by modern browsers), submits the read-only canned query form and then clicks a link to the attacker's off-site page, exposing their CSRFToken in the attacker's HTTP referer logs
- The attacker then tricks that user into visiting their own malicious web page which includes a POST form that auto-submits against the writable canned query that the attacker wishes to exploit, including the CSRF token as a hidden field
The attacker would need full knowledge of the URL and form layout of the Datasette instance that they are exploiting.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951,Security issue: read-only canned queries leak CSRF token in URL,
https://github.com/simonw/datasette/issues/918#issuecomment-671070486,https://api.github.com/repos/simonw/datasette/issues/918,671070486,MDEyOklzc3VlQ29tbWVudDY3MTA3MDQ4Ng==,9599,simonw,2020-08-09T16:11:59Z,2020-08-09T16:11:59Z,OWNER,Fix has been released in Datasette 0.46: https://datasette.readthedocs.io/en/latest/changelog.html#v0-46,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951,Security issue: read-only canned queries leak CSRF token in URL,
https://github.com/simonw/datasette/issues/335#issuecomment-671001457,https://api.github.com/repos/simonw/datasette/issues/335,671001457,MDEyOklzc3VlQ29tbWVudDY3MTAwMTQ1Nw==,9599,simonw,2020-08-09T03:37:39Z,2020-08-09T03:37:39Z,OWNER,"Here's what happened when I installed `homebrew-vd`: https://gist.github.com/simonw/7bfd971a62743d7ca248e6b5e696c240
It worked! And from digging around, it has a virtual environment at `/usr/local/Cellar/visidata/1.5.2/libexec/`
Which means `/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip` is a working `pip`
And I tried running these commands and confirmed that I get a `datasette` with an additional plugin:
```
/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install datasette
/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install datasette-graphql
/usr/local/Cellar/visidata/1.5.2/libexec/bin/datasette plugins
[
{
""name"": ""datasette-graphql"",
""static"": false,
""templates"": true,
""version"": ""0.11"",
""hooks"": [
""register_routes"",
""startup""
]
}
]
```
So I can package Datasette as a homebrew package AND I can give people instructions for installing plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-670999860,https://api.github.com/repos/simonw/datasette/issues/335,670999860,MDEyOklzc3VlQ29tbWVudDY3MDk5OTg2MA==,9599,simonw,2020-08-09T03:12:44Z,2020-08-09T03:12:44Z,OWNER,How would plugin installation work if Datasette was installed via homebrew?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-670999832,https://api.github.com/repos/simonw/datasette/issues/335,670999832,MDEyOklzc3VlQ29tbWVudDY3MDk5OTgzMg==,9599,simonw,2020-08-09T03:12:14Z,2020-08-09T03:12:14Z,OWNER,Another useful example: https://github.com/Homebrew/homebrew-core/blob/master/Formula/trailscraper.rb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8,669241341,MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ==,9599,simonw,2020-08-05T14:55:14Z,2020-08-05T14:55:14Z,MEMBER,"Looks like there's a column that's not consistently there, so the table got created without it.
Easiest fix is to add `alter=True` to this line:
https://github.com/dogsheep/swarm-to-sqlite/blob/f4a82633da927cde672c9d9af92930bfca2e3ddf/swarm_to_sqlite/utils.py#L94
That will cause `sqlite-utils` to notice if there's a missing column and add it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648245071,Error thrown: table photos has no column named hasSticker,
https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585598,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667585598,MDEyOklzc3VlQ29tbWVudDY2NzU4NTU5OA==,9599,simonw,2020-08-01T20:51:28Z,2020-08-01T20:51:28Z,OWNER,CLI documentation: https://github.com/simonw/sqlite-utils/commit/57e4eb8e5564af5d97f892b3be8342451ee177a2?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371,Support tokenize option for FTS,
https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585561,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667585561,MDEyOklzc3VlQ29tbWVudDY2NzU4NTU2MQ==,9599,simonw,2020-08-01T20:50:59Z,2020-08-01T20:50:59Z,OWNER,Turns out it works for FTS4 as well: https://www.sqlite.org/fts3.html#tokenizer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371,Support tokenize option for FTS,
https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667584567,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667584567,MDEyOklzc3VlQ29tbWVudDY2NzU4NDU2Nw==,9599,simonw,2020-08-01T20:41:09Z,2020-08-01T20:41:09Z,OWNER,API documentation here: https://github.com/simonw/sqlite-utils/commit/617e6f070c85be66ea04c80b78dafd08c875f8c8?short_path=e156262#diff-e1562629b8def6da772d9b0903faf703,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371,Support tokenize option for FTS,
https://github.com/simonw/datasette/issues/900#issuecomment-667431123,https://api.github.com/repos/simonw/datasette/issues/900,667431123,MDEyOklzc3VlQ29tbWVudDY2NzQzMTEyMw==,9599,simonw,2020-07-31T23:56:33Z,2020-07-31T23:56:33Z,OWNER,I think this is the same issue as #865. I'll look at these together!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url,
https://github.com/simonw/datasette/issues/899#issuecomment-667430790,https://api.github.com/repos/simonw/datasette/issues/899,667430790,MDEyOklzc3VlQ29tbWVudDY2NzQzMDc5MA==,9599,simonw,2020-07-31T23:54:40Z,2020-07-31T23:54:40Z,OWNER,"There's no mechanism that can do this at the moment.
You could absolutely support this with a plugin, probably using the `asgi_wrapper` plugin hook. There's an existing package at https://pypi.org/project/asgi-ratelimit/ which may be usable for this - it may even be possible to configure that using https://github.com/simonw/datasette-configure-asgi rather than using it to write a custom plugin.
Using a separate revers proxy would also be a good way to solve this. It depends which option would work best in your environment.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660827546,How to setup a request limit per user,
https://github.com/simonw/datasette/issues/913#issuecomment-667430352,https://api.github.com/repos/simonw/datasette/issues/913,667430352,MDEyOklzc3VlQ29tbWVudDY2NzQzMDM1Mg==,9599,simonw,2020-07-31T23:52:10Z,2020-07-31T23:52:10Z,OWNER,The bigger question here is when this mechanism should be used in place of `metadata.json` or `metadata.yml`. Especially since I'm already considering renaming or reworking that mechanism since plugin configuration has nothing to do with database metadata: #493,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/913#issuecomment-667429616,https://api.github.com/repos/simonw/datasette/issues/913,667429616,MDEyOklzc3VlQ29tbWVudDY2NzQyOTYxNg==,9599,simonw,2020-07-31T23:48:25Z,2020-07-31T23:49:59Z,OWNER,"I could let plugins add additional options to `datasette serve` - but what if two plugins both try to register an option with the same name?
A better solution could be to use the existing `--config` option - and allow plugins to register their own, namespaced config options. So you could do things like:
datasette my.db --config datasette-insert:unsafe:1
Maybe even drop the `datasette-` prefix?
datasette my.db --config insert:unsafe:1
I think I prefer keeping the prefix to be honest - it makes it more obvious that this is a setting which comes from a specific named plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/913#issuecomment-667429690,https://api.github.com/repos/simonw/datasette/issues/913,667429690,MDEyOklzc3VlQ29tbWVudDY2NzQyOTY5MA==,9599,simonw,2020-07-31T23:48:48Z,2020-07-31T23:48:48Z,OWNER,"Here's the code in Datasette that parses `--config` options at the moment:
https://github.com/simonw/datasette/blob/7ca8c0521ac1ea48a3cd8d0fe9275d1316e54b43/datasette/cli.py#L25-L40","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/849#issuecomment-667424128,https://api.github.com/repos/simonw/datasette/issues/849,667424128,MDEyOklzc3VlQ29tbWVudDY2NzQyNDEyOA==,9599,simonw,2020-07-31T23:21:56Z,2020-07-31T23:23:24Z,OWNER,"I'm going to change the default branch on the GitHub repository. If something breaks I can always change it back again.
Done that! Default is now `main`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-667424020,https://api.github.com/repos/simonw/datasette/issues/849,667424020,MDEyOklzc3VlQ29tbWVudDY2NzQyNDAyMA==,9599,simonw,2020-07-31T23:21:30Z,2020-07-31T23:21:30Z,OWNER,https://github.com/simonw/datasette/tree/main branch now exists and will automatically mirror master (and vice-versa).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-667295759,https://api.github.com/repos/simonw/datasette/issues/849,667295759,MDEyOklzc3VlQ29tbWVudDY2NzI5NTc1OQ==,9599,simonw,2020-07-31T18:45:35Z,2020-07-31T18:45:35Z,OWNER,"Watch out for places in the documentation that might link to `master` - e.g. here:
https://github.com/simonw/datasette/blob/2d7fa8b9058dfbf9c7c371cdeec115d32a177dc9/docs/custom_templates.rst#L247","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/sqlite-utils/issues/124#issuecomment-664105302,https://api.github.com/repos/simonw/sqlite-utils/issues/124,664105302,MDEyOklzc3VlQ29tbWVudDY2NDEwNTMwMg==,9599,simonw,2020-07-27T03:54:24Z,2020-07-30T22:57:51Z,OWNER,"Documentation: https://github.com/simonw/sqlite-utils/commit/814d4a7f90991be865d38aac45ff12e36df1c67d?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40
> You can pass named parameters to the query using -p:
>
> $ sqlite-utils query dogs.db ""select :num * :num2"" -p num 5 -p num2 6
> [{"":num * :num2"": 30}]
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665802405,sqlite-utils query should support named parameters,
https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666752039,https://api.github.com/repos/simonw/sqlite-utils/issues/129,666752039,MDEyOklzc3VlQ29tbWVudDY2Njc1MjAzOQ==,9599,simonw,2020-07-30T22:40:55Z,2020-07-30T22:40:55Z,OWNER,"This should be a separate command from `insert-files`. SQLite Archives should use a table with this schema:
```sql
CREATE TABLE sqlar(
name TEXT PRIMARY KEY, -- name of the file
mode INT, -- access permissions
mtime INT, -- last modification time
sz INT, -- original file size
data BLOB -- compressed content
);
```
`insert-files` currently treats the table name as a required argument - but it's not necessary for this table. Also there shouldn't be any support for the `--column` option.
So if I write this command it should be this instead:
sqlite-utils sqlar files.db file.txt file2.txt
But at that point, why bother? Users can use `sqlite3 files.db -Ac *.txt` instead.
So I'm not going to bother implementing this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",668308777,"""insert-files --sqlar"" for creating SQLite archives",
https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666063689,https://api.github.com/repos/simonw/sqlite-utils/issues/127,666063689,MDEyOklzc3VlQ29tbWVudDY2NjA2MzY4OQ==,9599,simonw,2020-07-30T03:08:51Z,2020-07-30T03:08:51Z,OWNER,Documentation at the bottom of this section: https://github.com/simonw/sqlite-utils/blob/8fe1e6d1be021aeeb8f08b0f77f03b75a83b6f75/docs/cli.rst#inserting-binary-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390,Ability to insert files piped to insert-files stdin,
https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666047928,https://api.github.com/repos/simonw/sqlite-utils/issues/127,666047928,MDEyOklzc3VlQ29tbWVudDY2NjA0NzkyOA==,9599,simonw,2020-07-30T02:31:05Z,2020-07-30T02:31:05Z,OWNER,"Maybe could do this using an improved version of this lambda? Could teach it to look for `-` and read from `sys.stdin` if it sees it.
https://github.com/simonw/sqlite-utils/blob/710454d72aed5094573e642344fd075a0ef5372c/sqlite_utils/cli.py#L839","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390,Ability to insert files piped to insert-files stdin,
https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666046819,https://api.github.com/repos/simonw/sqlite-utils/issues/129,666046819,MDEyOklzc3VlQ29tbWVudDY2NjA0NjgxOQ==,9599,simonw,2020-07-30T02:28:34Z,2020-07-30T02:28:34Z,OWNER,This code looks useful as inspiration: https://github.com/j4mie/sqlsite/blob/f2dadb8db5ed7880f8872b6591d8cb1487f777ea/sqlsite/sqlar.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",668308777,"""insert-files --sqlar"" for creating SQLite archives",
https://github.com/simonw/datasette/issues/909#issuecomment-666010395,https://api.github.com/repos/simonw/datasette/issues/909,666010395,MDEyOklzc3VlQ29tbWVudDY2NjAxMDM5NQ==,9599,simonw,2020-07-30T00:56:17Z,2020-07-30T00:56:17Z,OWNER,"```
$ curl -I https://latest.datasette.io/fixtures.db
HTTP/1.1 200 OK
content-disposition: attachment; filename=""fixtures.db""
content-type: application/octet-stream
Date: Thu, 30 Jul 2020 00:56:05 GMT
Server: Google Frontend
Transfer-Encoding: chunked
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",667467128,AsgiFileDownload: filename not correctly passed,
https://github.com/simonw/datasette/issues/909#issuecomment-665854704,https://api.github.com/repos/simonw/datasette/issues/909,665854704,MDEyOklzc3VlQ29tbWVudDY2NTg1NDcwNA==,9599,simonw,2020-07-29T19:22:31Z,2020-07-29T19:22:31Z,OWNER,"I think this results in a bug where the ""download database"" link doesn't include the correct filename: https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/datasette/views/database.py#L110-L131","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",667467128,AsgiFileDownload: filename not correctly passed,
https://github.com/simonw/datasette/pull/910#issuecomment-665663131,https://api.github.com/repos/simonw/datasette/issues/910,665663131,MDEyOklzc3VlQ29tbWVudDY2NTY2MzEzMQ==,22429695,codecov[bot],2020-07-29T13:26:14Z,2020-07-29T13:26:14Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=h1) Report
> Merging [#910](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/3c33b421320c0be81a625ca7307b2e4416a9ed5b&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/910/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #910 +/- ##
=======================================
Coverage 83.55% 83.55%
=======================================
Files 27 27
Lines 3666 3666
=======================================
Hits 3063 3063
Misses 603 603
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=footer). Last update [3c33b42...3493915](https://codecov.io/gh/simonw/datasette/pull/910?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",667840539,"Update pytest requirement from <5.5.0,>=5.2.2 to >=5.2.2,<6.1.0",
https://github.com/simonw/sqlite-utils/issues/128#issuecomment-664683608,https://api.github.com/repos/simonw/sqlite-utils/issues/128,664683608,MDEyOklzc3VlQ29tbWVudDY2NDY4MzYwOA==,9599,simonw,2020-07-27T23:09:22Z,2020-07-27T23:09:22Z,OWNER,"This seems to work, but needs more tests:
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..ee26433 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -7,6 +7,7 @@ import itertools
import json
import os
import pathlib
+import uuid
SQLITE_MAX_VARS = 999
@@ -40,11 +41,13 @@ COLUMN_TYPE_MAPPING = {
str: ""TEXT"",
bytes.__class__: ""BLOB"",
bytes: ""BLOB"",
+ memoryview: ""BLOB"",
datetime.datetime: ""TEXT"",
datetime.date: ""TEXT"",
datetime.time: ""TEXT"",
decimal.Decimal: ""FLOAT"",
None.__class__: ""TEXT"",
+ uuid.UUID: ""TEXT"",
# SQLite explicit types
""TEXT"": ""TEXT"",
""INTEGER"": ""INTEGER"",
@@ -1336,6 +1339,8 @@ def jsonify_if_needed(value):
return json.dumps(value, default=repr)
elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):
return value.isoformat()
+ elif isinstance(value, uuid.UUID):
+ return str(value)
else:
return value
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666639051,Support UUID and memoryview types,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664163524,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664163524,MDEyOklzc3VlQ29tbWVudDY2NDE2MzUyNA==,9599,simonw,2020-07-27T07:10:41Z,2020-07-27T07:10:41Z,OWNER,Docs: https://github.com/simonw/sqlite-utils/blob/ebc802f7ff0e640b6ae11ea525290fea0115228c/docs/cli.rst#inserting-binary-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/127#issuecomment-664163206,https://api.github.com/repos/simonw/sqlite-utils/issues/127,664163206,MDEyOklzc3VlQ29tbWVudDY2NDE2MzIwNg==,9599,simonw,2020-07-27T07:10:05Z,2020-07-27T07:10:05Z,OWNER,I tried to get this working but it was a bit tricky because `-` doesn't behave like a regular `pathlib.Path` - needs a bit more thought on how the implementation would work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390,Ability to insert files piped to insert-files stdin,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664128071,MDEyOklzc3VlQ29tbWVudDY2NDEyODA3MQ==,9599,simonw,2020-07-27T05:30:54Z,2020-07-27T05:30:54Z,OWNER,"Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.
cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664127741,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664127741,MDEyOklzc3VlQ29tbWVudDY2NDEyNzc0MQ==,9599,simonw,2020-07-27T05:29:48Z,2020-07-27T05:29:48Z,OWNER,"Test command:
```
sqlite-utils insert-files gifs.db *.gif \
-c filename:filename \
-c filepath:filepath \
-c absolutepath:absolutepath \
-c sha256:sha256 \
-c md5:md5 \
-c content:content \
-c mtime:mtime \
-c ctime:ctime \
-c mtime_iso:mtime_iso \
-c ctime_iso:ctime_iso \
-c size:size \
--pk absolutepath
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931279,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931279,MDEyOklzc3VlQ29tbWVudDY2MzkzMTI3OQ==,9599,simonw,2020-07-26T03:33:23Z,2020-07-27T04:30:49Z,OWNER,"One idea: `sqlite-utils insert-files`
It could work something like this:
sqlite-utils insert-files files.db /tmp/blah.jpg /tmp/foo.gif \
--table files \
-c key:filename -c hash:sha256 -c body:content \
--pk key
This would insert those two image files into the database in a table called `files` with a schema that looks something like this:
```sql
CREATE TABLE files (
key text primary key,
hash text,
body blob
);
```
The `-c key:filename` options here are the most interesting: they let you create the table with a specific layout. The bit before the `:` is the column name. The bit after the `:` can be a range of different things:
- `filename` - just the filename
- `filepath` - the full filepath (provided on the command-line)
- `absolutepath` - the filepath expanded to start with `/home/...` or whatever
- `sha256` - the SHA256 of the contents
- `md5` - the MD5
- `content` - the binary content itself
- `mtime` - the mtime (floating point timestamp)
- `ctime` - the ctime (floating point timestamp)
- `mtime_iso` - the mtime as an ISO datetime
- `ctime_iso` - the mtime as an ISO datetime
- `size` - the size of the file in bytes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-664106621,https://api.github.com/repos/simonw/sqlite-utils/issues/114,664106621,MDEyOklzc3VlQ29tbWVudDY2NDEwNjYyMQ==,9599,simonw,2020-07-27T04:01:13Z,2020-07-27T04:01:13Z,OWNER,Work in progress in `transform` branch here: https://github.com/simonw/sqlite-utils/tree/transform,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664106405,https://api.github.com/repos/simonw/sqlite-utils/issues/126,664106405,MDEyOklzc3VlQ29tbWVudDY2NDEwNjQwNQ==,9599,simonw,2020-07-27T04:00:08Z,2020-07-27T04:00:33Z,OWNER,"```
$ echo '[
{
""name"": ""transparent.gif"",
""content"": {
""$base64"": true,
""encoded"": ""R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7""
}
}
]' | sqlite-utils insert trans.db files - --pk=name
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665819048,Ability to insert binary data on the CLI using JSON,
https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664065597,https://api.github.com/repos/simonw/sqlite-utils/issues/126,664065597,MDEyOklzc3VlQ29tbWVudDY2NDA2NTU5Nw==,9599,simonw,2020-07-27T00:51:11Z,2020-07-27T00:51:11Z,OWNER,"I'm going to implement this as the reverse of #125 - binary columns in JSON are now output like this:
```json
{
""name"": ""lorem.txt"",
""mode"": 33188,
""mtime"": 1595805965,
""sz"": 16984,
""data"": {
""$base64"": true,
""encoded"": ""eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9""
}
}
]
```
So the `sqlite-utils insert` command should learn to spot `{""$base64"": true...}` values and base64 decode them before inserting them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665819048,Ability to insert binary data on the CLI using JSON,
https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664065341,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664065341,MDEyOklzc3VlQ29tbWVudDY2NDA2NTM0MQ==,9599,simonw,2020-07-27T00:49:41Z,2020-07-27T00:49:41Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/commit/20e543e9a492f2e764caae73c38e87f18eaec444?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570,"Output binary columns in ""sqlite-utils query"" JSON",
https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664062546,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664062546,MDEyOklzc3VlQ29tbWVudDY2NDA2MjU0Ng==,9599,simonw,2020-07-27T00:33:03Z,2020-07-27T00:33:03Z,OWNER,"I'm going to imitate how Datasette solves this problem:
```json
[
{
""name"": ""lorem.txt"",
""mode"": 33188,
""mtime"": 1595805965,
""sz"": 16984,
""data"": {
""$base64"": true,
""encoded"": ""eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9""
}
}
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570,"Output binary columns in ""sqlite-utils query"" JSON",
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048720,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664048720,MDEyOklzc3VlQ29tbWVudDY2NDA0ODcyMA==,9599,simonw,2020-07-26T22:32:50Z,2020-07-26T22:33:20Z,OWNER,"This seems to work in creating a SQLite archive containing all `.gif` files in the current directory:
/usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -c *.gif
Then listing files like this:
```
$ /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -t
copyable.gif
debug-allow.gif
flash.gif
table-md.gif
```
Here's the schema:
```
$ sqlite3 archive.db .schema
CREATE TABLE sqlar(
name TEXT PRIMARY KEY, -- name of the file
mode INT, -- access permissions
mtime INT, -- last modification time
sz INT, -- original file size
data BLOB -- compressed content
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048432,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664048432,MDEyOklzc3VlQ29tbWVudDY2NDA0ODQzMg==,9599,simonw,2020-07-26T22:29:31Z,2020-07-26T22:29:31Z,OWNER,"I'm trying to play with `sqlite3 -A` on my Mac.
`sqlite3 -A` tells me that it's an unknown option - but I used `brew info sqlite` to find my homebrew installed version and it turns out this works:
```
% /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 -A
Wrong number of arguments. Usage:
.archive ... Manage SQL archives
Each command must have exactly one of the following options:
-c, --create Create a new archive
-u, --update Add or update files with changed mtime
-i, --insert Like -u but always add even if unchanged
-t, --list List contents of archive
-x, --extract Extract files from archive
Optional arguments:
-v, --verbose Print each filename as it is processed
-f FILE, --file FILE Use archive FILE (default is current db)
-a FILE, --append FILE Open FILE using the apndvfs VFS
-C DIR, --directory DIR Read/extract files from directory DIR
-n, --dryrun Show the SQL that would have occurred
Examples:
.ar -cf ARCHIVE foo bar # Create ARCHIVE from files foo and bar
.ar -tf ARCHIVE # List members of ARCHIVE
.ar -xvf ARCHIVE # Verbosely extract files from ARCHIVE
See also:
http://sqlite.org/cli.html#sqlar_archive_support
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664013338,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664013338,MDEyOklzc3VlQ29tbWVudDY2NDAxMzMzOA==,9599,simonw,2020-07-26T16:57:35Z,2020-07-26T16:57:35Z,OWNER,"I should consider easy compatibility with https://www.sqlite.org/sqlar.html
> An SQLite Archive is an ordinary SQLite database file that contains the following table as part of its schema:
> ```
> CREATE TABLE sqlar(
> name TEXT PRIMARY KEY, -- name of the file
> mode INT, -- access permissions
> mtime INT, -- last modification time
> sz INT, -- original file size
> data BLOB -- compressed content
> );
> ```
> Each row of the SQLAR table holds the content of a single file. The filename (the full pathname relative to the root of the archive) is in the ""name"" field. The ""mode"" field is an integer which is the unix-style access permissions for the file. ""mtime"" is the modification time of the file in seconds since 1970. ""sz"" is the original uncompressed size of the file. The ""data"" field contains the file content. The content is usually compressed using [Deflate](http://zlib.net/), though not always. If the ""sz"" field is equal to the size of the ""data"" field, then the content is stored uncompressed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664012247,MDEyOklzc3VlQ29tbWVudDY2NDAxMjI0Nw==,9599,simonw,2020-07-26T16:48:46Z,2020-07-26T16:48:46Z,OWNER,"I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says ""these columns are base64 encoded, store the decoded data in a BLOB"".
That would solve inserting binary data using JSON too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570,"Output binary columns in ""sqlite-utils query"" JSON",
https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012148,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664012148,MDEyOklzc3VlQ29tbWVudDY2NDAxMjE0OA==,9599,simonw,2020-07-26T16:47:51Z,2020-07-26T16:47:51Z,OWNER,Best solution I can think of is to return the data as base64. It's a bit nasty since it means you can't round trip it back again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570,"Output binary columns in ""sqlite-utils query"" JSON",
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931426,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931426,MDEyOklzc3VlQ29tbWVudDY2MzkzMTQyNg==,9599,simonw,2020-07-26T03:35:58Z,2020-07-26T16:44:33Z,OWNER,Related: #123 (`--raw` option),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931662,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931662,MDEyOklzc3VlQ29tbWVudDY2MzkzMTY2Mg==,9599,simonw,2020-07-26T03:40:29Z,2020-07-26T03:40:29Z,OWNER,Maybe support `--replace` for replacing images with an existing primary key.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931317,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931317,MDEyOklzc3VlQ29tbWVudDY2MzkzMTMxNw==,9599,simonw,2020-07-26T03:33:54Z,2020-07-26T03:33:54Z,OWNER,"The command also accepts one or more directories, in which case it will recursively scan them for all files that they contain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/datasette/pull/868#issuecomment-663924220,https://api.github.com/repos/simonw/datasette/issues/868,663924220,MDEyOklzc3VlQ29tbWVudDY2MzkyNDIyMA==,702729,joshmgrant,2020-07-26T01:29:00Z,2020-07-26T01:29:00Z,NONE,"Ok, so it's been a while but I think I'm making progress. The good news: I have come up with a configuration change for running the tests on Windows on GitHub Actions. The bad news: there's a bunch of extraneous errors on the Windows case. I *think* this is due to Windows file IO and sqlite in a lot of cases, so I'm working through it. I will eventually clean up this PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646448486,initial windows ci setup,
https://github.com/simonw/datasette/issues/906#issuecomment-663779460,https://api.github.com/repos/simonw/datasette/issues/906,663779460,MDEyOklzc3VlQ29tbWVudDY2Mzc3OTQ2MA==,9599,simonw,2020-07-25T00:07:10Z,2020-07-25T00:07:10Z,OWNER,"Demo of the new functionality:
* https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=false
* https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665400224,"""allow"": true for anyone, ""allow"": false for nobody",
https://github.com/simonw/datasette/issues/908#issuecomment-663779179,https://api.github.com/repos/simonw/datasette/issues/908,663779179,MDEyOklzc3VlQ29tbWVudDY2Mzc3OTE3OQ==,9599,simonw,2020-07-25T00:05:48Z,2020-07-25T00:06:15Z,OWNER,"The documentation section here now has a bunch of different links to live demos illustrating different ""allow"" block syntax: https://github.com/simonw/datasette/blob/092874202c8748d6e0d4800eaf707c0145d95ffe/docs/authentication.rst#defining-permissions-with-allow-blocks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665407663,"Interactive debugging tool for ""allow"" blocks",
https://github.com/simonw/datasette/issues/906#issuecomment-663767678,https://api.github.com/repos/simonw/datasette/issues/906,663767678,MDEyOklzc3VlQ29tbWVudDY2Mzc2NzY3OA==,9599,simonw,2020-07-24T23:07:22Z,2020-07-24T23:07:22Z,OWNER,"Illustration of current system:
https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow=null - `null` allows
https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow={} - `{}` denies","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665400224,"""allow"": true for anyone, ""allow"": false for nobody",
https://github.com/simonw/datasette/issues/908#issuecomment-663765308,https://api.github.com/repos/simonw/datasette/issues/908,663765308,MDEyOklzc3VlQ29tbWVudDY2Mzc2NTMwOA==,9599,simonw,2020-07-24T22:57:15Z,2020-07-24T22:57:15Z,OWNER,Tool lives at https://latest.datasette.io/-/allow-debug,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665407663,"Interactive debugging tool for ""allow"" blocks",
https://github.com/simonw/datasette/issues/907#issuecomment-663764203,https://api.github.com/repos/simonw/datasette/issues/907,663764203,MDEyOklzc3VlQ29tbWVudDY2Mzc2NDIwMw==,9599,simonw,2020-07-24T22:53:07Z,2020-07-24T22:53:07Z,OWNER,"Actually that is already covered here:
https://github.com/simonw/datasette/blob/6be5654ffab282e8cf39cc138ba2d4496ebc7407/docs/authentication.rst#L158","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665403403,Allow documentation doesn't explain what happens with multiple allow keys,
https://github.com/simonw/datasette/pull/901#issuecomment-663727716,https://api.github.com/repos/simonw/datasette/issues/901,663727716,MDEyOklzc3VlQ29tbWVudDY2MzcyNzcxNg==,22429695,codecov[bot],2020-07-24T20:47:57Z,2020-07-24T20:47:57Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=h1) Report
> Merging [#901](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/d9a5ef1c32a4390e398653ebfd570f8e1a03d93e&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `83.33%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/901/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #901 +/- ##
==========================================
- Coverage 83.41% 83.41% -0.01%
==========================================
Files 27 27
Lines 3636 3642 +6
==========================================
+ Hits 3033 3038 +5
- Misses 603 604 +1
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.76% <66.66%> (-0.18%)` | :arrow_down: |
| [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <100.00%> (+0.09%)` | :arrow_up: |
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/901/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.47% <0.00%> (+0.07%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=footer). Last update [d9a5ef1...1285f28](https://codecov.io/gh/simonw/datasette/pull/901?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",662322234,Use None as a default arg,
https://github.com/simonw/datasette/issues/908#issuecomment-663726318,https://api.github.com/repos/simonw/datasette/issues/908,663726318,MDEyOklzc3VlQ29tbWVudDY2MzcyNjMxOA==,9599,simonw,2020-07-24T20:43:57Z,2020-07-24T20:45:38Z,OWNER,"I can implement this as a plugin. Or it could ship as part of Datasette, somewhere under the `/-/` namespace like the `PermissionsDebugView` and `MessagesDebugView` tools.
I'm going to ship it in Datasette core, to further reinforce the philosophy that debugging tools are important.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665407663,"Interactive debugging tool for ""allow"" blocks",
https://github.com/simonw/datasette/issues/907#issuecomment-663726146,https://api.github.com/repos/simonw/datasette/issues/907,663726146,MDEyOklzc3VlQ29tbWVudDY2MzcyNjE0Ng==,9599,simonw,2020-07-24T20:43:27Z,2020-07-24T20:43:27Z,OWNER,"It might be good to have a little interactive tool which helps debug these things, since there are quite a few edge-cases and the damage caused if people use them incorrectly is substantial.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665403403,Allow documentation doesn't explain what happens with multiple allow keys,
https://github.com/simonw/datasette/issues/456#issuecomment-663724675,https://api.github.com/repos/simonw/datasette/issues/456,663724675,MDEyOklzc3VlQ29tbWVudDY2MzcyNDY3NQ==,9599,simonw,2020-07-24T20:39:17Z,2020-07-24T20:39:17Z,OWNER,Yes this is still a bug!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",442327592,Installing installs the tests package,
https://github.com/simonw/datasette/pull/902#issuecomment-663724425,https://api.github.com/repos/simonw/datasette/issues/902,663724425,MDEyOklzc3VlQ29tbWVudDY2MzcyNDQyNQ==,9599,simonw,2020-07-24T20:38:42Z,2020-07-24T20:38:42Z,OWNER,Thanks for spotting this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",662439034,Don't install tests package,
https://github.com/simonw/datasette/issues/906#issuecomment-663720907,https://api.github.com/repos/simonw/datasette/issues/906,663720907,MDEyOklzc3VlQ29tbWVudDY2MzcyMDkwNw==,9599,simonw,2020-07-24T20:29:24Z,2020-07-24T20:29:24Z,OWNER,"Here are the existing test cases:
https://github.com/simonw/datasette/blob/2115d7e3457b48b3cf9c81551b9fed2d0e9cd111/tests/test_utils.py#L468-L505","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665400224,"""allow"": true for anyone, ""allow"": false for nobody",
https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-663143160,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48,663143160,MDEyOklzc3VlQ29tbWVudDY2MzE0MzE2MA==,9599,simonw,2020-07-23T17:46:07Z,2020-07-23T17:46:07Z,MEMBER,"Frustratingly, these links don't work on PyPI: https://pypi.org/project/twitter-to-sqlite/
There's an issue about that here: https://github.com/pypa/readme_renderer/issues/169","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663976976,Add a table of contents to the README,
https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662630868,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48,662630868,MDEyOklzc3VlQ29tbWVudDY2MjYzMDg2OA==,9599,simonw,2020-07-22T19:03:02Z,2020-07-22T19:03:02Z,MEMBER,Done!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663976976,Add a table of contents to the README,
https://github.com/dogsheep/twitter-to-sqlite/issues/48#issuecomment-662626901,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/48,662626901,MDEyOklzc3VlQ29tbWVudDY2MjYyNjkwMQ==,9599,simonw,2020-07-22T18:54:53Z,2020-07-22T18:54:53Z,MEMBER,I'm going to use a GitHub Action to run `npx markdown-toc README.md -i`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663976976,Add a table of contents to the README,
https://github.com/simonw/datasette/issues/905#issuecomment-662241702,https://api.github.com/repos/simonw/datasette/issues/905,662241702,MDEyOklzc3VlQ29tbWVudDY2MjI0MTcwMg==,9599,simonw,2020-07-22T04:59:46Z,2020-07-22T04:59:46Z,OWNER,"Deployed and working:
```
% curl -I 'https://fivethirtyeight.datasettes.com/fivethirtyeight.db'
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2020 04:59:23 GMT
Content-Type: application/octet-stream
Content-Length: 281845760
Connection: keep-alive
Set-Cookie: __cfduid=d550b15c99aa59144e49557ced64fc48a1595393963; expires=Fri, 21-Aug-20 04:59:23 GMT; path=/; domain=.datasettes.com; HttpOnly; SameSite=Lax
Via: 1.1 vegur
Cache-Control: max-age=14400
CF-Cache-Status: MISS
Accept-Ranges: bytes
cf-request-id: 04167d0c7100000540f98e8200000001
Expect-CT: max-age=604800, report-uri=""https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct""
Server: cloudflare
CF-RAY: 5b6a978d89b30540-LAX
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663317875,/database.db download should include content-length header,
https://github.com/simonw/datasette/issues/905#issuecomment-662114881,https://api.github.com/repos/simonw/datasette/issues/905,662114881,MDEyOklzc3VlQ29tbWVudDY2MjExNDg4MQ==,9599,simonw,2020-07-21T21:25:37Z,2020-07-21T21:25:37Z,OWNER,I can use `aiofiles.os.stat` for this: https://github.com/Tinche/aiofiles/blob/master/aiofiles/os.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663317875,/database.db download should include content-length header,
https://github.com/simonw/datasette/pull/902#issuecomment-661587375,https://api.github.com/repos/simonw/datasette/issues/902,661587375,MDEyOklzc3VlQ29tbWVudDY2MTU4NzM3NQ==,22429695,codecov[bot],2020-07-21T02:44:49Z,2020-07-21T02:44:49Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=h1) Report
> Merging [#902](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/d9a5ef1c32a4390e398653ebfd570f8e1a03d93e&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/902/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #902 +/- ##
=======================================
Coverage 83.41% 83.41%
=======================================
Files 27 27
Lines 3636 3636
=======================================
Hits 3033 3033
Misses 603 603
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=footer). Last update [d9a5ef1...9aa139d](https://codecov.io/gh/simonw/datasette/pull/902?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",662439034,Don't install tests package,
https://github.com/simonw/datasette/issues/456#issuecomment-661524006,https://api.github.com/repos/simonw/datasette/issues/456,661524006,MDEyOklzc3VlQ29tbWVudDY2MTUyNDAwNg==,32467826,abeyerpath,2020-07-21T01:15:07Z,2020-07-21T01:15:07Z,CONTRIBUTOR,"Bumping this, as the previous fix is passing the wrong type, and not actually addressing the issue...
The `exclude` argument needs an iterable of packages instead of a single string (but since `str` is iterable, it's currently excluding packages `t`, `e`, and `s`.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",442327592,Installing installs the tests package,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660548780,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660548780,MDEyOklzc3VlQ29tbWVudDY2MDU0ODc4MA==,9599,simonw,2020-07-18T22:02:37Z,2020-07-18T23:05:56Z,MEMBER,https://github-to-sqlite.dogsheep.net/github/tags?_facet=repo,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660551397,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660551397,MDEyOklzc3VlQ29tbWVudDY2MDU1MTM5Nw==,9599,simonw,2020-07-18T22:27:32Z,2020-07-18T23:05:45Z,MEMBER,"```sql
with most_recent_releases as (
with ranked as (
select
repo,
tag_name,
published_at,
row_number() OVER (
partition BY repo
ORDER BY
published_at DESC
) rank
FROM
releases
)
select
*
from
ranked
where
rank = 1
)
select
repos.full_name as repo,
most_recent_releases.tag_name as release,
commits.committer_date as release_commit_date,
(
select
count(*)
from
commits c2
where
c2.repo = repos.id
and c2.committer_date > commits.committer_date
) as commits_since_release,
'https://github.com/' || repos.full_name || '/compare/' || most_recent_releases.tag_name || '...' || repos.default_branch as view_commits
from
most_recent_releases
join repos on most_recent_releases.repo = repos.id
join tags on tags.repo = repos.id
and tags.name = most_recent_releases.tag_name
join commits on tags.sha = commits.sha
order by
commits_since_release desc
```
repo | release | release_commit_date | commits_since_release | view_commits
-- | -- | -- | -- | --
simonw/datasette | 0.45 | 2020-07-01T21:43:07Z | 9 | https://github.com/simonw/datasette/compare/0.45...master
dogsheep/twitter-to-sqlite | 0.21.1 | 2020-04-30T18:20:43Z | 2 | https://github.com/dogsheep/twitter-to-sqlite/compare/0.21.1...master
dogsheep/github-to-sqlite | 2.3 | 2020-07-09T23:26:34Z | 2 | https://github.com/dogsheep/github-to-sqlite/compare/2.3...master
dogsheep/dogsheep-photos | 0.4.1 | 2020-05-25T20:11:20Z | 2 | https://github.com/dogsheep/dogsheep-photos/compare/0.4.1...master
dogsheep/swarm-to-sqlite | 0.3.1 | 2020-03-28T02:29:41Z | 1 | https://github.com/dogsheep/swarm-to-sqlite/compare/0.3.1...master
dogsheep/hacker-news-to-sqlite | 0.3.1 | 2020-03-21T22:39:34Z | 1 | https://github.com/dogsheep/hacker-news-to-sqlite/compare/0.3.1...master
simonw/sqlite-utils | 2.11 | 2020-07-08T17:36:07Z | 0 | https://github.com/simonw/sqlite-utils/compare/2.11...master
dogsheep/healthkit-to-sqlite | 0.5 | 2020-03-28T01:50:51Z | 0 | https://github.com/dogsheep/healthkit-to-sqlite/compare/0.5...master
dogsheep/inaturalist-to-sqlite | 0.2 | 2020-03-24T00:35:44Z | 0 | https://github.com/dogsheep/inaturalist-to-sqlite/compare/0.2...master
dogsheep/genome-to-sqlite | 0.1 | 2019-09-19T15:38:10Z | 0 | https://github.com/dogsheep/genome-to-sqlite/compare/0.1...master
dogsheep/pocket-to-sqlite | 0.2 | 2020-03-27T22:23:16Z | 0 | https://github.com/dogsheep/pocket-to-sqlite/compare/0.2...master
https://github-to-sqlite.dogsheep.net/github?sql=with+most_recent_releases+as+%28%0D%0A++with+ranked+as+%28%0D%0A++++select%0D%0A++++++repo%2C%0D%0A++++++tag_name%2C%0D%0A++++++published_at%2C%0D%0A++++++row_number%28%29+OVER+%28%0D%0A++++++++partition+BY+repo%0D%0A++++++++ORDER+BY%0D%0A++++++++++published_at+DESC%0D%0A++++++%29+rank%0D%0A++++FROM%0D%0A++++++releases%0D%0A++%29%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++ranked%0D%0A++where%0D%0A++++rank+%3D+1%0D%0A%29%0D%0Aselect%0D%0A++repos.full_name+as+repo%2C%0D%0A++most_recent_releases.tag_name+as+release%2C%0D%0A++commits.committer_date+as+release_commit_date%2C%0D%0A++%28%0D%0A++++select%0D%0A++++++count%28*%29%0D%0A++++from%0D%0A++++++commits+c2%0D%0A++++where%0D%0A++++++c2.repo+%3D+repos.id%0D%0A++++++and+c2.committer_date+%3E+commits.committer_date%0D%0A++%29+as+commits_since_release%2C%0D%0A++%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos.full_name+%7C%7C+%27%2Fcompare%2F%27+%7C%7C+most_recent_releases.tag_name+%7C%7C+%27...%27+%7C%7C+repos.default_branch+as+view_commits%0D%0Afrom%0D%0A++most_recent_releases%0D%0A++join+repos+on+most_recent_releases.repo+%3D+repos.id%0D%0A++join+tags+on+tags.repo+%3D+repos.id%0D%0A++and+tags.name+%3D+most_recent_releases.tag_name%0D%0A++join+commits+on+tags.sha+%3D+commits.sha%0D%0Aorder+by%0D%0A++commits_since_release+desc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554811,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660554811,MDEyOklzc3VlQ29tbWVudDY2MDU1NDgxMQ==,9599,simonw,2020-07-18T23:03:13Z,2020-07-18T23:03:13Z,MEMBER,https://github-to-sqlite.dogsheep.net/github/tags now shows a `repo` column instead of a `repo_id` column.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554299,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660554299,MDEyOklzc3VlQ29tbWVudDY2MDU1NDI5OQ==,9599,simonw,2020-07-18T22:58:24Z,2020-07-18T23:02:52Z,MEMBER,"Deploying the fixed version like this:
```
$ gcloud config set run/region us-central1
$ gcloud config set project datasette-222320
$ datasette publish cloudrun /tmp/github.db \
-m demo-metadata.json \
--service github-to-sqlite \
--install=py-gfm \
--install='datasette-search-all>=0.3' \
--install='datasette-render-markdown>=1.1.2' \
--install=datasette-pretty-json \
--install=datasette-json-html \
--install=datasette-vega
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660554162,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660554162,MDEyOklzc3VlQ29tbWVudDY2MDU1NDE2Mg==,9599,simonw,2020-07-18T22:56:58Z,2020-07-18T22:56:58Z,MEMBER,"Manually fixing the database:
```
$ wget 'https://github-to-sqlite.dogsheep.net/github.db'
--2020-07-18 15:52:33-- https://github-to-sqlite.dogsheep.net/github.db
Resolving github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)... 172.217.5.115
Connecting to github-to-sqlite.dogsheep.net (github-to-sqlite.dogsheep.net)|172.217.5.115|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14626816 (14M) [application/octet-stream]
Saving to: ‘github.db’
github.db 100%[============================================================================>] 13.95M 1.22MB/s in 18s
2020-07-18 15:52:53 (773 KB/s) - ‘github.db’ saved [14626816/14626816]
$ sqlite3 github.db
SQLite version 3.28.0 2019-04-15 14:49:49
Enter "".help"" for usage hints.
sqlite> drop table tags;
sqlite> ^D
$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils dogsheep/healthkit-to-sqlite dogsheep/swarm-to-sqlite dogsheep/twitter-to-sqlite dogsheep/inaturalist-to-sqlite dogsheep/google-takeout-to-sqlite dogsheep/github-to-sqlite dogsheep/genome-to-sqlite dogsheep/pocket-to-sqlite dogsheep/hacker-news-to-sqlite dogsheep/dogsheep-photos
$ sqlite-utils tables github.db --counts
[{""table"": ""users"", ""count"": 4048},
{""table"": ""repos"", ""count"": 210},
...
{""table"": ""stars"", ""count"": 4140},
{""table"": ""tags"", ""count"": 188}]
$ sqlite-utils rows github.db tags
[{""repo"": 107914493, ""name"": ""0.45"", ""sha"": ""f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb""},
{""repo"": 107914493, ""name"": ""0.45a5"", ""sha"": ""676bb64c877d73f8ff496cef4632f5a8a5a9283c""},
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553711,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660553711,MDEyOklzc3VlQ29tbWVudDY2MDU1MzcxMQ==,9599,simonw,2020-07-18T22:52:16Z,2020-07-18T22:52:16Z,MEMBER,"I think the best fix is to download the `github.db` database, manually fix it and then manually deploy it to Cloud Run from my laptop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/45#issuecomment-660553646,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/45,660553646,MDEyOklzc3VlQ29tbWVudDY2MDU1MzY0Ng==,9599,simonw,2020-07-18T22:51:41Z,2020-07-18T22:51:41Z,MEMBER,"I could fix this by putting `REFRESH_DB` in a commit message:
https://github.com/dogsheep/github-to-sqlite/blob/4ae4aa6f172344b19ff3513707195ee6d2654bd4/.github/workflows/deploy-demo.yml#L41-L46
But... doing so would lose the data I've collected in https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc concerning the first time each dependent repo was spotted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660429601,Fix the demo - it breaks because of the tags table change,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660547502,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660547502,MDEyOklzc3VlQ29tbWVudDY2MDU0NzUwMg==,9599,simonw,2020-07-18T21:50:37Z,2020-07-18T21:50:37Z,MEMBER,"```
$ github-to-sqlite tags tags.db simonw/datasette dogsheep/github-to-sqlite
$ sqlite-utils tables tags.db --counts
[{""table"": ""users"", ""count"": 2},
{""table"": ""licenses"", ""count"": 1},
{""table"": ""repos"", ""count"": 2},
{""table"": ""tags"", ""count"": 76},
{""table"": ""licenses_fts"", ""count"": 1},
{""table"": ""licenses_fts_data"", ""count"": 3},
{""table"": ""licenses_fts_idx"", ""count"": 1},
{""table"": ""licenses_fts_docsize"", ""count"": 1},
{""table"": ""licenses_fts_config"", ""count"": 1},
{""table"": ""repos_fts"", ""count"": 2},
{""table"": ""repos_fts_data"", ""count"": 3},
{""table"": ""repos_fts_idx"", ""count"": 1},
{""table"": ""repos_fts_docsize"", ""count"": 2},
{""table"": ""repos_fts_config"", ""count"": 1},
{""table"": ""users_fts"", ""count"": 2},
{""table"": ""users_fts_data"", ""count"": 3},
{""table"": ""users_fts_idx"", ""count"": 1},
{""table"": ""users_fts_docsize"", ""count"": 2},
{""table"": ""users_fts_config"", ""count"": 1}]
$ sqlite-utils rows tags.db tags
[{""repo_id"": 107914493, ""name"": ""0.45"", ""sha"": ""f1f581b7ffcd5d8f3ae6c1c654d813a6641410eb""},
{""repo_id"": 107914493, ""name"": ""0.45a5"", ""sha"": ""676bb64c877d73f8ff496cef4632f5a8a5a9283c""},
{""repo_id"": 107914493, ""name"": ""0.45a4"", ""sha"": ""265483173bc8341dc02c8b782b9b59d2ce8bbedb""},
{""repo_id"": 107914493, ""name"": ""0.45a3"", ""sha"": ""1f55a4a2b68fa65e56a28baeb7f44122fdeca7e7""},
{""repo_id"": 107914493, ""name"": ""0.45a2"", ""sha"": ""1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f""},
{""repo_id"": 107914493, ""name"": ""0.45a1"", ""sha"": ""b59b92b1b0517cf18fa748ff9d0a0bf86298dd43""},
{""repo_id"": 107914493, ""name"": ""0.45a0"", ""sha"": ""dda932d818b34ccab11730a76554f0a3748d8348""},
{""repo_id"": 107914493, ""name"": ""0.44"", ""sha"": ""b906030235efbdff536405d66078f4868ce0d3bd""},
{""repo_id"": 107914493, ""name"": ""0.43"", ""sha"": ""d56f402822df102f9cf1a9a056449d01a15e3aae""},
{""repo_id"": 107914493, ""name"": ""0.42"", ""sha"": ""af6c6c5d6f929f951c0e63bfd1c82e37a071b50f""},
{""repo_id"": 107914493, ""name"": ""0.41"", ""sha"": ""182e5c8745c94576718315f7596ccc81e5e2417b""},
{""repo_id"": 107914493, ""name"": ""0.40"", ""sha"": ""8da108193b08abf140716f8ac499f32309dfe9cf""},
{""repo_id"": 107914493, ""name"": ""0.39"", ""sha"": ""dedd775512daee49925882654f252df61a9e3b6d""},
{""repo_id"": 107914493, ""name"": ""0.38"", ""sha"": ""7e357abbc38dcc9d19a2f1df3252668a48e941e4""},
{""repo_id"": 107914493, ""name"": ""0.37.1"", ""sha"": ""be20e6991eac2baa9b43e9b26ae209bae805ede5""},
{""repo_id"": 107914493, ""name"": ""0.37"", ""sha"": ""c9e6841482b299fceadc5ad548c2dbf58a8f1227""},
{""repo_id"": 107914493, ""name"": ""0.36"", ""sha"": ""b031fe97636b80b05fec409ee1dffb7d044fd4e9""},
{""repo_id"": 107914493, ""name"": ""0.35"", ""sha"": ""30b6f71b306a43605c99bef79302ed5cb22d1924""},
{""repo_id"": 107914493, ""name"": ""0.34"", ""sha"": ""e7f60d2a9b59752e20de8412f7b0a3e9a5359a31""},
{""repo_id"": 107914493, ""name"": ""0.33"", ""sha"": ""59e7014c8a0f4102d7dc79f517540c55c49e1554""},
{""repo_id"": 107914493, ""name"": ""0.32"", ""sha"": ""a95bedb9c423fa6d772c93ef47bc40f13a5bea50""},
{""repo_id"": 107914493, ""name"": ""0.31.2"", ""sha"": ""b51f258d00bb3c3b401f15d46a1fbd50394dbe1c""},
{""repo_id"": 107914493, ""name"": ""0.31.1"", ""sha"": ""a22c7761b61baa61b8e3da7d30887468d61d6b83""},
{""repo_id"": 107914493, ""name"": ""0.31"", ""sha"": ""7f89928062b1a1fdb2625a946f7cd5161e597401""},
{""repo_id"": 107914493, ""name"": ""0.30.2"", ""sha"": ""2bf7ce5f517d772a16d7855a35a8a75d4456aad7""},
{""repo_id"": 107914493, ""name"": ""0.30.1"", ""sha"": ""3ca290e0db03bb4747e24203c445873f74512107""},
{""repo_id"": 107914493, ""name"": ""0.30"", ""sha"": ""8050f9e1ece9afd0236ad38c6458c12a4ad917e6""},
{""repo_id"": 107914493, ""name"": ""0.29.3"", ""sha"": ""0fc8afde0eb5ef677f4ac31601540d6168c8208d""},
{""repo_id"": 107914493, ""name"": ""0.29.2"", ""sha"": ""6abe6faff6b035e9334dd05f8c741ae9b7a47440""},
{""repo_id"": 107914493, ""name"": ""0.29.1"", ""sha"": ""2a94f3719fb2c4335fcda374fa92f87272b02d34""},
{""repo_id"": 107914493, ""name"": ""0.29"", ""sha"": ""fb7ee8e0ad59a15083234a48e935525f6e7257dd""},
{""repo_id"": 107914493, ""name"": ""0.28"", ""sha"": ""e518f76c5f5dd0138032bfb26387f5bb91086a3f""},
{""repo_id"": 107914493, ""name"": ""0.27.1"", ""sha"": ""3f3f29ac9afe7c41ffc48a3bd2af473a53eecc8a""},
{""repo_id"": 107914493, ""name"": ""0.27"", ""sha"": ""436b8bc1d17c2ab415800ab209204f94e7f7929e""},
{""repo_id"": 107914493, ""name"": ""0.26.2"", ""sha"": ""a418c8b44f82d456be523c8690cf7236bb648c22""},
{""repo_id"": 107914493, ""name"": ""0.26.1"", ""sha"": ""4722acc73ce761556b18f5dcbe36b7fef2ee2c69""},
{""repo_id"": 107914493, ""name"": ""0.26"", ""sha"": ""424e146697309a54c05d5d1ba1f840849ddbafdc""},
{""repo_id"": 107914493, ""name"": ""0.25.2"", ""sha"": ""b5128fc53fce6a1bf3b16bad9f318451bc1d1263""},
{""repo_id"": 107914493, ""name"": ""0.25.1"", ""sha"": ""3dc0b3fa8c9b9bd81540ffe20c8b7e7a72465274""},
{""repo_id"": 107914493, ""name"": ""0.25"", ""sha"": ""57a71377c992753327a16b417daf79df7f506dd1""},
{""repo_id"": 107914493, ""name"": ""0.24"", ""sha"": ""28872a1fa789f314b0342f4e6182f1c78d6e2bca""},
{""repo_id"": 107914493, ""name"": ""0.23.2"", ""sha"": ""6df6f712b36f0fe75694174906e31242427a8d1d""},
{""repo_id"": 107914493, ""name"": ""0.23.1"", ""sha"": ""dea86b9fba78e032ad09673e884e764387daf209""},
{""repo_id"": 107914493, ""name"": ""0.23"", ""sha"": ""e04f5b0d348ef7275a0a5ab9eb53527105132885""},
{""repo_id"": 107914493, ""name"": ""0.22.1"", ""sha"": ""5d6252788230d168ba09f379d1d2af867e3302ab""},
{""repo_id"": 107914493, ""name"": ""0.22"", ""sha"": ""558d9d7bfef3dd633eb16389281b67d42c9bdeef""},
{""repo_id"": 107914493, ""name"": ""0.21"", ""sha"": ""403211de632cd15f0820cc9399305fc43c187b47""},
{""repo_id"": 107914493, ""name"": ""0.20"", ""sha"": ""3a5d7951ce8f35118ffdd7f8d86e09b909e1218c""},
{""repo_id"": 107914493, ""name"": ""0.19"", ""sha"": ""ba9bfa583179c25aaef94b1f44da7eba74620b9a""},
{""repo_id"": 107914493, ""name"": ""0.18"", ""sha"": ""43ae15c0d14b3e968e8d5bfef72ac0c39783c3a2""},
{""repo_id"": 107914493, ""name"": ""0.17"", ""sha"": ""fb988ace7c7e2bee5ac142a0eab22431d0675a77""},
{""repo_id"": 107914493, ""name"": ""0.16"", ""sha"": ""b6539ff04502536bd1fa96e3b1430bdafc456826""},
{""repo_id"": 107914493, ""name"": ""0.15"", ""sha"": ""7706fe0c67aba5cfe905c7906cae9e0c43cd75b2""},
{""repo_id"": 107914493, ""name"": ""0.14"", ""sha"": ""2edc652df6d786e4f2c3f073e3567002d248be09""},
{""repo_id"": 107914493, ""name"": ""0.13"", ""sha"": ""c160f15c3937f8fbe581276f811e8c58f9137bb1""},
{""repo_id"": 107914493, ""name"": ""0.12"", ""sha"": ""51bdd67691bd69082ae7690af8b905f06050ee80""},
{""repo_id"": 107914493, ""name"": ""0.11"", ""sha"": ""b0f3d4e375655f0764f3137dbcede324f9bbc0cb""},
{""repo_id"": 107914493, ""name"": ""0.10"", ""sha"": ""5928c11ee798a232aa4096706cd47e639d1c9fc2""},
{""repo_id"": 107914493, ""name"": ""0.9"", ""sha"": ""d75f423b6fcfc074b7c6f8f7679da8876f181edd""},
{""repo_id"": 107914493, ""name"": ""0.8"", ""sha"": ""fe279ab7b4ae99dab295d5cf4d39ad06d782997e""},
{""repo_id"": 107914493, ""name"": ""0.7"", ""sha"": ""6b3b05b6db0d2a7b7cec8b8dbb4ddc5e12a376b2""},
{""repo_id"": 207052882, ""name"": ""2.3"", ""sha"": ""7090e43d804724ef3b31ae5ca9efd6ac05f76cbc""},
{""repo_id"": 207052882, ""name"": ""2.2"", ""sha"": ""4fe69783b55465e7692a807d3a02a710f69c9c42""},
{""repo_id"": 207052882, ""name"": ""2.1"", ""sha"": ""9d7aed336c8e62bf372caa800cb4aae3985cbae9""},
{""repo_id"": 207052882, ""name"": ""2.0"", ""sha"": ""44611df1524a03ce305405e5902c9615e3c73a72""},
{""repo_id"": 207052882, ""name"": ""1.1"", ""sha"": ""5cd34bd07d704487d48ac741ee5da5317afe88d2""},
{""repo_id"": 207052882, ""name"": ""1.0.1"", ""sha"": ""3b7ab5685de89fcb6fc92d320c0e24b17be05570""},
{""repo_id"": 207052882, ""name"": ""1.0"", ""sha"": ""1ea30c8fb1d080bd5e38c577e3ad20bb527a2fe6""},
{""repo_id"": 207052882, ""name"": ""0.7"", ""sha"": ""e35eec4343aa560c58c1634cc228d0d46c442304""},
{""repo_id"": 207052882, ""name"": ""0.6"", ""sha"": ""9eb737090fafd0e5a7e314be48402374d99e9828""},
{""repo_id"": 207052882, ""name"": ""0.5"", ""sha"": ""ae9035f8fe5aff1c54bff4c6b4c2e808a44f0f2a""},
{""repo_id"": 207052882, ""name"": ""0.4"", ""sha"": ""8c6251c31a05c58c2bfbef114247642d1b3dbb44""},
{""repo_id"": 207052882, ""name"": ""0.3"", ""sha"": ""f697f247468516aa4ee13b1862b59e0dba18d00f""},
{""repo_id"": 207052882, ""name"": ""0.2"", ""sha"": ""0fe96bc50fb3d7b1c7e4577db0ddf207eaeebbb9""},
{""repo_id"": 207052882, ""name"": ""0.1.1"", ""sha"": ""321e0284c64dc48b2143311009886293c05edb07""},
{""repo_id"": 207052882, ""name"": ""0.1"", ""sha"": ""7387c88a3f84704548e81d43b91615c02b61a957""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/dogsheep/github-to-sqlite/issues/43#issuecomment-660536265,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/43,660536265,MDEyOklzc3VlQ29tbWVudDY2MDUzNjI2NQ==,9599,simonw,2020-07-18T20:15:12Z,2020-07-18T20:15:12Z,MEMBER,"I want to create a SQL query which shows me all of my repositories that have commits that are NOT in the most recent release.
The releases table doesn't have enough information for this because it doesn't tell you the commit hash associated with each release, just the tag: https://github-to-sqlite.dogsheep.net/github/releases","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660355904,github-to-sqlite tags command for fetching tags,
https://github.com/simonw/datasette/issues/898#issuecomment-660419792,https://api.github.com/repos/simonw/datasette/issues/898,660419792,MDEyOklzc3VlQ29tbWVudDY2MDQxOTc5Mg==,9599,simonw,2020-07-18T03:57:46Z,2020-07-18T03:57:46Z,OWNER,"This requires some thought. There are various testing utilities that don't exist yet that plugins might benefit from - off the top of my head:
- `assert_permissions_checked`
- `assert_template_rendered`
I should resist the temptation to provide a reusable version of `make_app_client` that provides a fully configured Datasette instance because I need to be able to change the design of the Datasette `fixtures.db` test database without accidentally breaking any plugins that depend on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",659873662,datasette.utils.testing module,
https://github.com/simonw/datasette/issues/898#issuecomment-660419499,https://api.github.com/repos/simonw/datasette/issues/898,660419499,MDEyOklzc3VlQ29tbWVudDY2MDQxOTQ5OQ==,9599,simonw,2020-07-18T03:55:13Z,2020-07-18T03:55:13Z,OWNER,Maybe I should make `httpx` a testing dependency of Datasette itself. It's usage is already encouraged in plugins by https://datasette.readthedocs.io/en/stable/testing_plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",659873662,datasette.utils.testing module,
https://github.com/simonw/datasette/issues/897#issuecomment-660318063,https://api.github.com/repos/simonw/datasette/issues/897,660318063,MDEyOklzc3VlQ29tbWVudDY2MDMxODA2Mw==,9599,simonw,2020-07-17T20:16:02Z,2020-07-17T20:16:02Z,OWNER,Documentation here: https://datasette.readthedocs.io/en/latest/internals.html#request-object,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",659580487,Request method for retrieving the unparsed request body,
https://github.com/simonw/datasette/issues/896#issuecomment-659773897,https://api.github.com/repos/simonw/datasette/issues/896,659773897,MDEyOklzc3VlQ29tbWVudDY1OTc3Mzg5Nw==,9599,simonw,2020-07-17T01:26:08Z,2020-07-17T01:26:08Z,OWNER,I manually tested it with those plugins and it seems to interoperate just fine - since both of those use `
` tags for the cases that I care about so they're already expecting white-space to be pre wrapped in some way.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659734703,https://api.github.com/repos/simonw/datasette/issues/896,659734703,MDEyOklzc3VlQ29tbWVudDY1OTczNDcwMw==,9599,simonw,2020-07-16T23:34:57Z,2020-07-16T23:34:57Z,OWNER,"I'm worried about how this will interact with some of the plugins:
* https://github.com/simonw/datasette-json-html
* https://github.com/simonw/datasette-pretty-json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659615034,https://api.github.com/repos/simonw/datasette/issues/896,659615034,MDEyOklzc3VlQ29tbWVudDY1OTYxNTAzNA==,9599,simonw,2020-07-16T19:14:07Z,2020-07-16T19:14:07Z,OWNER,"Demo: https://srccon-2020.datasette.io/srccon?sql=select+id%2C+day%2C+time%2C+event_name%2C+event_description%2C+facilitators+from+sessions+order+by+event_dtstart+limit+101
I really like this:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659610687,https://api.github.com/repos/simonw/datasette/issues/896,659610687,MDEyOklzc3VlQ29tbWVudDY1OTYxMDY4Nw==,9599,simonw,2020-07-16T19:05:43Z,2020-07-16T19:05:43Z,OWNER,I'm going to give this a go - if it turns out to be a bad idea I can revert it back out again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/895#issuecomment-659085528,https://api.github.com/repos/simonw/datasette/issues/895,659085528,MDEyOklzc3VlQ29tbWVudDY1OTA4NTUyOA==,9599,simonw,2020-07-16T00:32:47Z,2020-07-16T00:32:47Z,OWNER,"This was added in https://github.com/simonw/datasette/commit/504196341c49840270bd75ea1a1871ef386ba7ea - here's the relevant code (which only applies on the table page, not the query page):
https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/table.py#L196-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657747959,SQL query output should show numeric values in a different colour,
https://github.com/simonw/datasette/issues/892#issuecomment-657268433,https://api.github.com/repos/simonw/datasette/issues/892,657268433,MDEyOklzc3VlQ29tbWVudDY1NzI2ODQzMw==,9599,simonw,2020-07-12T20:02:17Z,2020-07-12T20:02:35Z,OWNER,"Fixed https://datasette.readthedocs.io/en/latest/
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",655465863,"""latest"" in new documentation navbar is invisible",
https://github.com/simonw/datasette/issues/892#issuecomment-657268051,https://api.github.com/repos/simonw/datasette/issues/892,657268051,MDEyOklzc3VlQ29tbWVudDY1NzI2ODA1MQ==,9599,simonw,2020-07-12T19:58:24Z,2020-07-12T19:58:24Z,OWNER,"```css
.wy-side-nav-search > div.version {
margin-top: -.4045em;
margin-bottom: .809em;
font-weight: normal;
color: rgba(255,255,255,0.3);
}
```
Fix can go here: https://github.com/simonw/datasette/blob/ee0ef016523a765b6ef6eaa43cad9ad568f78ae4/docs/_static/css/custom.css#L1-L3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",655465863,"""latest"" in new documentation navbar is invisible",
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-656363548,https://api.github.com/repos/simonw/sqlite-utils/issues/114,656363548,MDEyOklzc3VlQ29tbWVudDY1NjM2MzU0OA==,9599,simonw,2020-07-09T21:37:28Z,2020-07-09T21:37:28Z,OWNER,"I'm going to add a second method `.transform_table_sql(...)` - which returns the SQL that would have been executed but does NOT execute it.
Advanced callers can use this to include their own additional steps in the same transaction - e.g. recreating views or triggers.
More importantly it gives me a useful hook for writing some unit tests against the generated SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655898722,https://api.github.com/repos/simonw/sqlite-utils/issues/121,655898722,MDEyOklzc3VlQ29tbWVudDY1NTg5ODcyMg==,79913,tsibley,2020-07-09T04:53:08Z,2020-07-09T04:53:08Z,CONTRIBUTOR,"Yep, I agree that makes more sense for backwards compat and more casual use cases. I think it should be possible for the Database/Queryable methods to DTRT based on seeing if it's within a context-manager-managed transaction.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652961907,Improved (and better documented) support for transactions,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655786374,MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA==,9599,simonw,2020-07-08T22:16:54Z,2020-07-08T22:16:54Z,OWNER,"According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.
I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.
Foreign key relationships are important: it should handle these automatically as effectively as possible.
Likewise trigger changes: need to think about what this means.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655785396,MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng==,9599,simonw,2020-07-08T22:14:10Z,2020-07-08T22:14:10Z,OWNER,"Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated `INSERT INTO ... SELECT ...` query:
```python
def transform_table(
self,
columns=None,
rename=None,
change_type=None,
pk=None,
foreign_keys=None,
column_order=None,
not_null=None,
defaults=None,
hash_id=None,
extracts=None,
):
assert self.exists(), ""Cannot transform a table that doesn't exist yet""
columns = columns or self.columns_dict
if rename is not None or change_type is not None:
columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}
new_table_name = ""{}_new_{}"".format(self.name, os.urandom(6).hex())
previous_columns = set(self.columns_dict.keys())
with self.db.conn:
columns = {name: value for (name, value) in columns.items()}
new_table = self.db.create_table(
new_table_name,
columns,
pk=pk,
foreign_keys=foreign_keys,
column_order=column_order,
not_null=not_null,
defaults=defaults,
hash_id=hash_id,
extracts=extracts,
)
# Copy across data - but only for columns that exist in both
new_columns = set(columns.keys())
columns_to_copy = new_columns.intersection(previous_columns)
copy_sql = ""INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]"".format(
new_table=new_table_name,
old_table=self.name,
old_cols="", "".join(""[{}]"".format(col) for col in columns_to_copy),
new_cols="", "".join(""[{}]"".format(rename.get(col, col)) for col in columns_to_copy),
)
self.db.conn.execute(copy_sql)
# Drop the old table
self.db.conn.execute(""DROP TABLE [{}]"".format(self.name))
# Rename the new one
self.db.conn.execute(
""ALTER TABLE [{}] RENAME TO [{}]"".format(new_table_name, self.name)
)
return self
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655783875,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655783875,MDEyOklzc3VlQ29tbWVudDY1NTc4Mzg3NQ==,9599,simonw,2020-07-08T22:09:51Z,2020-07-08T22:10:16Z,OWNER,I can have a convenient `change_type={...}` parameter for changing column types too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655782477,MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw==,9599,simonw,2020-07-08T22:06:23Z,2020-07-08T22:06:23Z,OWNER,"Thinking about the method signature:
```python
def transform_table(
self,
columns,
pk=None,
foreign_keys=None,
column_order=None,
not_null=None,
defaults=None,
hash_id=None,
extracts=None,
):
```
This requires the caller to provide the exact set of columns for the new table.
It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.
Even better: allow column renaming using an optional `rename={...}` argument:
```python
db[""dogs""].transform_table(rename={""name"": ""dog_name""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655778058,MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA==,9599,simonw,2020-07-08T21:54:30Z,2020-07-08T21:54:30Z,OWNER,"Don't forget this step:
> If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677909,MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ==,9599,simonw,2020-07-08T18:16:39Z,2020-07-08T18:16:39Z,OWNER,"Since neither the term ""transform"" or ""migrate"" are used in the codebase at the moment, I think I'll go with `.transform_table()` - that leaves the term ""migrate"" available for any future database migrations system (similar to Django's).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677396,MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng==,9599,simonw,2020-07-08T18:15:39Z,2020-07-08T18:15:39Z,OWNER,"Alternative possible names:
- `.transform_table()`
- `.migrate()`
- `.transform()`
I'm torn between `.migrate_table()` and `.transform_table()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677099,MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ==,9599,simonw,2020-07-08T18:15:02Z,2020-07-08T18:15:02Z,OWNER,"I'm not so keen on that chained API - it's pretty complicated.
Here's an idea for a much simpler interface. Essentially it lets you say ""take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them"":
```python
db[""mytable""].migrate_table({""id"": int, ""name"": str""}, pk=""id"")
```
The `migrate_table()` method would take the same exact signature as the `table.create()` method: https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L615-L625","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910,https://api.github.com/repos/simonw/sqlite-utils/issues/119,655674910,MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA==,9599,simonw,2020-07-08T18:10:18Z,2020-07-08T18:10:18Z,OWNER,"This will work similar to how `.add_foreign_keys()` works: turn on `writable_schema` and rewrite the `sql` for that table in the `sqlite_master` table.
Here's that code today - it could be adapted to include removal of foreign keys that we no longer want:
https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L391-L401","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770,Ability to remove a foreign key,
https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896,https://api.github.com/repos/simonw/sqlite-utils/issues/121,655673896,MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng==,9599,simonw,2020-07-08T18:08:11Z,2020-07-08T18:08:11Z,OWNER,"I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction.
The one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:
1. It will break all of the existing code out there
2. It doesn't match to how I most commonly use this library - as an interactive tool in a Jupyter notebook, where I'm generally working against a brand new scratch database and any errors don't actually matter
So... how about this: IF you wrap your code in a `with db:` block then the `.insert()` and suchlike methods expect you to manage transactions yourself. But if you don't use the context manager they behave like they do at the moment (or maybe a bit more sensibly).
That way existing code works as it does today, lazy people like me can call `.insert()` without thinking about transactions, but people writing actual production code (as opposed to Jupyter hacks) have a sensible way to take control of the transactions themselves.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652961907,Improved (and better documented) support for transactions,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655653292,MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg==,9599,simonw,2020-07-08T17:26:02Z,2020-07-08T17:26:02Z,OWNER,"Awesome, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655652679,https://api.github.com/repos/simonw/sqlite-utils/issues/121,655652679,MDEyOklzc3VlQ29tbWVudDY1NTY1MjY3OQ==,79913,tsibley,2020-07-08T17:24:46Z,2020-07-08T17:24:46Z,CONTRIBUTOR,"Better transaction handling would be really great. Some of my thoughts on implementing better transaction discipline are in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728.
My preferences:
- Each CLI command should operate in a single transaction so that either the whole thing succeeds or the whole thing is rolled back. This avoids partially completed operations when an error occurs part way through processing. Partially completed operations are typically much harder to recovery from gracefully and may cause inconsistent data states.
- The Python API should be transaction-agnostic and rely on the caller to coordinate transactions. Only the caller knows how individual insert, create, update, etc operations/methods should be bundled conceptually into transactions. When the caller is the CLI, for example, that bundling would be at the CLI command-level. Other callers might want to break up operations into multiple transactions. Transactions are usually most useful when controlled at the application-level (like logging configuration) instead of the library level. The library needs to provide an API that's conducive to transaction use, though.
- The Python API should provide a context manager to provide consistent transactions handling with more useful defaults than Python's `sqlite3` module. The latter issues implicit `BEGIN` statements by default for most DML (`INSERT`, `UPDATE`, `DELETE`, … but not `SELECT`, I believe), but **not** DDL (`CREATE TABLE`, `DROP TABLE`, `CREATE VIEW`, …). Notably, the `sqlite3` module doesn't issue the implicit `BEGIN` until the first DML statement. It _does not_ issue it when entering the `with conn` block, like other DBAPI2-compatible modules do. The `with conn` block for `sqlite3` only arranges to commit or rollback an existing transaction when exiting. Including DDL and `SELECT`s in transactions is important for operation consistency, though. There are several existing bugs.python.org tickets about this and future changes are in the works, but sqlite-utils can provide its own API sooner. sqlite-utils's `Database` class could itself be a context manager (built on the `sqlite3` connection context manager) which additionally issues an explicit `BEGIN` when entering. This would then let Python API callers do something like:
```python
db = sqlite_utils.Database(path)
with db: # ← BEGIN issued here by Database.__enter__
db.insert(…)
db.create_view(…)
# ← COMMIT/ROLLBACK issue here by sqlite3.connection.__exit__
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652961907,Improved (and better documented) support for transactions,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655643078,MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA==,79913,tsibley,2020-07-08T17:05:59Z,2020-07-08T17:05:59Z,CONTRIBUTOR,"> The only thing missing from this PR is updates to the documentation.
Ah, yes, thanks for this reminder! I've repushed with doc bits added.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655290625,MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ==,9599,simonw,2020-07-08T05:15:45Z,2020-07-08T05:15:45Z,OWNER,"Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686,https://api.github.com/repos/simonw/sqlite-utils/issues/120,655289686,MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng==,9599,simonw,2020-07-08T05:13:11Z,2020-07-08T05:13:11Z,OWNER,"This is an excellent fix, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652816158,Fix query command's support for DML,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655286864,MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA==,9599,simonw,2020-07-08T05:05:27Z,2020-07-08T05:05:36Z,OWNER,"The only thing missing from this PR is updates to the documentation. Those need to go in two places:
- In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts
- In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data
Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655284168,MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA==,9599,simonw,2020-07-08T04:58:00Z,2020-07-08T04:58:00Z,OWNER,"Oops didn't mean to click ""close"" there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655284054,MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA==,9599,simonw,2020-07-08T04:57:38Z,2020-07-08T04:57:38Z,OWNER,Thoughts on transactions would be much appreciated in #121 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655283393,MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw==,9599,simonw,2020-07-08T04:55:18Z,2020-07-08T04:55:18Z,OWNER,"This is a really good idea - and thank you for the detailed discussion in the pull request.
I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:
with db.conn:
db[""table""].insert(...)
But it's not documented and I've not though very hard about it!
I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.
Lots to discuss here. I'll start a new issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655239728,MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA==,79913,tsibley,2020-07-08T02:16:42Z,2020-07-08T02:16:42Z,CONTRIBUTOR,"I fixed my original oops by moving the `DELETE FROM $table` out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next.
I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making `Table.insert_all` (or other operations) consistent and robust in the face of errors.
For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted):
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..4107ceb 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1028,6 +1028,11 @@ class Table(Queryable):
batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
self.last_rowid = None
self.last_pk = None
+ with self.db.conn:
+ # Explicit BEGIN is necessary because Python's sqlite3 doesn't
+ # issue implicit BEGINs for DDL, only DML. We mix DDL and DML
+ # below and might execute DDL first, e.g. for table creation.
+ self.db.conn.execute(""BEGIN"")
if truncate and self.exists():
self.db.conn.execute(""DELETE FROM [{}];"".format(self.name))
for chunk in chunks(itertools.chain([first_record], records), batch_size):
@@ -1038,7 +1043,11 @@ class Table(Queryable):
# Use the first batch to derive the table names
column_types = suggest_column_types(chunk)
column_types.update(columns or {})
- self.create(
+ # Not self.create() because that is wrapped in its own
+ # transaction and Python's sqlite3 doesn't support
+ # nested transactions.
+ self.db.create_table(
+ self.name,
column_types,
pk,
foreign_keys,
@@ -1139,7 +1148,6 @@ class Table(Queryable):
flat_values = list(itertools.chain(*values))
queries_and_params = [(sql, flat_values)]
- with self.db.conn:
for query, params in queries_and_params:
try:
result = self.db.conn.execute(query, params)
```
but that fails in tests because other methods call `insert/upsert/insert_all/upsert_all` in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed).
Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the `sqlite_utils/db.py` code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make `sqlite_utils/cli.py` begin and commit a transaction in each `@cli.command` function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to _require_ callers to begin a transaction rather than just immediately calling methods.
There is also the caveat that for each transaction, an explicit `BEGIN` is also necessary so that DDL as well as DML (as well as `SELECT`s) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655052451,MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ==,79913,tsibley,2020-07-07T18:45:23Z,2020-07-07T18:45:23Z,CONTRIBUTOR,"Ah, I see the problem. The truncate is inside a loop I didn't realize was there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655018966,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655018966,MDEyOklzc3VlQ29tbWVudDY1NTAxODk2Ng==,79913,tsibley,2020-07-07T17:41:06Z,2020-07-07T17:41:06Z,CONTRIBUTOR,"Hmm, while tests pass, this may not work as intended on larger datasets. Looking into it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/datasette/issues/784#issuecomment-654424704,https://api.github.com/repos/simonw/datasette/issues/784,654424704,MDEyOklzc3VlQ29tbWVudDY1NDQyNDcwNA==,9599,simonw,2020-07-06T19:31:53Z,2020-07-06T19:31:53Z,OWNER,Documentation: https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653966670,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653966670,MDEyOklzc3VlQ29tbWVudDY1Mzk2NjY3MA==,9599,simonw,2020-07-06T01:07:02Z,2020-07-06T01:07:02Z,MEMBER,OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962708,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962708,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjcwOA==,9599,simonw,2020-07-06T00:43:10Z,2020-07-06T00:43:10Z,MEMBER,I bet it's datasette-search-all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962669,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962669,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjY2OQ==,9599,simonw,2020-07-06T00:42:57Z,2020-07-06T00:42:57Z,MEMBER,"https://github-to-sqlite.dogsheep.net/-/plugins
```json
[
{
""name"": ""datasette-json-html"",
""static"": false,
""templates"": false,
""version"": ""0.6"",
""hooks"": [
""prepare_connection"",
""render_cell""
]
},
{
""name"": ""datasette-render-markdown"",
""static"": false,
""templates"": false,
""version"": ""1.1.2"",
""hooks"": [
""extra_template_vars"",
""render_cell""
]
},
{
""name"": ""datasette-pretty-json"",
""static"": false,
""templates"": false,
""version"": ""0.2"",
""hooks"": [
""render_cell""
]
},
{
""name"": ""datasette-search-all"",
""static"": false,
""templates"": true,
""version"": ""0.2.1"",
""hooks"": [
""asgi_wrapper"",
""extra_template_vars""
]
},
{
""name"": ""datasette-vega"",
""static"": true,
""templates"": false,
""version"": ""0.6.2"",
""hooks"": [
""extra_css_urls"",
""extra_js_urls""
]
}
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962530,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962530,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjUzMA==,9599,simonw,2020-07-06T00:42:13Z,2020-07-06T00:42:13Z,MEMBER,So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653962418,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653962418,MDEyOklzc3VlQ29tbWVudDY1Mzk2MjQxOA==,9599,simonw,2020-07-06T00:41:38Z,2020-07-06T00:41:38Z,MEMBER,"https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653960989,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653960989,MDEyOklzc3VlQ29tbWVudDY1Mzk2MDk4OQ==,9599,simonw,2020-07-06T00:32:34Z,2020-07-06T00:32:34Z,MEMBER,Same error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/dogsheep/github-to-sqlite/issues/41#issuecomment-653947916,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/41,653947916,MDEyOklzc3VlQ29tbWVudDY1Mzk0NzkxNg==,9599,simonw,2020-07-05T22:40:47Z,2020-07-05T22:40:47Z,MEMBER,Might be that it's not got enough RAM. I'll try deploying to a larger instance.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651159727,Demo is failing to deploy,
https://github.com/simonw/datasette/pull/890#issuecomment-653314465,https://api.github.com/repos/simonw/datasette/issues/890,653314465,MDEyOklzc3VlQ29tbWVudDY1MzMxNDQ2NQ==,9599,simonw,2020-07-03T03:07:41Z,2020-07-03T03:07:41Z,OWNER,"This is an excellent fix. Thanks!
Not sure why codecov is complaining. I'm going to merge it as-is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",650305298,Load only python files from plugins-dir.,
https://github.com/simonw/datasette/pull/890#issuecomment-653309545,https://api.github.com/repos/simonw/datasette/issues/890,653309545,MDEyOklzc3VlQ29tbWVudDY1MzMwOTU0NQ==,22429695,codecov[bot],2020-07-03T02:52:25Z,2020-07-03T03:03:00Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=h1) Report
> Merging [#890](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/57879dc8b346a435804a9e45ffaacbf2a0228bc6&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `80.00%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/890/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #890 +/- ##
==========================================
- Coverage 83.42% 83.40% -0.02%
==========================================
Files 27 27
Lines 3632 3634 +2
==========================================
+ Hits 3030 3031 +1
- Misses 602 603 +1
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/890/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.99% <80.00%> (-0.17%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=footer). Last update [57879dc...745af3b](https://codecov.io/gh/simonw/datasette/pull/890?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",650305298,Load only python files from plugins-dir.,
https://github.com/simonw/datasette/pull/848#issuecomment-643711117,https://api.github.com/repos/simonw/datasette/issues/848,643711117,MDEyOklzc3VlQ29tbWVudDY0MzcxMTExNw==,22429695,codecov[bot],2020-06-14T03:05:55Z,2020-07-03T02:44:09Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=h1) Report
> Merging [#848](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/57879dc8b346a435804a9e45ffaacbf2a0228bc6&el=desc) will **decrease** coverage by `0.60%`.
> The diff coverage is `0.00%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/848/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #848 +/- ##
==========================================
- Coverage 83.42% 82.82% -0.61%
==========================================
Files 27 26 -1
Lines 3632 3540 -92
==========================================
- Hits 3030 2932 -98
- Misses 602 608 +6
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `71.34% <0.00%> (-0.89%)` | :arrow_down: |
| [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `77.77% <0.00%> (-3.40%)` | :arrow_down: |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.58% <0.00%> (-1.58%)` | :arrow_down: |
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `90.90% <0.00%> (-0.42%)` | :arrow_down: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.84% <0.00%> (-0.09%)` | :arrow_down: |
| [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <0.00%> (ø)` | |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <0.00%> (ø)` | |
| [datasette/default\_permissions.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `100.00% <0.00%> (ø)` | |
| [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | | |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.40% <0.00%> (+<0.01%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/simonw/datasette/pull/848/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=footer). Last update [57879dc...0d100d1](https://codecov.io/gh/simonw/datasette/pull/848?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638270441,Reload support for config_dir mode.,
https://github.com/simonw/datasette/issues/889#issuecomment-653002499,https://api.github.com/repos/simonw/datasette/issues/889,653002499,MDEyOklzc3VlQ29tbWVudDY1MzAwMjQ5OQ==,49260,amjith,2020-07-02T13:22:13Z,2020-07-02T13:22:13Z,CONTRIBUTOR,"I was able to narrow this down to the fact that lifespan protocol is turned on.
I see the workaround you've used here: https://github.com/simonw/datasette-debug-asgi/commit/72d568d32a3159c763ce908c0b269736935c6987
If so, maybe it's time to update some of the asg_wrapper [plugins](https://datasette.readthedocs.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649907676,asgi_wrapper plugin hook is crashing at startup,
https://github.com/simonw/datasette/issues/889#issuecomment-652990131,https://api.github.com/repos/simonw/datasette/issues/889,652990131,MDEyOklzc3VlQ29tbWVudDY1Mjk5MDEzMQ==,49260,amjith,2020-07-02T12:58:11Z,2020-07-02T13:00:18Z,CONTRIBUTOR,"FWIW, this error does NOT happen in datasette 0.45a4.
It only started on 0.45a5","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649907676,asgi_wrapper plugin hook is crashing at startup,
https://github.com/simonw/datasette/issues/886#issuecomment-652732460,https://api.github.com/repos/simonw/datasette/issues/886,652732460,MDEyOklzc3VlQ29tbWVudDY1MjczMjQ2MA==,9599,simonw,2020-07-02T01:52:02Z,2020-07-02T01:52:02Z,OWNER,In investigating this I'm not convinced 500 errors are being correctly raised by errors in canned writable queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649429772,Reconsider how _actor_X magic parameter deals with missing values,
https://github.com/simonw/datasette/issues/886#issuecomment-652731459,https://api.github.com/repos/simonw/datasette/issues/886,652731459,MDEyOklzc3VlQ29tbWVudDY1MjczMTQ1OQ==,9599,simonw,2020-07-02T01:48:08Z,2020-07-02T01:48:08Z,OWNER,"A common error with this (and other) magic parameters is for the database query to result in the following:
You did not supply a value for binding 3.
This is a pretty crufty error. I'm inclined to say that ANY missing or invalid magic parameter should be treated as a `None` value instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649429772,Reconsider how _actor_X magic parameter deals with missing values,
https://github.com/simonw/datasette/issues/887#issuecomment-652711822,https://api.github.com/repos/simonw/datasette/issues/887,652711822,MDEyOklzc3VlQ29tbWVudDY1MjcxMTgyMg==,9599,simonw,2020-07-02T00:31:33Z,2020-07-02T00:31:33Z,OWNER,"If a canned query has a title defined that will be used instead: https://latest.datasette.io/fixtures/neighborhood_search
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/issues/887#issuecomment-652711562,https://api.github.com/repos/simonw/datasette/issues/887,652711562,MDEyOklzc3VlQ29tbWVudDY1MjcxMTU2Mg==,9599,simonw,2020-07-02T00:30:43Z,2020-07-02T00:30:43Z,OWNER,"Demo has updated: https://latest.datasette.io/fixtures/magic_parameters
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/pull/883#issuecomment-652710178,https://api.github.com/repos/simonw/datasette/issues/883,652710178,MDEyOklzc3VlQ29tbWVudDY1MjcxMDE3OA==,9599,simonw,2020-07-02T00:25:44Z,2020-07-02T00:25:44Z,OWNER,This is a great idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/issues/887#issuecomment-652709199,https://api.github.com/repos/simonw/datasette/issues/887,652709199,MDEyOklzc3VlQ29tbWVudDY1MjcwOTE5OQ==,9599,simonw,2020-07-02T00:21:54Z,2020-07-02T00:21:54Z,OWNER,"Example in the live demo: https://latest.datasette.io/fixtures/magic_parameters
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/issues/885#issuecomment-652681996,https://api.github.com/repos/simonw/datasette/issues/885,652681996,MDEyOklzc3VlQ29tbWVudDY1MjY4MTk5Ng==,9599,simonw,2020-07-01T22:44:47Z,2020-07-01T22:44:47Z,OWNER,https://simonwillison.net/2020/Jul/1/datasette-045/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649373451,Blog entry about the release,
https://github.com/simonw/datasette/issues/882#issuecomment-652663177,https://api.github.com/repos/simonw/datasette/issues/882,652663177,MDEyOklzc3VlQ29tbWVudDY1MjY2MzE3Nw==,9599,simonw,2020-07-01T21:48:08Z,2020-07-01T21:48:08Z,OWNER,https://datasette.readthedocs.io/en/latest/changelog.html#v0-45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648673556,Release notes for 0.45,
https://github.com/simonw/datasette/issues/880#issuecomment-652646487,https://api.github.com/repos/simonw/datasette/issues/880,652646487,MDEyOklzc3VlQ29tbWVudDY1MjY0NjQ4Nw==,9599,simonw,2020-07-01T21:05:48Z,2020-07-01T21:05:48Z,OWNER,"I've been testing the WIP using this in the console:
```javascript
fetch('/data/add_name.json', {
method: 'POST',
body: 'name=XXXfetch',
credentials: 'omit',
headers: {'Content-Type': 'application/x-www-form-urlencoded'}
})
.then(response => console.log(response))
```
Against a canned query configured like this:
```yaml
databases:
data:
queries:
add_name:
sql: insert into names (name) values (:name)
write: true
```
I haven't got it to work yet. Latest error is this one:
```
INFO: Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
Traceback (most recent call last):
File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 975, in route_path
await response.asgi_send(send)
AttributeError: 'tuple' object has no attribute 'asgi_send'
INFO: 127.0.0.1:49938 - ""POST /data/add_name.json HTTP/1.1"" 500 Internal Server Error
```
It looks like I'm going to have to rethink how the `BaseView` code around tables, formats and hashes is structured in order to fix this. That's a big refactoring! I'm moving this to a new milestone for Datasette 0.46.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/882#issuecomment-652604569,https://api.github.com/repos/simonw/datasette/issues/882,652604569,MDEyOklzc3VlQ29tbWVudDY1MjYwNDU2OQ==,9599,simonw,2020-07-01T19:27:17Z,2020-07-01T19:27:17Z,OWNER,Don't forget to update the news in the README.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648673556,Release notes for 0.45,
https://github.com/simonw/datasette/issues/877#issuecomment-652597975,https://api.github.com/repos/simonw/datasette/issues/877,652597975,MDEyOklzc3VlQ29tbWVudDY1MjU5Nzk3NQ==,9599,simonw,2020-07-01T19:12:15Z,2020-07-01T19:12:15Z,OWNER,The latest release of https://github.com/simonw/datasette-auth-tokens (0.2) now supports SQL configuration of tokens.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652520496,https://api.github.com/repos/simonw/datasette/issues/877,652520496,MDEyOklzc3VlQ29tbWVudDY1MjUyMDQ5Ng==,9599,simonw,2020-07-01T16:26:52Z,2020-07-01T16:26:52Z,OWNER,Tokens get verified by plugins. So far there's only one: https://github.com/simonw/datasette-auth-tokens - which has you hard-coding plugins in a configuration file. I have a issue there to add support for database-backed tokens too: https://github.com/simonw/datasette-auth-tokens/issues/1,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/pull/883#issuecomment-652394742,https://api.github.com/repos/simonw/datasette/issues/883,652394742,MDEyOklzc3VlQ29tbWVudDY1MjM5NDc0Mg==,3243482,abdusco,2020-07-01T12:41:13Z,2020-07-01T12:41:13Z,CONTRIBUTOR,"Well tests need to be updated.
I need to get tests working on Windows.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/pull/883#issuecomment-652311990,https://api.github.com/repos/simonw/datasette/issues/883,652311990,MDEyOklzc3VlQ29tbWVudDY1MjMxMTk5MA==,22429695,codecov[bot],2020-07-01T09:40:40Z,2020-07-01T09:40:40Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=h1) Report
> Merging [#883](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/676bb64c877d73f8ff496cef4632f5a8a5a9283c&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/883/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #883 +/- ##
=======================================
Coverage 83.42% 83.42%
=======================================
Files 27 27
Lines 3632 3632
=======================================
Hits 3030 3030
Misses 602 602
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=footer). Last update [676bb64...251884f](https://codecov.io/gh/simonw/datasette/pull/883?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/pull/883#issuecomment-652297139,https://api.github.com/repos/simonw/datasette/issues/883,652297139,MDEyOklzc3VlQ29tbWVudDY1MjI5NzEzOQ==,3243482,abdusco,2020-07-01T09:11:29Z,2020-07-01T09:11:29Z,CONTRIBUTOR,"Turns out we should include hidden tables in the result dict, or we're breaking tests. I've committed a refactor https://github.com/simonw/datasette/pull/883/commits/4f06e1bf6fbe4b73be770b87f610bf7c0e6e3ea7","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/issues/877#issuecomment-652255960,https://api.github.com/repos/simonw/datasette/issues/877,652255960,MDEyOklzc3VlQ29tbWVudDY1MjI1NTk2MA==,3243482,abdusco,2020-07-01T07:52:25Z,2020-07-01T08:10:00Z,CONTRIBUTOR,"I am calling the API from another origin, so injecting CSRF token into templates wouldn't work.
EDIT:
I'll try the new version, it sounds promising","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652261382,https://api.github.com/repos/simonw/datasette/issues/877,652261382,MDEyOklzc3VlQ29tbWVudDY1MjI2MTM4Mg==,3243482,abdusco,2020-07-01T08:03:17Z,2020-07-01T08:03:23Z,CONTRIBUTOR,Bearer tokens sound interesting. Where do tokens come from? An auth provider of my choosing? How do they get verified?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652182990,https://api.github.com/repos/simonw/datasette/issues/877,652182990,MDEyOklzc3VlQ29tbWVudDY1MjE4Mjk5MA==,9599,simonw,2020-07-01T04:29:38Z,2020-07-01T04:42:59Z,OWNER,"Have you tried the method described here? https://datasette.readthedocs.io/en/latest/internals.html#csrf-protection - I'm happy to bulk out that section of the documentation if that doesn't help solve your problem.
I just closed #835 which should make CSRF protection easier to work with - it won't interfere with requests without cookies or requests with `Authentication: Bearer token` tokens. See also https://github.com/simonw/asgi-csrf/issues/11
You can try out `pip install datasette==0.45a5` to get those features. Hopefully releasing a full 0.45 tomorrow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652166115,https://api.github.com/repos/simonw/datasette/issues/877,652166115,MDEyOklzc3VlQ29tbWVudDY1MjE2NjExNQ==,3243482,abdusco,2020-07-01T03:28:07Z,2020-07-01T03:28:07Z,CONTRIBUTOR,"Does this mean custom routes get to expose endpoints accepting POST requests? I've tried earlier to add some POST endpoints, but requests were being rejected by Datasette due to CSRF","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/812#issuecomment-652165709,https://api.github.com/repos/simonw/datasette/issues/812,652165709,MDEyOklzc3VlQ29tbWVudDY1MjE2NTcwOQ==,9599,simonw,2020-07-01T03:26:35Z,2020-07-01T03:26:35Z,OWNER,"This case may not be covered without extra work:
https://github.com/simonw/datasette/blob/3ec5b1abf6afa2d22a3378092809a1a8c0249d26/datasette/views/database.py#L122-L123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634112607,Ability to customize what happens when a view permission fails,
https://github.com/simonw/datasette/issues/812#issuecomment-652163450,https://api.github.com/repos/simonw/datasette/issues/812,652163450,MDEyOklzc3VlQ29tbWVudDY1MjE2MzQ1MA==,9599,simonw,2020-07-01T03:18:51Z,2020-07-01T03:20:28Z,OWNER,"This can be a plugin hook:
```python
@hookspec
def forbidden(datasette, request, message, send):
""Custom response for a 403 forbidden error""
```
If the hook returns a `Response` object, it will be returned to the user. Plugins are likely to want to return a redirect response.
Maybe the hook can instead use the `send` argument to respond to the request and return `True` which means ""I've responded to this""?
I'm going to leave `send` off for the moment - I can add that in the future if it turns out it would have been a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634112607,Ability to customize what happens when a view permission fails,
https://github.com/simonw/datasette/issues/880#issuecomment-652162722,https://api.github.com/repos/simonw/datasette/issues/880,652162722,MDEyOklzc3VlQ29tbWVudDY1MjE2MjcyMg==,9599,simonw,2020-07-01T03:16:07Z,2020-07-01T03:16:07Z,OWNER,The response from this will never be a 302 - it will always be a 200 if the response worked or a 400 for bad parameters or a 500 for errors. The body returned will always be in JSON format.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/859#issuecomment-652160909,https://api.github.com/repos/simonw/datasette/issues/859,652160909,MDEyOklzc3VlQ29tbWVudDY1MjE2MDkwOQ==,3243482,abdusco,2020-07-01T03:09:32Z,2020-07-01T03:10:21Z,CONTRIBUTOR,"I've just realized Datasette tries to count hidden tables too. There are 5 visible tables, 25 hidden tables, which I haven't realize earlier to consider their effect. I've turned off counting for hidden tables to see if it has any effect.
What's the point of counting FTS tables?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/835#issuecomment-652159398,https://api.github.com/repos/simonw/datasette/issues/835,652159398,MDEyOklzc3VlQ29tbWVudDY1MjE1OTM5OA==,9599,simonw,2020-07-01T03:03:51Z,2020-07-01T03:03:51Z,OWNER,I'm going to add some tests for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/876#issuecomment-652106227,https://api.github.com/repos/simonw/datasette/issues/876,652106227,MDEyOklzc3VlQ29tbWVudDY1MjEwNjIyNw==,9599,simonw,2020-06-30T23:49:55Z,2020-06-30T23:50:04Z,OWNER,"Done: https://latest.datasette.io/-/patterns
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647879783,Add log out link to the pattern portfolio,
https://github.com/simonw/datasette/issues/879#issuecomment-652105722,https://api.github.com/repos/simonw/datasette/issues/879,652105722,MDEyOklzc3VlQ29tbWVudDY1MjEwNTcyMg==,9599,simonw,2020-06-30T23:48:06Z,2020-06-30T23:48:06Z,OWNER,Updated documentation: https://datasette.readthedocs.io/en/latest/pages.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648569227,Database page documentation still talks about hashes in URLs,
https://github.com/simonw/datasette/issues/832#issuecomment-652103895,https://api.github.com/repos/simonw/datasette/issues/832,652103895,MDEyOklzc3VlQ29tbWVudDY1MjEwMzg5NQ==,9599,simonw,2020-06-30T23:41:22Z,2020-06-30T23:41:22Z,OWNER,I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651999516,https://api.github.com/repos/simonw/datasette/issues/832,651999516,MDEyOklzc3VlQ29tbWVudDY1MTk5OTUxNg==,9599,simonw,2020-06-30T19:33:49Z,2020-06-30T21:34:59Z,OWNER,"Tests needed for this:
- If a user has view table but NOT view database / view instance, can they view the table page?
- If a user has view canned query but NOT view database / view instance, can they view the canned query page?
- If a user has view database but NOT view instance, can they view the database page?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651995453,https://api.github.com/repos/simonw/datasette/issues/832,651995453,MDEyOklzc3VlQ29tbWVudDY1MTk5NTQ1Mw==,9599,simonw,2020-06-30T19:25:13Z,2020-06-30T19:25:26Z,OWNER,I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651994978,https://api.github.com/repos/simonw/datasette/issues/832,651994978,MDEyOklzc3VlQ29tbWVudDY1MTk5NDk3OA==,9599,simonw,2020-06-30T19:24:12Z,2020-06-30T19:24:12Z,OWNER,"Hah... but check_permission` is a method on `BaseView`. Here are the various permission methods at the moment:
https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14
And on BaseView:
https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651993977,https://api.github.com/repos/simonw/datasette/issues/832,651993977,MDEyOklzc3VlQ29tbWVudDY1MTk5Mzk3Nw==,9599,simonw,2020-06-30T19:22:06Z,2020-06-30T19:22:06Z,OWNER,`permission_allowed` is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on `datasette.`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651993537,https://api.github.com/repos/simonw/datasette/issues/832,651993537,MDEyOklzc3VlQ29tbWVudDY1MTk5MzUzNw==,9599,simonw,2020-06-30T19:21:15Z,2020-06-30T19:21:15Z,OWNER,"I could rename `permission_allowed()` to `check_permission()` and have a complementary `check_permissions()` method.
This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary `permission_allowed()` alias which prints a deprecation warning to the console, then remove that at 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651992737,https://api.github.com/repos/simonw/datasette/issues/832,651992737,MDEyOklzc3VlQ29tbWVudDY1MTk5MjczNw==,9599,simonw,2020-06-30T19:19:33Z,2020-06-30T19:20:02Z,OWNER,"I already have this method on Datasette:
```python
async def permission_allowed(self, actor, action, resource=None, default=False):
```
What would be a good method name that complements that and indicates ""check a list of permissions in order""? Should it even run against the request or should you have to hand it `request.actor`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/877#issuecomment-651984989,https://api.github.com/repos/simonw/datasette/issues/877,651984989,MDEyOklzc3VlQ29tbWVudDY1MTk4NDk4OQ==,9599,simonw,2020-06-30T19:03:25Z,2020-06-30T19:03:25Z,OWNER,Relevant: #835,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-651984355,https://api.github.com/repos/simonw/datasette/issues/877,651984355,MDEyOklzc3VlQ29tbWVudDY1MTk4NDM1NQ==,9599,simonw,2020-06-30T19:02:15Z,2020-06-30T19:02:15Z,OWNER,"https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf
> Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form.
Sounds like regular CSRF protection to me.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/805#issuecomment-651302221,https://api.github.com/repos/simonw/datasette/issues/805,651302221,MDEyOklzc3VlQ29tbWVudDY1MTMwMjIyMQ==,9599,simonw,2020-06-29T19:02:45Z,2020-06-29T19:05:26Z,OWNER,"No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way.
So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good.
Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/805#issuecomment-651301202,https://api.github.com/repos/simonw/datasette/issues/805,651301202,MDEyOklzc3VlQ29tbWVudDY1MTMwMTIwMg==,9599,simonw,2020-06-29T19:00:37Z,2020-06-29T19:00:37Z,OWNER,"How about a blog? Pre-configured canned queries that are only available to `""root""`, plus datasette-template-sql and default templates for the index page and blog entry pages.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/875#issuecomment-651293559,https://api.github.com/repos/simonw/datasette/issues/875,651293559,MDEyOklzc3VlQ29tbWVudDY1MTI5MzU1OQ==,9599,simonw,2020-06-29T18:43:50Z,2020-06-29T18:43:50Z,OWNER,"
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/873#issuecomment-651203178,https://api.github.com/repos/simonw/datasette/issues/873,651203178,MDEyOklzc3VlQ29tbWVudDY1MTIwMzE3OA==,9599,simonw,2020-06-29T15:44:38Z,2020-06-29T15:44:54Z,OWNER,I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact `-p 0 --root` combination which isn't going to affect many people at all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-651193594,https://api.github.com/repos/simonw/datasette/issues/873,651193594,MDEyOklzc3VlQ29tbWVudDY1MTE5MzU5NA==,9599,simonw,2020-06-29T15:27:46Z,2020-06-29T15:27:46Z,OWNER,Uninstalling `datasette-debug-asgi` caused the server to startup correctly again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-651193131,https://api.github.com/repos/simonw/datasette/issues/873,651193131,MDEyOklzc3VlQ29tbWVudDY1MTE5MzEzMQ==,9599,simonw,2020-06-29T15:27:00Z,2020-06-29T15:27:00Z,OWNER,"Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486
Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding `lifespan=""on""` to the call to `uvicorn.run()`.
When I did that here:
`uvicorn.run(ds.app(), host=host, port=port, log_level=""info"", lifespan=""on"")`
The server failed to start with this error:
```
INFO: Started server process [68849]
INFO: Waiting for application startup.
ERROR: Exception in 'lifespan' protocol
Traceback (most recent call last):
File "".../uvicorn/lifespan/on.py"", line 48, in main
await app(scope, self.receive, self.send)
File "".../uvicorn/middleware/proxy_headers.py"", line 45, in __call__
return await self.app(scope, receive, send)
File "".../datasette_debug_asgi.py"", line 9, in wrapped_app
if scope[""path""] == ""/-/asgi-scope"":
KeyError: 'path'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650910137,https://api.github.com/repos/simonw/datasette/issues/873,650910137,MDEyOklzc3VlQ29tbWVudDY1MDkxMDEzNw==,9599,simonw,2020-06-29T05:16:32Z,2020-06-29T05:16:32Z,OWNER,I'm not convinced that function is ever actually being called - I added a `print()` statement to it and it's not executing. I don't think the tests cover it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650909476,https://api.github.com/repos/simonw/datasette/issues/873,650909476,MDEyOklzc3VlQ29tbWVudDY1MDkwOTQ3Ng==,9599,simonw,2020-06-29T05:14:08Z,2020-06-29T05:14:08Z,OWNER,"I already have a `AsgiLifespan` class:
https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905
It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894
Could that startup function also output the `--root` login URL, if needed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650909136,https://api.github.com/repos/simonw/datasette/issues/873,650909136,MDEyOklzc3VlQ29tbWVudDY1MDkwOTEzNg==,9599,simonw,2020-06-29T05:12:58Z,2020-06-29T05:12:58Z,OWNER,"On startup Datasette currently outputs:
```
INFO: Waiting for application startup.
INFO: ASGI 'lifespan' protocol appears unsupported.
INFO: Application startup complete.
```
So the ASGI lifespan protocol is almost certainly the right way to solve this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650908854,https://api.github.com/repos/simonw/datasette/issues/873,650908854,MDEyOklzc3VlQ29tbWVudDY1MDkwODg1NA==,9599,simonw,2020-06-29T05:12:04Z,2020-06-29T05:12:04Z,OWNER,Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic `--root` link a second after the server starts up somehow.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650908534,https://api.github.com/repos/simonw/datasette/issues/873,650908534,MDEyOklzc3VlQ29tbWVudDY1MDkwODUzNA==,9599,simonw,2020-06-29T05:11:06Z,2020-06-29T05:11:06Z,OWNER,"Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650907323,https://api.github.com/repos/simonw/datasette/issues/873,650907323,MDEyOklzc3VlQ29tbWVudDY1MDkwNzMyMw==,9599,simonw,2020-06-29T05:07:16Z,2020-06-29T05:07:16Z,OWNER,"This line is interesting: is this a hook I can attach to somehow?
```python
await self.lifespan.startup()
```
From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650906533,https://api.github.com/repos/simonw/datasette/issues/873,650906533,MDEyOklzc3VlQ29tbWVudDY1MDkwNjUzMw==,9599,simonw,2020-06-29T05:04:44Z,2020-06-29T05:04:44Z,OWNER,The challenge is... can we run our own custom code after that line has executed that has access to `server` and can hence access `server.servers[0].sockets[0].getsockname()[1]` to find the port?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650906318,https://api.github.com/repos/simonw/datasette/issues/873,650906318,MDEyOklzc3VlQ29tbWVudDY1MDkwNjMxOA==,9599,simonw,2020-06-29T05:04:04Z,2020-06-29T05:04:12Z,OWNER,"Within uvicorn it does this:
```python
if port == 0:
port = server.sockets[0].getsockname()[1]
```
That `server` variable is later stashed here:
```
self.servers = [server]
```
Where `self` is the instance of `class Server` - which is the class that Uvicorn instantiates and calls `.run()` on when we do `uvicorn.run()` here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650905399,https://api.github.com/repos/simonw/datasette/issues/873,650905399,MDEyOklzc3VlQ29tbWVudDY1MDkwNTM5OQ==,9599,simonw,2020-06-29T05:01:03Z,2020-06-29T05:01:03Z,OWNER,This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/875#issuecomment-650899265,https://api.github.com/repos/simonw/datasette/issues/875,650899265,MDEyOklzc3VlQ29tbWVudDY1MDg5OTI2NQ==,9599,simonw,2020-06-29T04:34:32Z,2020-06-29T04:34:32Z,OWNER,"From https://github.com/simonw/datasette/issues/840#issuecomment-643454625
> Another problem: what to display in the ""you are logged in as"", since we don't dictate an actor design.
>
> I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.
>
> The default will look for the first available of the following keys:
>
> * display
> * name
> * username
> * login
> * id","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/875#issuecomment-650898808,https://api.github.com/repos/simonw/datasette/issues/875,650898808,MDEyOklzc3VlQ29tbWVudDY1MDg5ODgwOA==,9599,simonw,2020-06-29T04:32:31Z,2020-06-29T04:33:30Z,OWNER,"I could borrow the implementation for this from `datasette-auth-github`
https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html
```html+jinja
{% extends ""default:base.html"" %}
{% block extra_head %}
{% endblock %}
{% block nav %}
{{ super() }}
{% if auth and auth.username %}
{% endif %}
{% endblock %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/840#issuecomment-650895874,https://api.github.com/repos/simonw/datasette/issues/840,650895874,MDEyOklzc3VlQ29tbWVudDY1MDg5NTg3NA==,9599,simonw,2020-06-29T04:18:59Z,2020-06-29T04:19:11Z,OWNER,"Now just need the ""Logged in as: XXX <logout>"" navigation item.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/simonw/datasette/issues/840#issuecomment-650891502,https://api.github.com/repos/simonw/datasette/issues/840,650891502,MDEyOklzc3VlQ29tbWVudDY1MDg5MTUwMg==,9599,simonw,2020-06-29T03:58:08Z,2020-06-29T03:58:08Z,OWNER,"Step one: a ""logout"" page at `/-/logout` - which shows you a single CSRF-protected ""logout"" button if you do a GET against it and logs you out if you do a POST against it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/simonw/datasette/issues/805#issuecomment-650891257,https://api.github.com/repos/simonw/datasette/issues/805,650891257,MDEyOklzc3VlQ29tbWVudDY1MDg5MTI1Nw==,9599,simonw,2020-06-29T03:56:48Z,2020-06-29T03:56:48Z,OWNER,Using `datasette-glitch` and the new https://github.com/simonw/datasette-write - currently running on `datasette==0.45a4` - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the `/-/write` interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/864#issuecomment-650847013,https://api.github.com/repos/simonw/datasette/issues/864,650847013,MDEyOklzc3VlQ29tbWVudDY1MDg0NzAxMw==,9599,simonw,2020-06-29T00:41:55Z,2020-06-29T00:41:55Z,OWNER,To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650846625,https://api.github.com/repos/simonw/datasette/issues/864,650846625,MDEyOklzc3VlQ29tbWVudDY1MDg0NjYyNQ==,9599,simonw,2020-06-29T00:39:47Z,2020-06-29T00:39:47Z,OWNER,"I think the fix is to move the `""show_messages""` variable to here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650846473,https://api.github.com/repos/simonw/datasette/issues/864,650846473,MDEyOklzc3VlQ29tbWVudDY1MDg0NjQ3Mw==,9599,simonw,2020-06-29T00:39:04Z,2020-06-29T00:39:04Z,OWNER,"Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends `base.html`. For example, this code in a plugin:
```python
return Response.html(
await datasette.render_template(
""write.html"",
{""databases"": databases, ""sql"": request.args.get(""sql"") or """"},
request=request,
)
)
```
This won't display messages. The reason is that the messages are made available to the template context in the `BaseView.render()` method here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650842514,https://api.github.com/repos/simonw/datasette/issues/864,650842514,MDEyOklzc3VlQ29tbWVudDY1MDg0MjUxNA==,9599,simonw,2020-06-29T00:12:59Z,2020-06-29T00:12:59Z,OWNER,"> I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
https://github.com/simonw/datasette/issues/870#issuecomment-650842381","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/870#issuecomment-650842381,https://api.github.com/repos/simonw/datasette/issues/870,650842381,MDEyOklzc3VlQ29tbWVudDY1MDg0MjM4MQ==,9599,simonw,2020-06-29T00:12:07Z,2020-06-29T00:12:07Z,OWNER,I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650838972,https://api.github.com/repos/simonw/datasette/issues/870,650838972,MDEyOklzc3VlQ29tbWVudDY1MDgzODk3Mg==,9599,simonw,2020-06-28T23:46:40Z,2020-06-28T23:46:40Z,OWNER,I'm going to create the single `Request()` instance in the `DatasetteRouter` class - at the beginning of the `route_path` method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650838691,https://api.github.com/repos/simonw/datasette/issues/870,650838691,MDEyOklzc3VlQ29tbWVudDY1MDgzODY5MQ==,9599,simonw,2020-06-28T23:44:12Z,2020-06-28T23:44:25Z,OWNER,"This code is interesting:
https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955
I want to change the signature of that `return await view(new_scope, receive, send)` method to instead take `(request, send)` - so I can have a single shared request object that's created just once per HTTP request.
The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared `Request` instance? Should its `.scope` be replaced with alternative scopes as it travels through the codebase?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650834666,https://api.github.com/repos/simonw/datasette/issues/870,650834666,MDEyOklzc3VlQ29tbWVudDY1MDgzNDY2Ng==,9599,simonw,2020-06-28T23:07:19Z,2020-06-28T23:07:19Z,OWNER,So now the problem is simpler: I need to get `BaseView` to a state where it can accept a shared `request` object and it can be used in conjunction with `register_routes()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650834251,https://api.github.com/repos/simonw/datasette/issues/870,650834251,MDEyOklzc3VlQ29tbWVudDY1MDgzNDI1MQ==,9599,simonw,2020-06-28T23:03:28Z,2020-06-28T23:03:28Z,OWNER,"I'm going to ditch that `AsgiView` class too, by combining it into `BaseView`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650820068,https://api.github.com/repos/simonw/datasette/issues/870,650820068,MDEyOklzc3VlQ29tbWVudDY1MDgyMDA2OA==,9599,simonw,2020-06-28T20:52:09Z,2020-06-28T20:53:00Z,OWNER,"Maybe I could add a `as_request_view` method as an alternative to `as_asgi`:
https://github.com/simonw/datasette/blob/a8bcafc1775c8a8655b365ae22a3d64f6361c74a/datasette/utils/asgi.py#L150-L174
Or I could teach the `Router` to spot the `dispatch_request` method and call it directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/847#issuecomment-650819895,https://api.github.com/repos/simonw/datasette/issues/847,650819895,MDEyOklzc3VlQ29tbWVudDY1MDgxOTg5NQ==,9599,simonw,2020-06-28T20:50:21Z,2020-06-28T20:50:21Z,OWNER,I'm happy enough with https://codecov.io/gh/simonw/datasette that I'm not going to spend any more time on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/870#issuecomment-650818309,https://api.github.com/repos/simonw/datasette/issues/870,650818309,MDEyOklzc3VlQ29tbWVudDY1MDgxODMwOQ==,9599,simonw,2020-06-28T20:36:28Z,2020-06-28T20:36:52Z,OWNER,"Since `AsgiRouter` is only used as the super-class of the `DatasetteRouter` class maybe I should get rid of `AsgiRouter` entirely - no point in having a Datasette-specific subclass of it if the parent class isn't ever used by anything else.
I could also rename it to just `Router` which is a nicer name than `DatasetteRouter`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650818086,https://api.github.com/repos/simonw/datasette/issues/870,650818086,MDEyOklzc3VlQ29tbWVudDY1MDgxODA4Ng==,9599,simonw,2020-06-28T20:34:33Z,2020-06-28T20:34:33Z,OWNER,"The key to all of this may be the `DatasetteRouter` class. It deals with `scope` right now but if it internally dealt with `request` that could be enough to fix #864 by adding logic needed by the `.add_message()` mechanism.
https://github.com/simonw/datasette/blob/0991ea75cc7b265389aa8362414a305ba532d31a/datasette/app.py#L904-L938","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650815278,https://api.github.com/repos/simonw/datasette/issues/870,650815278,MDEyOklzc3VlQ29tbWVudDY1MDgxNTI3OA==,9599,simonw,2020-06-28T20:09:07Z,2020-06-28T20:11:21Z,OWNER,"There's a lot of complex logic in the `DataView` class, which handles conditionally returning content as `.json` or as HTML or as `.csv`.
That view subclasses `AsgiView` which is itself request-aware, so maybe I don't need to reconsider how those classes work - just figure out how to hook them up with `register_routes`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/871#issuecomment-650812444,https://api.github.com/repos/simonw/datasette/issues/871,650812444,MDEyOklzc3VlQ29tbWVudDY1MDgxMjQ0NA==,9599,simonw,2020-06-28T19:43:27Z,2020-06-28T19:43:27Z,OWNER,"Currently:
> `_timestamp_epoch`
>
> The number of seconds since the Unix epoch.
>
> `_timestamp_date_utc`
>
> The date in UTC, e.g. `2020-06-01`
>
> `_timestamp_datetime_utc`
>
> The ISO 8601 datetime in UTC, e.g. `2020-06-24T18:01:07Z`
I'm going to rename them to:
- `_now_epoch`
- `_now_date_utc`
- `_now_datetime_utc`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646840273,Rename the _timestamp magic parameters to _now,
https://github.com/simonw/datasette/issues/834#issuecomment-650811919,https://api.github.com/repos/simonw/datasette/issues/834,650811919,MDEyOklzc3VlQ29tbWVudDY1MDgxMTkxOQ==,9599,simonw,2020-06-28T19:38:50Z,2020-06-28T19:38:50Z,OWNER,"I have two plugins in progress that use this hook now:
- https://github.com/simonw/datasette-init creates tables and views on startup
- https://github.com/simonw/datasette-glitch outputs the login-as-root secret link on Glitch","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/805#issuecomment-650784162,https://api.github.com/repos/simonw/datasette/issues/805,650784162,MDEyOklzc3VlQ29tbWVudDY1MDc4NDE2Mg==,9599,simonw,2020-06-28T15:48:32Z,2020-06-28T15:48:32Z,OWNER,https://github.com/simonw/datasette-glitch is my new plugin that outputs the root login link on Glitch when the server starts.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/834#issuecomment-643657067,https://api.github.com/repos/simonw/datasette/issues/834,643657067,MDEyOklzc3VlQ29tbWVudDY0MzY1NzA2Nw==,9599,simonw,2020-06-13T17:59:42Z,2020-06-28T04:01:52Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#startup-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/842#issuecomment-650684635,https://api.github.com/repos/simonw/datasette/issues/842,650684635,MDEyOklzc3VlQ29tbWVudDY1MDY4NDYzNQ==,9599,simonw,2020-06-28T03:30:31Z,2020-06-28T03:30:31Z,OWNER,Live demo: https://latest.datasette.io/fixtures/magic_parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/805#issuecomment-650681496,https://api.github.com/repos/simonw/datasette/issues/805,650681496,MDEyOklzc3VlQ29tbWVudDY1MDY4MTQ5Ng==,9599,simonw,2020-06-28T03:11:51Z,2020-06-28T03:11:51Z,OWNER,I can use magic parameters from #842 in this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/842#issuecomment-650679100,https://api.github.com/repos/simonw/datasette/issues/842,650679100,MDEyOklzc3VlQ29tbWVudDY1MDY3OTEwMA==,9599,simonw,2020-06-28T03:00:44Z,2020-06-28T03:00:44Z,OWNER,I'm going to add some canned queries to the `metadata.json` used by the live demo that illustrate this feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650678951,https://api.github.com/repos/simonw/datasette/issues/842,650678951,MDEyOklzc3VlQ29tbWVudDY1MDY3ODk1MQ==,9599,simonw,2020-06-28T02:59:52Z,2020-06-28T02:59:52Z,OWNER,"Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters
Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/pull/869#issuecomment-650600176,https://api.github.com/repos/simonw/datasette/issues/869,650600176,MDEyOklzc3VlQ29tbWVudDY1MDYwMDE3Ng==,22429695,codecov[bot],2020-06-27T18:41:31Z,2020-06-28T02:54:21Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=h1) Report
> Merging [#869](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c&el=desc) will **increase** coverage by `0.23%`.
> The diff coverage is `90.62%`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/869/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #869 +/- ##
==========================================
+ Coverage 82.99% 83.23% +0.23%
==========================================
Files 26 27 +1
Lines 3547 3609 +62
==========================================
+ Hits 2944 3004 +60
- Misses 603 605 +2
```
| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.45% <86.36%> (-1.88%)` | :arrow_down: |
| [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <91.17%> (ø)` | |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.07% <100.00%> (+0.81%)` | :arrow_up: |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.87% <100.00%> (+0.02%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=footer). Last update [1bb33da...9e693a7](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646734280,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650648434,https://api.github.com/repos/simonw/datasette/issues/842,650648434,MDEyOklzc3VlQ29tbWVudDY1MDY0ODQzNA==,9599,simonw,2020-06-27T23:27:35Z,2020-06-27T23:37:38Z,OWNER,I'm going to rename `_request_X` to `_header_X` as that better reflects what it now does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/pull/868#issuecomment-650600606,https://api.github.com/repos/simonw/datasette/issues/868,650600606,MDEyOklzc3VlQ29tbWVudDY1MDYwMDYwNg==,9599,simonw,2020-06-27T18:44:28Z,2020-06-27T18:44:28Z,OWNER,"This is really exciting! Thanks so much for looking into this.
I'm interested in moving CI for this repo over to GitHub Actions, so I'd be fine with you getting this to work as an Action rather than through Travis. If you can get it working in Travis though I'll happily land that and figure out how to convert that to GitHub Actions later on.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646448486,initial windows ci setup,
https://github.com/simonw/datasette/issues/835#issuecomment-650598710,https://api.github.com/repos/simonw/datasette/issues/835,650598710,MDEyOklzc3VlQ29tbWVudDY1MDU5ODcxMA==,9599,simonw,2020-06-27T18:32:22Z,2020-06-27T18:32:22Z,OWNER,"Skipping CSRF on `Authorization: Bearer xxx` headers also makes sense for JWT applications, which tend to send JWTs using that form of header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/842#issuecomment-650593122,https://api.github.com/repos/simonw/datasette/issues/842,650593122,MDEyOklzc3VlQ29tbWVudDY1MDU5MzEyMg==,9599,simonw,2020-06-27T18:03:02Z,2020-06-27T18:03:10Z,OWNER,"> Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.
This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.
https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650458857,https://api.github.com/repos/simonw/datasette/issues/842,650458857,MDEyOklzc3VlQ29tbWVudDY1MDQ1ODg1Nw==,9599,simonw,2020-06-27T00:11:04Z,2020-06-27T00:11:04Z,OWNER,Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650455793,https://api.github.com/repos/simonw/datasette/issues/842,650455793,MDEyOklzc3VlQ29tbWVudDY1MDQ1NTc5Mw==,9599,simonw,2020-06-26T23:57:30Z,2020-06-27T00:00:16Z,OWNER,"Maybe I should ship a default `_scope_headers_...` parameter instead, which reads from a dictionary of `scope[""headers""]` - https://asgi-scope.now.sh/ shows what those look like.
```
{'client': ('148.64.98.14', 0),
'headers': [[b'host', b'asgi-scope.now.sh'],
[b'x-forwarded-for', b'148.64.98.14'],
[b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'],
[b'x-forwarded-host', b'asgi-scope.now.sh'],
[b'accept',
b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
b'webp,*/*;q=0.8'],
[b'x-real-ip', b'148.64.98.14'],
[b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'],
[b'upgrade-insecure-requests', b'1'],
[b'x-vercel-trace', b'sfo1'],
[b'x-forwarded-proto', b'https'],
[b'accept-language', b'en-US,en;q=0.5'],
[b'user-agent',
b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko'
b'/20100101 Firefox/77.0'],
[b'x-vercel-forwarded-for', b'148.64.98.14'],
[b'accept-encoding', b'gzip, deflate, br'],
[b'dnt', b'1'],
[b'te', b'trailers']],
'http_version': '1.1',
'method': 'GET',
'path': '/',
'query_string': b'',
'raw_path': b'/',
'root_path': '',
'scheme': 'https',
'server': ('asgi-scope.now.sh', 80),
'type': 'http'}
```
I'm going to have `_request_X` actually mean ""find the first value for X in `scope[""headers""`]"" - with underscores converted to hyphens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650455353,https://api.github.com/repos/simonw/datasette/issues/842,650455353,MDEyOklzc3VlQ29tbWVudDY1MDQ1NTM1Mw==,9599,simonw,2020-06-26T23:55:40Z,2020-06-26T23:55:40Z,OWNER,"`_request_ip` is actually quite hard to implement - should it take into account things like the `x-forwarded-for` header?
It probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment.
As such I think I'll leave that for a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/867#issuecomment-649931714,https://api.github.com/repos/simonw/datasette/issues/867,649931714,MDEyOklzc3VlQ29tbWVudDY0OTkzMTcxNA==,9599,simonw,2020-06-26T03:12:51Z,2020-06-26T03:12:51Z,OWNER,"Here's the relevant code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/datasette/app.py#L1057-L1070
And the relevant test code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/test_plugins.py#L567-L573
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/plugins/my_plugin.py#L162-L196","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",645975649,register_routes() should support non-async view functions too,
https://github.com/simonw/datasette/issues/842#issuecomment-649014757,https://api.github.com/repos/simonw/datasette/issues/842,649014757,MDEyOklzc3VlQ29tbWVudDY0OTAxNDc1Nw==,9599,simonw,2020-06-24T19:15:46Z,2020-06-24T19:31:52Z,OWNER,I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646271834,https://api.github.com/repos/simonw/datasette/issues/842,646271834,MDEyOklzc3VlQ29tbWVudDY0NjI3MTgzNA==,9599,simonw,2020-06-18T19:49:41Z,2020-06-24T18:49:22Z,OWNER,"But then what kind of magic parameters might plugins want to add?
Here's a crazy idea: `_scrapedcontent_url` - it would look for the `url` column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function `scrapedcontent()` would need to optionally be sent the full row dictionary being inserted too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646270702,https://api.github.com/repos/simonw/datasette/issues/842,646270702,MDEyOklzc3VlQ29tbWVudDY0NjI3MDcwMg==,9599,simonw,2020-06-18T19:47:19Z,2020-06-24T18:48:48Z,OWNER,"Brainstorming more potential magic parameters:
* `_actor_id`
* `_actor_name`
* `_request_ip`
* `_request_user_agent`
* `_cookie_cookiename`
* `_signedcookie_cookiename` - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, `_signedcookie_cookiename_mynamespace`. Might not need special signed cookie support since `actor` is already usually from a signed cookie.
* `_timestamp_unix` (not happy with these names yet)
* `_timestamp_localtime`
* `_timestamp_datetime`
* `_timestamp_utc`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-649000075,https://api.github.com/repos/simonw/datasette/issues/842,649000075,MDEyOklzc3VlQ29tbWVudDY0OTAwMDA3NQ==,9599,simonw,2020-06-24T18:46:36Z,2020-06-24T18:47:37Z,OWNER,"Another magic parameter that would be useful would be `_random`. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.
Maybe `_random_chars_128` to create a 128 character long random string (using `os.urandom(64).hex()`).
This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately `_random_chars:128` wouldn't work because these parameters are used in a SQLite query where `:` has special meaning: `insert into blah (secret) values (:_random_chars:128)` wouldn't make sense.
Actually this is already supported by the proposed design - `_random_chars_128` would become `random(""chars_128"")` so the `random()` function could split off the 128 itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/865#issuecomment-648998264,https://api.github.com/repos/simonw/datasette/issues/865,648998264,MDEyOklzc3VlQ29tbWVudDY0ODk5ODI2NA==,9599,simonw,2020-06-24T18:43:02Z,2020-06-24T18:43:02Z,OWNER,Thanks for the bug report. Yes I think #838 may be the same issue. Will investigate.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/858#issuecomment-648997857,https://api.github.com/repos/simonw/datasette/issues/858,648997857,MDEyOklzc3VlQ29tbWVudDY0ODk5Nzg1Nw==,9599,simonw,2020-06-24T18:42:10Z,2020-06-24T18:42:10Z,OWNER,I really need to get myself a Windows 10 development environment working so I can dig into this kind of bug properly. I have a gaming PC lying around that I could re-task for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/pull/866#issuecomment-648818707,https://api.github.com/repos/simonw/datasette/issues/866,648818707,MDEyOklzc3VlQ29tbWVudDY0ODgxODcwNw==,22429695,codecov[bot],2020-06-24T13:26:14Z,2020-06-24T13:26:14Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=h1) Report
> Merging [#866](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/866/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #866 +/- ##
=======================================
Coverage 82.99% 82.99%
=======================================
Files 26 26
Lines 3547 3547
=======================================
Hits 2944 2944
Misses 603 603
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=footer). Last update [1a5b7d3...fb64dda](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644610729,"Update pytest-asyncio requirement from <0.13,>=0.10 to >=0.10,<0.15",
https://github.com/simonw/datasette/issues/838#issuecomment-648800356,https://api.github.com/repos/simonw/datasette/issues/838,648800356,MDEyOklzc3VlQ29tbWVudDY0ODgwMDM1Ng==,6739646,tballison,2020-06-24T12:51:48Z,2020-06-24T12:51:48Z,NONE,">But also want to say thanks for a great tool
+1!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/865#issuecomment-648799963,https://api.github.com/repos/simonw/datasette/issues/865,648799963,MDEyOklzc3VlQ29tbWVudDY0ODc5OTk2Mw==,6739646,tballison,2020-06-24T12:51:01Z,2020-06-24T12:51:01Z,NONE,This seems to be a duplicate of: https://github.com/simonw/datasette/issues/838,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/859#issuecomment-648669523,https://api.github.com/repos/simonw/datasette/issues/859,648669523,MDEyOklzc3VlQ29tbWVudDY0ODY2OTUyMw==,3243482,abdusco,2020-06-24T08:13:23Z,2020-06-24T10:30:36Z,CONTRIBUTOR,"I tried setting `cache_size_kb=0` then `cache_size_kb=100000`, still getting this behavior. I even changed `Database::table_counts` and lowered time limit to 1
```py
table_count = (
await self.execute(
""select count(*) from [{}]"".format(table),
custom_time_limit=1,
)
).rows[0][0]
counts[table] = table_count
```
I feel like 10 seconds is a magic number, like a processing timeout and datasette gives up and returns the page.
Index page loads instantly, table page, query page, as well. But when I return to database page after some time, it loads in 10s.
EDIT:
It's always like 10 + 0.3s, like 10s wait and timeout then 300ms to render the page","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/864#issuecomment-648580556,https://api.github.com/repos/simonw/datasette/issues/864,648580556,MDEyOklzc3VlQ29tbWVudDY0ODU4MDU1Ng==,9599,simonw,2020-06-24T04:40:49Z,2020-06-24T04:40:49Z,OWNER,The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-648580236,https://api.github.com/repos/simonw/datasette/issues/864,648580236,MDEyOklzc3VlQ29tbWVudDY0ODU4MDIzNg==,9599,simonw,2020-06-24T04:39:39Z,2020-06-24T04:39:39Z,OWNER,"Urgh, fixing this is going to be a bit of a pain.
Here's where I added that custom `dispatch_request()` method - it was to implement flash messaging in #790: https://github.com/simonw/datasette/blame/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/views/base.py#L85
If I want this to be made available to `register_routes()` views as well, I'm going to have to move the logic somewhere else. In particular I need to make sure that the `request` object is created once and used throughout the whole request cycle.
Currently `register_routes()` view functions get their own separate request object which is created here:
https://github.com/simonw/datasette/blob/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/app.py#L1057-L1068
So I'm going to have to refactor this quite a bit to get that shared request object which can be passed both to `register_routes` views and to my various `BaseView` subclasses.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648442511,https://api.github.com/repos/simonw/sqlite-utils/issues/117,648442511,MDEyOklzc3VlQ29tbWVudDY0ODQ0MjUxMQ==,9599,simonw,2020-06-23T21:39:41Z,2020-06-23T21:39:41Z,OWNER,"So there are two sides to supporting this:
- Being able to sensibly introspect composite foreign keys
- Being able to define composite foreign keys when creating a table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644161221,Support for compound (composite) foreign keys,
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440634,https://api.github.com/repos/simonw/sqlite-utils/issues/117,648440634,MDEyOklzc3VlQ29tbWVudDY0ODQ0MDYzNA==,9599,simonw,2020-06-23T21:35:16Z,2020-06-23T21:35:16Z,OWNER,Relevant discussion: https://github.com/simonw/sqlite-generate/issues/8#issuecomment-648438056,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644161221,Support for compound (composite) foreign keys,