html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1708#issuecomment-1095687566,https://api.github.com/repos/simonw/datasette/issues/1708,1095687566,IC_kwDOBm6k_c5BTt2O,9599,2022-04-11T23:24:30Z,2022-04-11T23:24:30Z,OWNER,"## Redesigned template context

**Warning:** if you use any custom templates with your Datasette instance they are likely to break when you upgrade to 1.0.

The template context has been redesigned to be based on the documented JSON API. This means that the template context can be considered stable going forward, so any custom templates you implement should continue to work when you upgrade Datasette in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200649124,
https://github.com/simonw/datasette/issues/1705#issuecomment-1095673947,https://api.github.com/repos/simonw/datasette/issues/1705,1095673947,IC_kwDOBm6k_c5BTqhb,9599,2022-04-11T23:03:49Z,2022-04-11T23:03:49Z,OWNER,I'll also encourage testing against both Datasette 0.x and Datasette 1.0 using a GitHub Actions matrix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1197926598,
https://github.com/simonw/datasette/issues/1710#issuecomment-1095673670,https://api.github.com/repos/simonw/datasette/issues/1710,1095673670,IC_kwDOBm6k_c5BTqdG,9599,2022-04-11T23:03:25Z,2022-04-11T23:03:25Z,OWNER,"Dupe of:
- #1705","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200649889,
https://github.com/simonw/datasette/issues/1709#issuecomment-1095671940,https://api.github.com/repos/simonw/datasette/issues/1709,1095671940,IC_kwDOBm6k_c5BTqCE,9599,2022-04-11T23:00:39Z,2022-04-11T23:01:41Z,OWNER,"- #262
- #782 
- #1509","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200649502,
https://github.com/simonw/datasette/issues/1711#issuecomment-1095672127,https://api.github.com/repos/simonw/datasette/issues/1711,1095672127,IC_kwDOBm6k_c5BTqE_,9599,2022-04-11T23:00:58Z,2022-04-11T23:00:58Z,OWNER,- #1510,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200650491,
https://github.com/simonw/datasette/issues/1707#issuecomment-1095277937,https://api.github.com/repos/simonw/datasette/issues/1707,1095277937,IC_kwDOBm6k_c5BSJ1x,9599,2022-04-11T16:32:31Z,2022-04-11T16:33:00Z,OWNER,"That's a really interesting idea!

That page is one of the least developed at the moment. There's plenty of room for it to grow new useful features.

I like this suggestion because it feels like a good opportunity to introduce some unobtrusive JavaScript. Could use a details/summary element that uses `fetch()` to load in the extra data for example.

Could even do something with the `<datasette-table>` Web Component here... https://github.com/simonw/datasette-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200224939,
https://github.com/simonw/datasette/issues/1699#issuecomment-1094453751,https://api.github.com/repos/simonw/datasette/issues/1699,1094453751,IC_kwDOBm6k_c5BPAn3,25778,2022-04-11T01:32:12Z,2022-04-11T01:32:12Z,CONTRIBUTOR,"Was looking through old issues and realized a bunch of this got discussed in #1101 (including by me!), so sorry to rehash all this. Happy to help with whatever piece of it I can. Would be very excited to be able to use format plugins with exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,
https://github.com/simonw/datasette/issues/1706#issuecomment-1094152642,https://api.github.com/repos/simonw/datasette/issues/1706,1094152642,IC_kwDOBm6k_c5BN3HC,9599,2022-04-10T01:11:54Z,2022-04-10T01:11:54Z,OWNER,"This relates to this much larger vision:
- #417 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1198822563,
https://github.com/simonw/datasette/issues/1706#issuecomment-1094152173,https://api.github.com/repos/simonw/datasette/issues/1706,1094152173,IC_kwDOBm6k_c5BN2_t,9599,2022-04-10T01:08:50Z,2022-04-10T01:08:50Z,OWNER,This is a good idea - it matches the way `datasette .` works for mutable database files already.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1198822563,
https://github.com/simonw/datasette/pull/1693#issuecomment-1093454899,https://api.github.com/repos/simonw/datasette/issues/1693,1093454899,IC_kwDOBm6k_c5BLMwz,9599,2022-04-08T23:07:04Z,2022-04-08T23:07:04Z,OWNER,"Tests failed here due to this issue:
- https://github.com/psf/black/pull/2987

A future Black release should fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1184850337,
https://github.com/simonw/datasette/pull/1703#issuecomment-1092850719,https://api.github.com/repos/simonw/datasette/issues/1703,1092850719,IC_kwDOBm6k_c5BI5Qf,22429695,2022-04-08T13:18:04Z,2022-04-08T13:18:04Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1703](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (73aabe6) into [main](https://codecov.io/gh/simonw/datasette/commit/90d1be9952db9aaddc21a536e4d00a8de44765d7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (90d1be9) will **not change** coverage.
> The diff coverage is `n/a`.

```diff
@@           Coverage Diff           @@
##             main    #1703   +/-   ##
=======================================
  Coverage   91.75%   91.75%           
=======================================
  Files          34       34           
  Lines        4573     4573           
=======================================
  Hits         4196     4196           
  Misses        377      377           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [90d1be9...73aabe6](https://codecov.io/gh/simonw/datasette/pull/1703?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1197298420,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092386254,https://api.github.com/repos/simonw/datasette/issues/1699,1092386254,IC_kwDOBm6k_c5BHH3O,25778,2022-04-08T02:39:25Z,2022-04-08T02:39:25Z,CONTRIBUTOR,"And just to think this through a little more, here's what `stream_geojson` might look like:

```python
async def stream_geojson(datasette, columns, rows, database, stream):
    db = datasette.get_database(database)
    for row in rows:
        feature = await row_to_geojson(row, db)
        stream.write(feature + ""\n"") # just assuming newline mode for now
```

Alternately, that could be an async generator, like this:

```python
async def stream_geojson(datasette, columns, rows, database):
    db = datasette.get_database(database)
    for row in rows:
        feature = await row_to_geojson(row, db)
        yield feature
```

Not sure which makes more sense, but I think this pattern would open up a lot of possibility. If you had your [stream_indented_json](https://til.simonwillison.net/python/output-json-array-streaming) function, you could do `yield from stream_indented_json(rows, 2)` and be one your way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092370880,https://api.github.com/repos/simonw/datasette/issues/1699,1092370880,IC_kwDOBm6k_c5BHEHA,25778,2022-04-08T02:07:40Z,2022-04-08T02:07:40Z,CONTRIBUTOR,"So maybe `render_output_render` returns something like this:

```python
@hookimpl
def register_output_renderer(datasette):
    return {
        ""extension"": ""geojson"",
        ""render"": render_geojson,
        ""stream"": stream_geojson,
        ""can_render"": can_render_geojson,
    }
```

And stream gets an iterator, instead of a list of rows, so it can efficiently handle large queries. Maybe it also gets passed a destination stream, or it returns an iterator. I'm not sure what makes more sense. Either way, that might cover both CLI exports and streaming responses.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092361727,https://api.github.com/repos/simonw/datasette/issues/1699,1092361727,IC_kwDOBm6k_c5BHB3_,9599,2022-04-08T01:47:43Z,2022-04-08T01:47:43Z,OWNER,"A render mode for that plugin hook that writes to a stream is exactly what I have in mind:
- #1062 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092357672,https://api.github.com/repos/simonw/datasette/issues/1699,1092357672,IC_kwDOBm6k_c5BHA4o,25778,2022-04-08T01:39:40Z,2022-04-08T01:39:40Z,CONTRIBUTOR,"> My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like .geojson and .yml and suchlike - also work for the datasette query command that would make a lot of sense to me.

That's my thinking, too. It's really the thing I've been wanting since writing `datasette-geojson`, since I'm always exporting with `datasette --get`. The workflow I'm always looking for is something like this:

```sh
cd alltheplaces-datasette
datasette query dunkin_in_suffolk -f geojson -o dunkin_in_suffolk.geojson
```

I think this probably needs either a new plugin hook separate from `register_output_renderer` or a way to use that without going through the HTTP stack. Or maybe a render mode that writes to a stream instead of a response. Maybe there's a new key in the dictionary that `register_output_renderer` returns that handles CLI exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,
https://github.com/simonw/datasette/issues/1699#issuecomment-1092321966,https://api.github.com/repos/simonw/datasette/issues/1699,1092321966,IC_kwDOBm6k_c5BG4Ku,9599,2022-04-08T00:20:32Z,2022-04-08T00:20:56Z,OWNER,"If we do this I'm keen to have it be more than just an alternative to the existing `sqlite-utils` command - especially since if I add `sqlite-utils` as a dependency of Datasette in the future that command will be installed as part of `pip install datasette` anyway.

My best thought on how to differentiate them so far is plugins: if Datasette plugins that provide alternative outputs - like `.geojson` and `.yml` and suchlike - also work for the `datasette query` command that would make a lot of sense to me.

One way that could work: a `--fmt geojson` option to this command which uses the plugin that was registered for the specified extension.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1193090967,
https://github.com/simonw/datasette/issues/1549#issuecomment-1087428593,https://api.github.com/repos/simonw/datasette/issues/1549,1087428593,IC_kwDOBm6k_c5A0Nfx,536941,2022-04-04T11:17:13Z,2022-04-04T11:17:13Z,CONTRIBUTOR,"another way to get the behavior of downloading the file is to use the download attribute of the anchor tag

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-download","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077620955,
https://github.com/simonw/datasette/issues/1698#issuecomment-1086784547,https://api.github.com/repos/simonw/datasette/issues/1698,1086784547,IC_kwDOBm6k_c5AxwQj,9599,2022-04-03T06:10:24Z,2022-04-03T06:10:24Z,OWNER,Warning added here: https://docs.datasette.io/en/latest/publish.html#publishing-to-google-cloud-run,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1190828163,
https://github.com/simonw/datasette/issues/1697#issuecomment-1085323192,https://api.github.com/repos/simonw/datasette/issues/1697,1085323192,IC_kwDOBm6k_c5AsLe4,9599,2022-04-01T02:01:51Z,2022-04-01T02:01:51Z,OWNER,"Huh, turns out `Request.fake()` wasn't yet documented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1189113609,
https://github.com/simonw/datasette/pull/1574#issuecomment-1084216224,https://api.github.com/repos/simonw/datasette/issues/1574,1084216224,IC_kwDOBm6k_c5An9Og,33631,2022-03-31T07:45:25Z,2022-03-31T07:45:25Z,NONE,"@simonw I like that you want to go ""slim by default"". Do you want another PR for that or should I just wait?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,
https://github.com/simonw/datasette/issues/1696#issuecomment-1083351437,https://api.github.com/repos/simonw/datasette/issues/1696,1083351437,IC_kwDOBm6k_c5AkqGN,9599,2022-03-30T16:20:49Z,2022-03-30T16:21:02Z,OWNER,"Maybe like this:

<img width=""499"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/160883280-c19d5a22-e923-491f-8bf4-1a4f5215d684.png"">

```html
<h3>283 rows
  where dcode = 3 <span style=""color: #aaa; font-size: 0.9em"">(Human Related: Other)</span>
    </h3>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1186696202,
https://github.com/simonw/datasette/issues/1692#issuecomment-1082663746,https://api.github.com/repos/simonw/datasette/issues/1692,1082663746,IC_kwDOBm6k_c5AiCNC,9599,2022-03-30T06:14:39Z,2022-03-30T06:14:51Z,OWNER,"I like your design, though I think it should be `""nomodule"": True` for consistency with the other options.

I think `""async"": True` is worth supporting too.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182227211,
https://github.com/simonw/datasette/issues/1692#issuecomment-1082661795,https://api.github.com/repos/simonw/datasette/issues/1692,1082661795,IC_kwDOBm6k_c5AiBuj,9599,2022-03-30T06:11:41Z,2022-03-30T06:11:41Z,OWNER,This is a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182227211,
https://github.com/simonw/datasette/issues/1695#issuecomment-1082617386,https://api.github.com/repos/simonw/datasette/issues/1695,1082617386,IC_kwDOBm6k_c5Ah24q,9599,2022-03-30T04:46:18Z,2022-03-30T04:46:18Z,OWNER,"` selected = (column_qs, str(row[""value""])) in qs_pairs` is wrong.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1185868354,
https://github.com/simonw/datasette/issues/1695#issuecomment-1082617241,https://api.github.com/repos/simonw/datasette/issues/1695,1082617241,IC_kwDOBm6k_c5Ah22Z,9599,2022-03-30T04:45:55Z,2022-03-30T04:45:55Z,OWNER,"Relevant template: https://github.com/simonw/datasette/blob/e73fa72917ca28c152208d62d07a490c81cadf52/datasette/templates/table.html#L168-L172

Populated from here: https://github.com/simonw/datasette/blob/c496f2b663ff0cef908ffaaa68b8cb63111fb5f2/datasette/facets.py#L246-L253","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1185868354,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1082476727,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1082476727,IC_kwDOCGYnMM5AhUi3,770231,2022-03-29T23:52:38Z,2022-03-29T23:52:38Z,NONE,"@simonw Thanks for looking into it and documenting the solution!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/datasette/pull/1694#issuecomment-1081860312,https://api.github.com/repos/simonw/datasette/issues/1694,1081860312,IC_kwDOBm6k_c5Ae-DY,22429695,2022-03-29T13:17:30Z,2022-03-29T13:17:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1694](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (83ff967) into [main](https://codecov.io/gh/simonw/datasette/commit/e73fa72917ca28c152208d62d07a490c81cadf52?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e73fa72) will **not change** coverage.
> The diff coverage is `n/a`.

```diff
@@           Coverage Diff           @@
##             main    #1694   +/-   ##
=======================================
  Coverage   91.74%   91.74%           
=======================================
  Files          34       34           
  Lines        4565     4565           
=======================================
  Hits         4188     4188           
  Misses        377      377           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e73fa72...83ff967](https://codecov.io/gh/simonw/datasette/pull/1694?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1184850675,
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1081079506,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1081079506,IC_kwDOCGYnMM5Ab_bS,24938923,2022-03-28T19:58:55Z,2022-03-28T20:05:57Z,NONE,"Sure, it is from the documentation example:
[Extracting columns into a separate table](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table)


```
wget ""https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true""

sqlite-utils insert global.db power_plants \
    'global_power_plant_database.csv?raw=true' --csv
# Extract those columns:
sqlite-utils extract global.db power_plants country country_long \
    --table countries \
    --fk-column country_id \
    --rename country_long name
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1081047053,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1081047053,IC_kwDOCGYnMM5Ab3gN,9599,2022-03-28T19:22:37Z,2022-03-28T19:22:37Z,OWNER,Wrote about this in my weeknotes: https://simonwillison.net/2022/Mar/28/datasette-auth0/#new-features-as-documentation,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1080141111,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1080141111,IC_kwDOCGYnMM5AYaU3,9599,2022-03-28T03:25:57Z,2022-03-28T03:54:37Z,OWNER,"So now this should solve your problem:
```
echo '[{""name"": ""notaword""}, {""name"": ""word""}]
' | python3 -m sqlite_utils insert listings.db listings - --convert '
import enchant
d = enchant.Dict(""en_US"")

def convert(row):
    global d
    row[""is_dictionary_word""] = d.check(row[""name""])
'
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/datasette/issues/1688#issuecomment-1079806857,https://api.github.com/repos/simonw/datasette/issues/1688,1079806857,IC_kwDOBm6k_c5AXIuJ,9020979,2022-03-27T01:01:14Z,2022-03-27T01:01:14Z,CONTRIBUTOR,"Thank you! I went through the cookiecutter template, and published my first package here: https://github.com/hydrosquall/datasette-nteract-data-explorer","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181432624,
https://github.com/simonw/datasette/issues/1690#issuecomment-1079788375,https://api.github.com/repos/simonw/datasette/issues/1690,1079788375,IC_kwDOBm6k_c5AXENX,9599,2022-03-26T22:43:00Z,2022-03-26T22:43:00Z,OWNER,Then I can update this section of the documentation which currently recommends the above pattern: https://docs.datasette.io/en/stable/authentication.html#the-ds-actor-cookie,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182141761,
https://github.com/simonw/datasette/issues/1690#issuecomment-1079788346,https://api.github.com/repos/simonw/datasette/issues/1690,1079788346,IC_kwDOBm6k_c5AXEM6,9599,2022-03-26T22:42:40Z,2022-03-26T22:42:40Z,OWNER,"I don't want to do a `response.set_actor_cookie()` method because I like `Response` not to carry too many Datasette-specific features.

So `datasette.set_actor_cookie(response, actor, expire_after=None)` would be a better place for this I think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182141761,
https://github.com/simonw/datasette/issues/1689#issuecomment-1079779040,https://api.github.com/repos/simonw/datasette/issues/1689,1079779040,IC_kwDOBm6k_c5AXB7g,9599,2022-03-26T21:35:57Z,2022-03-26T21:35:57Z,OWNER,Fixed: https://docs.datasette.io/en/latest/internals.html#add-message-request-message-type-datasette-info,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1182065616,
https://github.com/simonw/datasette/issues/1688#issuecomment-1079550754,https://api.github.com/repos/simonw/datasette/issues/1688,1079550754,IC_kwDOBm6k_c5AWKMi,9020979,2022-03-26T01:27:27Z,2022-03-26T03:16:29Z,CONTRIBUTOR,"> Is there a way to serve a static assets when using the plugins/ directory method instead of installing plugins as a new python package?

As a workaround, I found I can serve my statics from a non-plugin specific folder using the [--static](https://docs.datasette.io/en/stable/custom_templates.html#serving-static-files) CLI flag.

```bash
datasette ~/Library/Safari/History.db \
  --plugins-dir=plugins/ \
  --static assets:dist/
```

It's not ideal because it means I'll change the cache pattern path depending on how the plugin is running (via pip install or as a one off script), but it's usable as a workaround.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181432624,
https://github.com/simonw/datasette/issues/1688#issuecomment-1079582485,https://api.github.com/repos/simonw/datasette/issues/1688,1079582485,IC_kwDOBm6k_c5AWR8V,9599,2022-03-26T03:15:34Z,2022-03-26T03:15:34Z,OWNER,"Yup, you're right in what you figured out here: stand-alone plugins can't currently package static assets other then using the static folder.

The `datasette-plugin` cookiecutter template should make creating a Python package pretty easy though: https://github.com/simonw/datasette-plugin

You can run that yourself, or you can run it using this GitHub template repository: https://github.com/simonw/datasette-plugin-template-repository 

","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181432624,
https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1079441621,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1079441621,IC_kwDOCGYnMM5AVvjV,9599,2022-03-25T21:18:37Z,2022-03-25T21:18:37Z,OWNER,Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-newline-delimited-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1079407962,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1079407962,IC_kwDOCGYnMM5AVnVa,9599,2022-03-25T20:25:10Z,2022-03-25T20:25:18Z,OWNER,"Can you share either your whole `global.db` table or a shrunk down example that illustrates the bug?

My hunch is that you may have a table or column with a name that triggers the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,
https://github.com/simonw/sqlite-utils/issues/422#issuecomment-1079406708,https://api.github.com/repos/simonw/sqlite-utils/issues/422,1079406708,IC_kwDOCGYnMM5AVnB0,9599,2022-03-25T20:23:21Z,2022-03-25T20:23:21Z,OWNER,"Fixing this would require a bump to 4.0 because it would break existing code.

The alternative would be to introduce a new `ignore_nulls=True` parameter which users can change to `ignore_nulls=False`. Or come up with better wording for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1181236173,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079404281,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079404281,IC_kwDOCGYnMM5AVmb5,9599,2022-03-25T20:19:50Z,2022-03-25T20:19:50Z,OWNER,Now documented here: https://sqlite-utils.datasette.io/en/latest/cli.html#using-a-convert-function-to-execute-initialization,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079384771,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079384771,IC_kwDOCGYnMM5AVhrD,9599,2022-03-25T19:51:34Z,2022-03-25T19:53:01Z,OWNER,"This works:
```
% sqlite-utils insert dogs.db dogs dogs.json --convert '
import random
print(""seeding"")
random.seed(10)
print(random.random())

def convert(row):
    global random
    print(row)
    row[""random_score""] = random.random()
'
seeding
0.5714025946899135
{'id': 1, 'name': 'Cleo'}
{'id': 2, 'name': 'Pancakes'}
{'id': 3, 'name': 'New dog'}
(sqlite-utils) sqlite-utils % sqlite-utils rows dogs.db dogs
[{""id"": 1, ""name"": ""Cleo"", ""random_score"": 0.4288890546751146},
 {""id"": 2, ""name"": ""Pancakes"", ""random_score"": 0.5780913011344704},
 {""id"": 3, ""name"": ""New dog"", ""random_score"": 0.20609823213950174}]
```
Having to use `global random` inside the function is frustrating but apparently necessary. https://stackoverflow.com/a/56552138/6083","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079376283,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079376283,IC_kwDOCGYnMM5AVfmb,9599,2022-03-25T19:39:30Z,2022-03-25T19:43:35Z,OWNER,"Actually this doesn't work as I thought. This demo shows that the initialization code is run once per item, not a single time at the start of the run:
```
% sqlite-utils insert dogs.db dogs dogs.json --convert '
import random
print(""seeding"")
random.seed(10)
print(random.random())

def convert(row):
    print(row)
    row[""random_score""] = random.random()
'
seeding
0.5714025946899135
seeding
0.5714025946899135
seeding
0.5714025946899135
seeding
0.5714025946899135
```
Also that `print(row)` line is not being printed anywhere that gets to the console for some reason.

... my mistake, that happened because I changed this line in order to try to get local imports to work:
```python
    try:
        exec(code, globals, locals)
        return globals[""convert""]
    except (AttributeError, SyntaxError, NameError, KeyError, TypeError):
```
It should be `locals[""convert""]`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1079243535,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1079243535,IC_kwDOCGYnMM5AU_MP,9599,2022-03-25T17:25:12Z,2022-03-25T17:25:12Z,OWNER,"That documentation is split across a few places. This is the only bit that talks about `def convert()` pattern right now:

- https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns

But that's for `sqlite-utils convert` - the documentation for `sqlite-utils insert --convert` at https://sqlite-utils.datasette.io/en/stable/cli.html#applying-conversions-while-inserting-data doesn't mention it.

Since both `sqlite-utils convert` and `sqlite-utils insert --convert` apply the same rules to the code, they should link to a shared explanation in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/datasette/pull/1685#issuecomment-1079018557,https://api.github.com/repos/simonw/datasette/issues/1685,1079018557,IC_kwDOBm6k_c5AUIQ9,22429695,2022-03-25T13:16:48Z,2022-03-25T13:16:48Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1685](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (933ce47) into [main](https://codecov.io/gh/simonw/datasette/commit/c496f2b663ff0cef908ffaaa68b8cb63111fb5f2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c496f2b) will **not change** coverage.
> The diff coverage is `n/a`.

```diff
@@           Coverage Diff           @@
##             main    #1685   +/-   ##
=======================================
  Coverage   91.74%   91.74%           
=======================================
  Files          34       34           
  Lines        4565     4565           
=======================================
  Hits         4188     4188           
  Misses        377      377           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c496f2b...933ce47](https://codecov.io/gh/simonw/datasette/pull/1685?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180778860,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078343231,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078343231,IC_kwDOCGYnMM5ARjY_,9599,2022-03-24T21:16:10Z,2022-03-24T21:17:20Z,OWNER,"Aha! This may be possible already: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L311-L316

And yes, this does indeed work - you can do something like this:

```
echo '{""name"": ""harry""}' | sqlite-utils insert db.db people - --convert '
import time
# Simulate something expensive
time.sleep(1)

def convert(row):
    row[""upper""] = row[""name""].upper()
'
```
And after running that:
```
sqlite-utils dump db.db 
BEGIN TRANSACTION;
CREATE TABLE [people] (
   [name] TEXT,
   [upper] TEXT
);
INSERT INTO ""people"" VALUES('harry','HARRY');
COMMIT;
```
So this is a documentation issue - there's a trick for it but I didn't know what the trick was!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078328774,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078328774,IC_kwDOCGYnMM5ARf3G,9599,2022-03-24T21:12:33Z,2022-03-24T21:12:33Z,OWNER,"Here's how the `_compile_code()` mechanism works at the moment: https://github.com/simonw/sqlite-utils/blob/396f80fcc60da8dd844577114f7920830a2e5403/sqlite_utils/utils.py#L308-L342

At the end it does this:
```python
    return locals[""fn""]
```
So it's already building and then returning a function.

The question is if there's a sensible way to allow people to further customize that function by executing some code first, in a way that's easy to explain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078322301,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078322301,IC_kwDOCGYnMM5AReR9,9599,2022-03-24T21:10:52Z,2022-03-24T21:10:52Z,OWNER,"I can think of three ways forward:

- Figure out a pattern that gets that local file import workaround to work
- Add another option such as `--convert-init` that lets you pass code that will be executed once at the start
- Come up with a pattern where the `--convert` code can run some initialization code and then return a function which will be called against each value

I quite like the idea of that third option - I'm going to prototype it and see if I can work something out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/sqlite-utils/issues/420#issuecomment-1078315922,https://api.github.com/repos/simonw/sqlite-utils/issues/420,1078315922,IC_kwDOCGYnMM5ARcuS,9599,2022-03-24T21:09:27Z,2022-03-24T21:09:27Z,OWNER,"Yeah, this is WAY harder than it should be.

There's a clumsy workaround you could use which looks something like this: create a file `my_enchant.py` containing:

```python
import enchant
d = enchant.Dict(""en_US"")

def check(word):
    return d.check(word)
```
Then run `sqlite-utils` like this:

```
PYTHONPATH=. cat items.json | jq '.data' | sqlite-utils insert listings.db listings - --convert 'my_enchant.check(value)' --import my_enchant
```
Except I tried that and it doesn't work! I don't know the right pattern for getting `--import` to work with modules in the same directory.

So yeah, this is definitely a big feature gap.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178546862,
https://github.com/simonw/datasette/issues/1684#issuecomment-1078126065,https://api.github.com/repos/simonw/datasette/issues/1684,1078126065,IC_kwDOBm6k_c5AQuXx,536941,2022-03-24T20:08:56Z,2022-03-24T20:13:19Z,CONTRIBUTOR,"would be nice if the behavior was

1. try to facet all the columns
2. for bigger tables try to facet the indexed columns
3. for the biggest tables, turn off autofacetting completely

This is based on my assumption that what determines autofaceting is the rarity of unique values. Which may not be true!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1179998071,
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1077671779,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1077671779,IC_kwDOCGYnMM5AO_dj,25778,2022-03-24T14:11:33Z,2022-03-24T14:11:43Z,CONTRIBUTOR,"Coming back to this. I was about to add a utility function to [datasette-geojson]() to convert lat/lng columns to geometries. Thankfully I googled first. There's a SpatiaLite function for this: [MakePoint](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html#p0).

```sql
select MakePoint(longitude, latitude) as geometry from places;
```

I'm not sure if that would work with `conversions`, since it needs two columns, but it's an option for tables that already have latitude, longitude columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,
https://github.com/simonw/datasette/issues/1581#issuecomment-1077047295,https://api.github.com/repos/simonw/datasette/issues/1581,1077047295,IC_kwDOBm6k_c5AMm__,536941,2022-03-24T04:08:18Z,2022-03-24T04:08:18Z,CONTRIBUTOR,this has been addressed by the datasette-hashed-urls plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1089529555,
https://github.com/simonw/datasette/pull/1582#issuecomment-1077047152,https://api.github.com/repos/simonw/datasette/issues/1582,1077047152,IC_kwDOBm6k_c5AMm9w,536941,2022-03-24T04:07:58Z,2022-03-24T04:07:58Z,CONTRIBUTOR,this has been obviated by the datasette-hashed-urls plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1090055810,
https://github.com/simonw/datasette/issues/1682#issuecomment-1076696791,https://api.github.com/repos/simonw/datasette/issues/1682,1076696791,IC_kwDOBm6k_c5ALRbX,9599,2022-03-23T18:45:49Z,2022-03-23T18:45:49Z,OWNER,"The problem is here in `QueryView`: https://github.com/simonw/datasette/blob/d7c793d7998388d915f8d270079c68a77a785051/datasette/views/database.py#L206-L238

It should be resolving `database` based on the route path, as seen in other methods like this one: https://github.com/simonw/datasette/blob/d7c793d7998388d915f8d270079c68a77a785051/datasette/views/table.py#L270-L279
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178521513,
https://github.com/simonw/datasette/issues/1670#issuecomment-1076683297,https://api.github.com/repos/simonw/datasette/issues/1670,1076683297,IC_kwDOBm6k_c5ALOIh,9599,2022-03-23T18:32:32Z,2022-03-23T18:32:32Z,OWNER,Added this to news on https://datasette.io/ https://github.com/simonw/datasette.io/commit/fd3ec57cdd5b935f75cbf52a86b3aabf2c97d217,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,
https://github.com/simonw/datasette/issues/1670#issuecomment-1076666293,https://api.github.com/repos/simonw/datasette/issues/1670,1076666293,IC_kwDOBm6k_c5ALJ-1,9599,2022-03-23T18:16:29Z,2022-03-23T18:16:29Z,OWNER,https://docs.datasette.io/en/stable/changelog.html#v0-61,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,
https://github.com/simonw/datasette/issues/1670#issuecomment-1076665837,https://api.github.com/repos/simonw/datasette/issues/1670,1076665837,IC_kwDOBm6k_c5ALJ3t,9599,2022-03-23T18:16:01Z,2022-03-23T18:16:01Z,OWNER,"https://github.com/simonw/datasette/releases/tag/0.61

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,
https://github.com/simonw/sqlite-utils/pull/419#issuecomment-1076662556,https://api.github.com/repos/simonw/sqlite-utils/issues/419,1076662556,IC_kwDOCGYnMM5ALJEc,22429695,2022-03-23T18:12:47Z,2022-03-23T18:12:47Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#419](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (228f736) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/93fa79d30b1531bea281d0eb6b925c4e61bc1aa6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (93fa79d) will **not change** coverage.
> The diff coverage is `n/a`.

```diff
@@           Coverage Diff           @@
##             main     #419   +/-   ##
=======================================
  Coverage   96.55%   96.55%           
=======================================
  Files           6        6           
  Lines        2498     2498           
=======================================
  Hits         2412     2412           
  Misses         86       86           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [93fa79d...228f736](https://codecov.io/gh/simonw/sqlite-utils/pull/419?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1178484369,
https://github.com/simonw/datasette/issues/1670#issuecomment-1076652046,https://api.github.com/repos/simonw/datasette/issues/1670,1076652046,IC_kwDOBm6k_c5ALGgO,9599,2022-03-23T18:02:30Z,2022-03-23T18:02:30Z,OWNER,"Two new things to add to the release notes from https://github.com/simonw/datasette/compare/0.61a0...main
- https://github.com/simonw/datasette/issues/1678
- https://github.com/simonw/datasette/issues/1675 (now also a documented API)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,
https://github.com/simonw/datasette/issues/1670#issuecomment-1076647495,https://api.github.com/repos/simonw/datasette/issues/1670,1076647495,IC_kwDOBm6k_c5ALFZH,9599,2022-03-23T17:58:16Z,2022-03-23T17:58:16Z,OWNER,"I think the release notes are fine, but they need an opening paragraph highlighting the changes that are most likely to break backwards compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,
https://github.com/simonw/datasette/pull/1574#issuecomment-1076645636,https://api.github.com/repos/simonw/datasette/issues/1574,1076645636,IC_kwDOBm6k_c5ALE8E,9599,2022-03-23T17:56:35Z,2022-03-23T17:56:35Z,OWNER,I'd actually like to switch to slim as the default - I think Datasette should ship the smallest possible container that can still support extra packages being installed using `apt-get install`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1084193403,
https://github.com/simonw/datasette/pull/1665#issuecomment-1076644362,https://api.github.com/repos/simonw/datasette/issues/1665,1076644362,IC_kwDOBm6k_c5ALEoK,9599,2022-03-23T17:55:39Z,2022-03-23T17:55:39Z,OWNER,Thanks for the PR - I spotted an error about this and went through and fixed this in all of my repos the other day: https://github.com/search?o=desc&q=user%3Asimonw+google-github-actions%2Fsetup-gcloud%40v0&s=committer-date&type=Commits,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1173828092,
https://github.com/simonw/datasette/issues/1670#issuecomment-1076638278,https://api.github.com/repos/simonw/datasette/issues/1670,1076638278,IC_kwDOBm6k_c5ALDJG,9599,2022-03-23T17:50:55Z,2022-03-23T17:50:55Z,OWNER,"Release notes are mostly written for the alpha, just need to clean them up a bit https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/docs/changelog.rst#061a0-2022-03-19","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174423568,
https://github.com/simonw/datasette/issues/1681#issuecomment-1075438684,https://api.github.com/repos/simonw/datasette/issues/1681,1075438684,IC_kwDOBm6k_c5AGeRc,9599,2022-03-22T17:45:50Z,2022-03-22T17:49:09Z,OWNER,"I would expect this to break against SQL views that include calculated columns though - something like this:

```sql
create view this_will_break as select pk + 1 as pk_plus_one, 0.5 as score from searchable;
```
Confirmed: the filter interface for that view plain doesn't work for any comparison against that table - except for `score > 0` since `0` is converted to an integer. `0.1` breaks though because it doesn't get converted as it doesn't match `.isdigit()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1177101697,
https://github.com/simonw/datasette/issues/1681#issuecomment-1075437598,https://api.github.com/repos/simonw/datasette/issues/1681,1075437598,IC_kwDOBm6k_c5AGeAe,9599,2022-03-22T17:44:42Z,2022-03-22T17:45:04Z,OWNER,"My hunch is that this mechanism doesn't actually do anything useful at all, because of the type conversion that automatically happens for data from tables based on the column type affinities, see:
- #1671

So either remove the `self.numeric` type conversion bit entirely, or prove that it is necessary and upgrade it to be able to handle floating point values too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1177101697,
https://github.com/simonw/datasette/issues/1671#issuecomment-1075432283,https://api.github.com/repos/simonw/datasette/issues/1671,1075432283,IC_kwDOBm6k_c5AGctb,9599,2022-03-22T17:39:04Z,2022-03-22T17:43:12Z,OWNER,"Note that Datasette does already have special logic to convert parameters to integers for numeric comparisons like `>`:

https://github.com/simonw/datasette/blob/c4c9dbd0386e46d2bf199f0ed34e4895c98cb78c/datasette/filters.py#L203-L212

Though... it looks like there's a bug in that? It doesn't account for `float` values - `""3.5"".isdigit()` return `False` - probably for the best, because `int(3.5)` would break that value anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1075435185,https://api.github.com/repos/simonw/datasette/issues/1671,1075435185,IC_kwDOBm6k_c5AGdax,9599,2022-03-22T17:42:09Z,2022-03-22T17:42:09Z,OWNER,"Also made me realize that this query:
```sql
select * from sortable where sortable > :p0
```
Only works here thanks to the column affinity thing kicking in too: https://latest.datasette.io/fixtures?sql=select+*+from+sortable+where+sortable+%3E+%3Ap0&p0=70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1075428030,https://api.github.com/repos/simonw/datasette/issues/1671,1075428030,IC_kwDOBm6k_c5AGbq-,9599,2022-03-22T17:34:30Z,2022-03-22T17:34:30Z,OWNER,"No, I think I need to use `cast` - I can't think of any way to ask SQLite ""for this query, what types are the columns that will come back from it?""

Even the details from the `explain` trick explored in #1293 don't seem to come back with column type information: https://latest.datasette.io/fixtures?sql=explain+select+pk%2C+text1%2C+text2%2C+[name+with+.+and+spaces]+from+searchable_view+where+%22pk%22+%3D+%3Ap0&p0=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1075425513,https://api.github.com/repos/simonw/datasette/issues/1671,1075425513,IC_kwDOBm6k_c5AGbDp,9599,2022-03-22T17:31:53Z,2022-03-22T17:31:53Z,OWNER,"The alternative to using `cast` here would be for Datasette to convert the `""1""` to a `1` in Python code before passing it as a param.

This feels a bit neater to me, but I still then need to solve the problem of how to identify the ""type"" of a column that I want to use in a query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/339#issuecomment-1074479932,https://api.github.com/repos/simonw/datasette/issues/339,1074479932,IC_kwDOBm6k_c5AC0M8,9599,2022-03-21T22:22:34Z,2022-03-21T22:22:34Z,OWNER,Closing this as obsolete since Datasette no longer uses Sanic.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",340396247,
https://github.com/simonw/datasette/issues/276#issuecomment-1074479768,https://api.github.com/repos/simonw/datasette/issues/276,1074479768,IC_kwDOBm6k_c5AC0KY,9599,2022-03-21T22:22:20Z,2022-03-21T22:22:20Z,OWNER,"I'm closing this issue because this is now solved by a number of neat plugins:

- https://datasette.io/plugins/datasette-geojson-map shows the geometry from SpatiaLite columns on a map
- https://datasette.io/plugins/datasette-leaflet-geojson can be used to display inline maps next to each column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,
https://github.com/simonw/datasette/issues/1671#issuecomment-1074478299,https://api.github.com/repos/simonw/datasette/issues/1671,1074478299,IC_kwDOBm6k_c5ACzzb,9599,2022-03-21T22:20:26Z,2022-03-21T22:20:26Z,OWNER,"Thinking about options for fixing this...

The following query works fine:
```sql
select * from test_view where cast(has_expired as text) = '1'
```
I don't want to start using this for every query, because one of the goals of Datasette is to help people who are learning SQL:
- #1613

If someone clicks on ""View and edit SQL"" from a filtered table page I don't want them to have to wonder why that `cast` is there.

But... for querying views, the `cast` turns out to be necessary.

So one fix would be to get the SQL generating logic to use casts like this any time it is operating against a view.

An even better fix would be to detect which columns in a view come from a table and which ones might not, and only use casts for the columns that aren't definitely from a table.

The trick I was exploring here might be able to help with that:
- #1293 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1074470568,https://api.github.com/repos/simonw/datasette/issues/1671,1074470568,IC_kwDOBm6k_c5ACx6o,9599,2022-03-21T22:11:14Z,2022-03-21T22:12:49Z,OWNER,"I wonder if this will be a problem with generated columns, or with SQLite strict tables?

My hunch is that strict tables will continue to work without any changes, because https://www.sqlite.org/stricttables.html says nothing about their impact on comparison operations. I should test this to make absolutely sure though.

Generated columns have a type, so my hunch is they will continue to work fine too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1074468450,https://api.github.com/repos/simonw/datasette/issues/1671,1074468450,IC_kwDOBm6k_c5ACxZi,9599,2022-03-21T22:08:35Z,2022-03-21T22:10:00Z,OWNER,"Relevant section of the SQLite documentation: [3.2. Affinity Of Expressions](https://www.sqlite.org/datatype3.html#affinity_of_expressions):

> When an expression is a simple reference to a column of a real table (not a [VIEW](https://www.sqlite.org/lang_createview.html) or subquery) then the expression has the same affinity as the table column.

In your example, `has_expired` is no longer a simple reference to a column of a real table, hence the bug.

Then [4.2. Type Conversions Prior To Comparison](https://www.sqlite.org/datatype3.html#type_conversions_prior_to_comparison) fills in the rest:

> SQLite may attempt to convert values between the storage classes INTEGER, REAL, and/or TEXT before performing a comparison. Whether or not any conversions are attempted before the comparison takes place depends on the type affinity of the operands. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1671#issuecomment-1074465536,https://api.github.com/repos/simonw/datasette/issues/1671,1074465536,IC_kwDOBm6k_c5ACwsA,9599,2022-03-21T22:04:31Z,2022-03-21T22:04:31Z,OWNER,"Oh this is fascinating! I replicated the bug (thanks for the steps to reproduce) and it looks like this is down to the following:

<img width=""1276"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/159370986-1e2fc513-6d6c-4a2f-96dd-dccf5a680fe4.png"">

Against views, `where has_expired = 1` returns different results from `where has_expired = '1'`

This doesn't happen against tables because of SQLite's [type affinity](https://www.sqlite.org/datatype3.html#type_affinity) mechanism, which handles the type conversion automatically.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1174655187,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074459746,https://api.github.com/repos/simonw/datasette/issues/1679,1074459746,IC_kwDOBm6k_c5ACvRi,9599,2022-03-21T21:55:45Z,2022-03-21T21:55:45Z,OWNER,I'm going to change the original logic to set n=1 for times that are `<= 20ms` - and update the comments to make it more obvious what is happening.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074458506,https://api.github.com/repos/simonw/datasette/issues/1679,1074458506,IC_kwDOBm6k_c5ACu-K,9599,2022-03-21T21:53:47Z,2022-03-21T21:53:47Z,OWNER,"Oh interesting, it turns out there is ONE place in the code that sets the `ms` to less than 20 - this test fixture: https://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/tests/fixtures.py#L224-L226","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074454687,https://api.github.com/repos/simonw/datasette/issues/1679,1074454687,IC_kwDOBm6k_c5ACuCf,9599,2022-03-21T21:48:02Z,2022-03-21T21:48:02Z,OWNER,"Here's another microbenchmark that measures how many nanoseconds it takes to run 1,000 vmops:

```python
import sqlite3
import time

db = sqlite3.connect("":memory:"")

i = 0
out = []

def count():
    global i
    i += 1000
    out.append(((i, time.perf_counter_ns())))

db.set_progress_handler(count, 1000)

print(""Start:"", time.perf_counter_ns())
all = db.execute(""""""
with recursive counter(x) as (
  select 0
    union
  select x + 1 from counter
)
select * from counter limit 10000;
"""""").fetchall()
print(""End:"", time.perf_counter_ns())

print()
print(""So how long does it take to execute 1000 ops?"")

prev_time_ns = None
for i, time_ns in out:
    if prev_time_ns is not None:
        print(time_ns - prev_time_ns, ""ns"")
    prev_time_ns = time_ns
```
Running it:
```
% python nanobench.py
Start: 330877620374821
End: 330877632515822

So how long does it take to execute 1000 ops?
47290 ns
49573 ns
48226 ns
45674 ns
53238 ns
47313 ns
52346 ns
48689 ns
47092 ns
87596 ns
69999 ns
52522 ns
52809 ns
53259 ns
52478 ns
53478 ns
65812 ns
```
87596ns is 0.087596ms - so even a measure rate of every 1000 ops is easily finely grained enough to capture differences of less than 0.1ms.

If anything I could bump that default 1000 up - and I can definitely eliminate the `if ms < 50` branch entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074446576,https://api.github.com/repos/simonw/datasette/issues/1679,1074446576,IC_kwDOBm6k_c5ACsDw,9599,2022-03-21T21:38:27Z,2022-03-21T21:38:27Z,OWNER,"OK here's a microbenchmark script:
```python
import sqlite3
import timeit

db = sqlite3.connect("":memory:"")
db_with_progress_handler_1 = sqlite3.connect("":memory:"")
db_with_progress_handler_1000 = sqlite3.connect("":memory:"")

db_with_progress_handler_1.set_progress_handler(lambda: None, 1)
db_with_progress_handler_1000.set_progress_handler(lambda: None, 1000)

def execute_query(db):
    cursor = db.execute(""""""
    with recursive counter(x) as (
    select 0
        union
    select x + 1 from counter
    )
    select * from counter limit 10000;
    """""")
    list(cursor.fetchall())


print(""Without progress_handler"")
print(timeit.timeit(lambda: execute_query(db), number=100))

print(""progress_handler every 1000 ops"")
print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1000), number=100))

print(""progress_handler every 1 op"")
print(timeit.timeit(lambda: execute_query(db_with_progress_handler_1), number=100))
```
Results:
```
% python3 bench.py
Without progress_handler
0.8789225700311363
progress_handler every 1000 ops
0.8829826560104266
progress_handler every 1 op
2.8892734259716235
```

So running every 1000 ops makes almost no difference at all, but running every single op is a 3.2x performance degradation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074439309,https://api.github.com/repos/simonw/datasette/issues/1679,1074439309,IC_kwDOBm6k_c5ACqSN,9599,2022-03-21T21:28:58Z,2022-03-21T21:28:58Z,OWNER,"David Raymond solved it there: https://sqlite.org/forum/forumpost/330c8532d8a88bcd

> Don't forget to step through the results. All .execute() has done is prepared it.
>
>     db.execute(query).fetchall()

Sure enough, adding that gets the VM steps number up to 190,007 which is close enough that I'm happy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1676#issuecomment-1074378472,https://api.github.com/repos/simonw/datasette/issues/1676,1074378472,IC_kwDOBm6k_c5ACbbo,9599,2022-03-21T20:18:10Z,2022-03-21T20:18:10Z,OWNER,Maybe there is a better name for this method that helps emphasize its cascading nature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074347023,https://api.github.com/repos/simonw/datasette/issues/1679,1074347023,IC_kwDOBm6k_c5ACTwP,9599,2022-03-21T19:48:59Z,2022-03-21T19:48:59Z,OWNER,Posed a question about that here: https://sqlite.org/forum/forumpost/de9ff10fa7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074341924,https://api.github.com/repos/simonw/datasette/issues/1679,1074341924,IC_kwDOBm6k_c5ACSgk,9599,2022-03-21T19:42:08Z,2022-03-21T19:42:08Z,OWNER,"Here's the Python-C implementation of `set_progress_handler`: https://github.com/python/cpython/blob/4674fd4e938eb4a29ccd5b12c15455bd2a41c335/Modules/_sqlite/connection.c#L1177-L1201

It calls `sqlite3_progress_handler(self->db, n, progress_callback, ctx);`

https://www.sqlite.org/c3ref/progress_handler.html says:

> The parameter N is the approximate number of [virtual machine instructions](https://www.sqlite.org/opcode.html) that are evaluated between successive invocations of the callback X

So maybe VM-steps and virtual machine instructions are different things?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074337997,https://api.github.com/repos/simonw/datasette/issues/1679,1074337997,IC_kwDOBm6k_c5ACRjN,9599,2022-03-21T19:37:08Z,2022-03-21T19:37:08Z,OWNER,"This is weird:
```python
import sqlite3

db = sqlite3.connect("":memory:"")

i = 0

def count():
    global i
    i += 1


db.set_progress_handler(count, 1)

db.execute(""""""
with recursive counter(x) as (
  select 0
    union
  select x + 1 from counter
)
select * from counter limit 10000;
"""""")

print(i)
```
Outputs `24`. But if you try the same thing in the SQLite console:
```
sqlite> .stats vmstep
sqlite> with recursive counter(x) as (
   ...>   select 0
   ...>     union
   ...>   select x + 1 from counter
   ...> )
   ...> select * from counter limit 10000;
...
VM-steps: 200007
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074332718,https://api.github.com/repos/simonw/datasette/issues/1679,1074332718,IC_kwDOBm6k_c5ACQQu,9599,2022-03-21T19:31:10Z,2022-03-21T19:31:10Z,OWNER,How long does it take for SQLite to execute 1000 opcodes anyway?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074332325,https://api.github.com/repos/simonw/datasette/issues/1679,1074332325,IC_kwDOBm6k_c5ACQKl,9599,2022-03-21T19:30:44Z,2022-03-21T19:30:44Z,OWNER,So it looks like even for facet suggestion `n=1000` always - it's never reduced to `n=1`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1679#issuecomment-1074331743,https://api.github.com/repos/simonw/datasette/issues/1679,1074331743,IC_kwDOBm6k_c5ACQBf,9599,2022-03-21T19:30:05Z,2022-03-21T19:30:05Z,OWNER,"https://github.com/simonw/datasette/blob/1a7750eb29fd15dd2eea3b9f6e33028ce441b143/datasette/app.py#L118-L122 sets it to 50ms for facet suggestion but that's not going to pass `ms < 50`:

```python
    Setting(
        ""facet_suggest_time_limit_ms"",
        50,
        ""Time limit for calculating a suggested facet"",
    ),
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175854982,
https://github.com/simonw/datasette/issues/1660#issuecomment-1074321862,https://api.github.com/repos/simonw/datasette/issues/1660,1074321862,IC_kwDOBm6k_c5ACNnG,9599,2022-03-21T19:19:01Z,2022-03-21T19:19:01Z,OWNER,I've simplified this a ton now. I'm going to keep working on this in the long-term but I think this issue can be closed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,
https://github.com/simonw/datasette/issues/1678#issuecomment-1074302559,https://api.github.com/repos/simonw/datasette/issues/1678,1074302559,IC_kwDOBm6k_c5ACI5f,9599,2022-03-21T19:04:03Z,2022-03-21T19:04:03Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-resource-none,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175715988,
https://github.com/simonw/datasette/issues/1660#issuecomment-1074287177,https://api.github.com/repos/simonw/datasette/issues/1660,1074287177,IC_kwDOBm6k_c5ACFJJ,9599,2022-03-21T18:51:42Z,2022-03-21T18:51:42Z,OWNER,`BaseView` is looking a LOT slimmer now that I've moved all of the permissions stuff out of it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1170144879,
https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074256603,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1074256603,IC_kwDOCGYnMM5AB9rb,9954,2022-03-21T18:19:41Z,2022-03-21T18:19:41Z,NONE,"That makes sense; just a little hint that points folks towards doing the right thing might be helpful!

fwiw, the reason I was using jq in the first place was just a quick way to extract one attribute from an actual JSON array. When I initially imported it, I got a table with a bunch of embedded JSON values, rather than a native table, because each array entry had two attributes, one with the data I _actually_ wanted. Not sure how common a use-case this is, though (and easily fixed, aside from the jq weirdness!)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,
https://github.com/simonw/sqlite-utils/issues/417#issuecomment-1074243540,https://api.github.com/repos/simonw/sqlite-utils/issues/417,1074243540,IC_kwDOCGYnMM5AB6fU,9599,2022-03-21T18:08:03Z,2022-03-21T18:08:03Z,OWNER,"I've not really thought about standards as much here as I should. It looks like there are two competing specs for newline-delimited JSON!

http://ndjson.org/ is the one I've been using in `sqlite-utils` - and https://github.com/ndjson/ndjson-spec#31-serialization says:

> The JSON texts MUST NOT contain newlines or carriage returns.

https://jsonlines.org/ is the other one. It is slightly less clear, but it does say this:

> 2. Each Line is a Valid JSON Value
>
> The most common values will be objects or arrays, but any JSON value is permitted.

My interpretation of both of these is that newlines in the middle of a JSON object shouldn't be allowed.

So what's `jq` doing here? It looks to me like that `jq` format is its own thing - it's not actually compatible with either of those two loose specs described above.

The `jq` docs seem to call this ""whitespace-separated JSON"": https://stedolan.github.io/jq/manual/v1.6/#Invokingjq

The thing I like about newline-delimited JSON is that it's really trivial to parse - loop through each line, run it through `json.loads()` and that's it. No need to try and unwrap JSON objects that might span multiple lines.

Unless someone has written a robust Python implementation of a `jq`-compatible whitespace-separated JSON parser, I'm inclined to leave this as is. I'd be fine adding some documentation that helps point people towards `jq -c` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175744654,
https://github.com/simonw/datasette/issues/1677#issuecomment-1074184240,https://api.github.com/repos/simonw/datasette/issues/1677,1074184240,IC_kwDOBm6k_c5ABsAw,9599,2022-03-21T17:20:17Z,2022-03-21T17:20:17Z,OWNER,"https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/views/base.py#L69-L77

This is weirdly different from how `check_permissions()` used to work, in that it doesn't differentiate between `None` and `False`.

https://github.com/simonw/datasette/blob/4a4164b81191dec35e423486a208b05a9edc65e4/datasette/views/base.py#L79-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175694248,
https://github.com/simonw/datasette/issues/1676#issuecomment-1074180312,https://api.github.com/repos/simonw/datasette/issues/1676,1074180312,IC_kwDOBm6k_c5ABrDY,9599,2022-03-21T17:16:45Z,2022-03-21T17:16:45Z,OWNER,"When looking at this code earlier I assumed that the following would check each permission in turn and fail if any of them failed:
```python
await self.ds.ensure_permissions(
    request.actor,
    [
        (""view-table"", (database, table)),
        (""view-database"", database),
        ""view-instance"",
    ]
)
```
But it's not quite that simple: if any of them fail, it fails... but if an earlier one returns `True` the whole stack passes even if there would have been a failure later on!

If that is indeed the right abstraction, I need to work to make the documentation as clear as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,
https://github.com/simonw/datasette/issues/1676#issuecomment-1074178865,https://api.github.com/repos/simonw/datasette/issues/1676,1074178865,IC_kwDOBm6k_c5ABqsx,9599,2022-03-21T17:15:27Z,2022-03-21T17:15:27Z,OWNER,This method here: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/datasette/app.py#L632-L664,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175690070,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074177827,https://api.github.com/repos/simonw/datasette/issues/1675,1074177827,IC_kwDOBm6k_c5ABqcj,9599,2022-03-21T17:14:31Z,2022-03-21T17:14:31Z,OWNER,"Updated documentation: https://github.com/simonw/datasette/blob/e627510b760198ccedba9e5af47a771e847785c9/docs/internals.rst#await-ensure_permissionsactor-permissions

> This method allows multiple permissions to be checked at onced. It raises a `datasette.Forbidden` exception if any of the checks are denied before one of them is explicitly granted.
> 
> This is useful when you need to check multiple permissions at once. For example, an actor should be able to view a table if either one of the following checks returns `True` or not a single one of them returns `False`:

That's pretty hard to understand! I'm going to open a separate issue to reconsider if this is a useful enough abstraction given how confusing it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074161523,https://api.github.com/repos/simonw/datasette/issues/1675,1074161523,IC_kwDOBm6k_c5ABmdz,9599,2022-03-21T16:59:55Z,2022-03-21T17:00:03Z,OWNER,Also calling that function `permissions_allowed()` is confusing because there is a plugin hook with a similar name already: https://docs.datasette.io/en/stable/plugin_hooks.html#permission-allowed-datasette-actor-action-resource,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074158890,https://api.github.com/repos/simonw/datasette/issues/1675,1074158890,IC_kwDOBm6k_c5ABl0q,9599,2022-03-21T16:57:15Z,2022-03-21T16:57:15Z,OWNER,"Idea: `ds.permission_allowed()` continues to just return `True` or `False`.

A new `ds.ensure_permissions(...)` method is added which raises a `Forbidden` exception if a check fails (hence the different name)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074156779,https://api.github.com/repos/simonw/datasette/issues/1675,1074156779,IC_kwDOBm6k_c5ABlTr,9599,2022-03-21T16:55:08Z,2022-03-21T16:56:02Z,OWNER,"One benefit of the current design of `check_permissions` that raises an exception is that the exception includes information on WHICH of the permission checks failed. Returning just `True` or `False` loses that information.

I could return an object which evaluates to `False` but also carries extra information? Bit weird, I've never seen anything like that in other Python code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074143209,https://api.github.com/repos/simonw/datasette/issues/1675,1074143209,IC_kwDOBm6k_c5ABh_p,9599,2022-03-21T16:46:05Z,2022-03-21T16:46:05Z,OWNER,"The other difference though is that `ds.permission_allowed(...)` works against an actor, while `check_permission()` works against a request (though just to access `request.actor`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074142617,https://api.github.com/repos/simonw/datasette/issues/1675,1074142617,IC_kwDOBm6k_c5ABh2Z,9599,2022-03-21T16:45:27Z,2022-03-21T16:45:27Z,OWNER,"Though at that point `check_permission` is such a light wrapper around `self.ds.permission_allowed()` that there's little point in it existing at all.

So maybe `check_permisions()` becomes `ds.permissions_allowed()`.

`permission_allowed()` v.s. `permissions_allowed()` is a bit of a subtle naming difference, but I think it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,
https://github.com/simonw/datasette/issues/1675#issuecomment-1074141457,https://api.github.com/repos/simonw/datasette/issues/1675,1074141457,IC_kwDOBm6k_c5ABhkR,9599,2022-03-21T16:44:09Z,2022-03-21T16:44:09Z,OWNER,"A slightly odd thing about these methods is that they either fail silently or they raise a `Forbidden` exception.

Maybe they should instead return `True` or `False` and the calling code could decide if it wants to raise the exception? That would make them more usable and a little less surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1175648453,