html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/1362#issuecomment-855427396,https://api.github.com/repos/simonw/datasette/issues/1362,855427396,MDEyOklzc3VlQ29tbWVudDg1NTQyNzM5Ng==,9599,simonw,2021-06-06T16:46:17Z,2021-06-06T16:46:17Z,OWNER,"Mind you, since that plugin hook looks like this:

```python
@hookimpl
def extra_body_script():
    return {
        ""module"": True,
        ""script"": ""console.log('Your JavaScript goes here...')""
    }
```
Having it calculate a sha256 hash wouldn't be difficult.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426750,https://api.github.com/repos/simonw/datasette/issues/1362,855426750,MDEyOklzc3VlQ29tbWVudDg1NTQyNjc1MA==,9599,simonw,2021-06-06T16:41:30Z,2021-06-06T16:44:49Z,OWNER,"This is from the current `base.html` template: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/base.html#L62-L66

Which includes this: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/_close_open_menus.html#L1-L16

The `body_scripts` bit is for this `extra_body_script` plugin hook, which is the thing that will be the most affected by implementing CSP: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426516,https://api.github.com/repos/simonw/datasette/issues/1362,855426516,MDEyOklzc3VlQ29tbWVudDg1NTQyNjUxNg==,9599,simonw,2021-06-06T16:39:34Z,2021-06-06T16:39:34Z,OWNER,The reason Datasette uses small inline scripts right now is to avoid the overhead of an extra HTTP request for a JavaScript file - but these are both inherently cachable and perform much better under HTTP/2 so that's likely a false optimization.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426314,https://api.github.com/repos/simonw/datasette/issues/1362,855426314,MDEyOklzc3VlQ29tbWVudDg1NTQyNjMxNA==,9599,simonw,2021-06-06T16:38:04Z,2021-06-06T16:38:04Z,OWNER,"The other option for inline scripts is the CSP nonce:

    Content-Security-Policy: script-src 'nonce-2726c7f26c'

Then:

    <script nonce=""2726c7f26c"">
      var inline = 1;
    </script>

Since an attacker can't guess what the nonce will be it prevents them from injecting their own script block - this seems easier to make available to plugins than a full hashing mechanism, just make `{{ csp_nonce() }}` available to the template.

That template function can then be smart enough to set a flag which Datasette uses to decide if the `script-src 'nonce-2726c7f26c'` policy should be sent or not.

Presumably this would also require adding `Content-Security-Policy` to the `Vary` header though, which will have a nasty effect on Cloudflare and Fastly and such like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418899,https://api.github.com/repos/simonw/datasette/issues/1362,855418899,MDEyOklzc3VlQ29tbWVudDg1NTQxODg5OQ==,9599,simonw,2021-06-06T15:42:55Z,2021-06-06T15:42:55Z,OWNER,Another consideration: testing that this works correctly could require adoption of a real browser test environment (probably Cypress or maybe Playwright) to execute tests that will fail if CSP is violated.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418698,https://api.github.com/repos/simonw/datasette/issues/1362,855418698,MDEyOklzc3VlQ29tbWVudDg1NTQxODY5OA==,9599,simonw,2021-06-06T15:41:24Z,2021-06-06T15:41:24Z,OWNER,"I think the best way to answer these questions is with some prototyping - of both Datasette and some of the existing JavaScript plugins.

I can start with a `datasette-experimental-csp` plugin that sets the header (and could even run an optional report URI mechanism).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418401,https://api.github.com/repos/simonw/datasette/issues/1362,855418401,MDEyOklzc3VlQ29tbWVudDg1NTQxODQwMQ==,9599,simonw,2021-06-06T15:39:38Z,2021-06-06T15:39:38Z,OWNER,"The security benefit of forcing all JavaScript plugins to be written as CSP-friendly external scripts is very compelling though.

Other plugin-heavy ecosystems such as WordPress have suffered greatly from insecurely written plugins - could this be a huge security win for the Datasette ecosystem generally?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418065,https://api.github.com/repos/simonw/datasette/issues/1362,855418065,MDEyOklzc3VlQ29tbWVudDg1NTQxODA2NQ==,9599,simonw,2021-06-06T15:37:11Z,2021-06-06T15:37:11Z,OWNER,"The easiest way to apply CSP is to remove all inline `<script>` blocks (Datasette has a few) and instead serve JavaScript as separate linked files.

It's possible to keep inline script blocks by calculating a hash of their content and adding a `Content-Security-Policy: script-src 'sha256-B2yPHKaXnvFWtRChIbabYmUBFZdVfKKXHbWtWidDVF8='` to the policy. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src

This could be achieved with some Django template tricks, but it feels very risky - and done carelessly could end up calculating a hash of a reflected XSS attack!

The biggest challenge I see around here involves plugins and custom templates. Adopting CSP would require plugins to avoid using any inline scripts, instead keeping their entire implementations in `.js` files.

That's maybe not a bad thing, but it represents a big commitment. It would need to be adopted before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/283#issuecomment-855369819,https://api.github.com/repos/simonw/datasette/issues/283,855369819,MDEyOklzc3VlQ29tbWVudDg1NTM2OTgxOQ==,9599,simonw,2021-06-06T09:40:18Z,2021-06-06T09:40:18Z,OWNER,"> One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`.
> 
> I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue.

That issue is fixed in #1276.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/1361#issuecomment-855308811,https://api.github.com/repos/simonw/datasette/issues/1361,855308811,MDEyOklzc3VlQ29tbWVudDg1NTMwODgxMQ==,9599,simonw,2021-06-05T23:16:21Z,2021-06-05T23:16:21Z,OWNER,That seems to have fixed it. I'd love to get rid of this `restore_working_directory` hack entirely.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855307292,https://api.github.com/repos/simonw/datasette/issues/1361,855307292,MDEyOklzc3VlQ29tbWVudDg1NTMwNzI5Mg==,9599,simonw,2021-06-05T22:59:35Z,2021-06-05T22:59:35Z,OWNER,"That broke things. Here's how `pytest-cov` fixed a similar issue: https://github.com/pytest-dev/pytest-cov/commit/7ccb1783bf8290447df58deeb41eae74296a6d9b

See also https://github.com/nedbat/coveragepy/issues/881 and https://github.com/pytest-dev/pytest-cov/issues/306 and https://github.com/nedbat/coveragepy/issues/824","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855306497,https://api.github.com/repos/simonw/datasette/issues/1361,855306497,MDEyOklzc3VlQ29tbWVudDg1NTMwNjQ5Nw==,9599,simonw,2021-06-05T22:51:01Z,2021-06-05T22:51:01Z,OWNER,I'm going to try removing that `restore_working_directory` fixture entirely.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855306347,https://api.github.com/repos/simonw/datasette/issues/1361,855306347,MDEyOklzc3VlQ29tbWVudDg1NTMwNjM0Nw==,9599,simonw,2021-06-05T22:49:30Z,2021-06-05T22:49:30Z,OWNER,"Stack Overflow: https://stackoverflow.com/a/49367679/6083

> The answer was that `os.chdir()` had been set to the deleted directory by accident. The directory was missing, but the error happened (it seems) at the attempt to get it with `os.getcwd()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1360#issuecomment-855303776,https://api.github.com/repos/simonw/datasette/issues/1360,855303776,MDEyOklzc3VlQ29tbWVudDg1NTMwMzc3Ng==,9599,simonw,2021-06-05T22:23:23Z,2021-06-05T22:23:23Z,OWNER,"Worth noting that I found this issue myself, and to my knowledge it has not been uncovered by anyone else prior to the patch being released.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912464443,"Security flaw, to be fixed in 0.56.1 and 0.57",
https://github.com/simonw/datasette/issues/1360#issuecomment-855303649,https://api.github.com/repos/simonw/datasette/issues/1360,855303649,MDEyOklzc3VlQ29tbWVudDg1NTMwMzY0OQ==,9599,simonw,2021-06-05T22:22:06Z,2021-06-05T22:22:06Z,OWNER,I've released fixes in both 0.56.1 and 0.57.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912464443,"Security flaw, to be fixed in 0.56.1 and 0.57",
https://github.com/simonw/datasette/issues/1358#issuecomment-855302339,https://api.github.com/repos/simonw/datasette/issues/1358,855302339,MDEyOklzc3VlQ29tbWVudDg1NTMwMjMzOQ==,9599,simonw,2021-06-05T22:08:16Z,2021-06-05T22:08:16Z,OWNER,"Release notes are in, pushing the release now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/issues/1358#issuecomment-855302320,https://api.github.com/repos/simonw/datasette/issues/1358,855302320,MDEyOklzc3VlQ29tbWVudDg1NTMwMjMyMA==,9599,simonw,2021-06-05T22:08:06Z,2021-06-05T22:08:06Z,OWNER,See #1360 for the security fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/issues/1358#issuecomment-855288228,https://api.github.com/repos/simonw/datasette/issues/1358,855288228,MDEyOklzc3VlQ29tbWVudDg1NTI4ODIyOA==,9599,simonw,2021-06-05T19:57:18Z,2021-06-05T19:57:18Z,OWNER,"There's also a security fix I need to make, so I'm not going to block this on wrapping up the work on the new Docker image testing from #1344 before putting out this release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/pull/1291#issuecomment-855287200,https://api.github.com/repos/simonw/datasette/issues/1291,855287200,MDEyOklzc3VlQ29tbWVudDg1NTI4NzIwMA==,9599,simonw,2021-06-05T19:48:36Z,2021-06-05T19:48:36Z,OWNER,"This is great, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849582643,Update docs: explain allow_download setting,
https://github.com/simonw/datasette/issues/1356#issuecomment-855282466,https://api.github.com/repos/simonw/datasette/issues/1356,855282466,MDEyOklzc3VlQ29tbWVudDg1NTI4MjQ2Ng==,9599,simonw,2021-06-05T19:05:06Z,2021-06-05T19:05:06Z,OWNER,Yeah that's a good point. I avoided making them sub-commands because `datasette serve` already supports the multitude of other arguments they also need... but actually that was just me being lazy - I can easily share arguments between multiple functions like I do in `sqlite-utils` itself: https://github.com/simonw/sqlite-utils/blob/d1a372b3006e6cf7d2017b3ddc484bf5c033e45d/sqlite_utils/cli.py#L46,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1238#issuecomment-855278998,https://api.github.com/repos/simonw/datasette/issues/1238,855278998,MDEyOklzc3VlQ29tbWVudDg1NTI3ODk5OA==,9599,simonw,2021-06-05T18:37:16Z,2021-06-05T18:37:16Z,OWNER,"Alternative idea: populate `request.scope` with a new `route_path` which is the base-url-stripped version, which we then use for other routing operations.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855278540,https://api.github.com/repos/simonw/datasette/issues/1238,855278540,MDEyOklzc3VlQ29tbWVudDg1NTI3ODU0MA==,9599,simonw,2021-06-05T18:33:25Z,2021-06-05T18:33:25Z,OWNER,"Got the test to pass by ensuring the tests don't accidentally double-rewrite the path.

Ran into a new problem:
```
    @pytest.mark.asyncio
    @pytest.mark.parametrize(
        ""prefix,expected_path"", [(None, ""/asgi-scope""), (""/prefix/"", ""/prefix/asgi-scope"")]
    )
    async def test_client_path(datasette, prefix, expected_path):
        original_base_url = datasette._settings[""base_url""]
        try:
            if prefix is not None:
                datasette._settings[""base_url""] = prefix
            response = await datasette.client.get(""/asgi-scope"")
            path = response.json()[""path""]
>           assert path == expected_path
E           AssertionError: assert '/asgi-scope' == '/prefix/asgi-scope'
E             - /prefix/asgi-scope
E             ? -------
E             + /asgi-scope
```
That test confirms that messing around with the `base_url` doesn't modify the ASGI scope... but the fix I'm using for this issue DOES modify the ASGI scope.

The question raised here is: should the ASGI scope stay unmodified when `base_url` is used?

I think it should. It doesn't make sense to obscure the ""real"" path just to get custom pages to work properly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855272693,https://api.github.com/repos/simonw/datasette/issues/1238,855272693,MDEyOklzc3VlQ29tbWVudDg1NTI3MjY5Mw==,9599,simonw,2021-06-05T17:45:42Z,2021-06-05T17:45:42Z,OWNER,"Applying this fix worked when I manually tested it:
```diff
         base_url = self.ds.setting(""base_url"")
         if base_url != ""/"" and path.startswith(base_url):
             path = ""/"" + path[len(base_url) :]
+            scope = dict(scope, path=path, raw_path=path.encode(""utf-8""))
         request = Request(scope, receive)
```
But... the test I wrote still failed. My hunch is that this is because deep within the test framework requests go through `ds.client` which may be applying its own changes relevant to `base_url`:

https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/utils/testing.py#L139","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855270917,https://api.github.com/repos/simonw/datasette/issues/1238,855270917,MDEyOklzc3VlQ29tbWVudDg1NTI3MDkxNw==,9599,simonw,2021-06-05T17:32:29Z,2021-06-05T17:32:29Z,OWNER,"This looks like the cause: https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/app.py#L1087-L1092

Note how `path` is modified... but then we create a new `Request()` that uses the old scope, which has unmodified `scope[""path""]` - and then the code later on looks at `request.scope[""path""]` when deciding if the request matches:

https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1154-L1155","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853567861,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853567861,MDEyOklzc3VlQ29tbWVudDg1MzU2Nzg2MQ==,9599,simonw,2021-06-03T05:12:21Z,2021-06-03T05:12:21Z,OWNER,I think this is more likely to happen in Datasette than in sqlite-utils - see https://github.com/simonw/datasette/issues/1356 for thoughts on this.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/datasette/issues/526#issuecomment-853567413,https://api.github.com/repos/simonw/datasette/issues/526,853567413,MDEyOklzc3VlQ29tbWVudDg1MzU2NzQxMw==,9599,simonw,2021-06-03T05:11:27Z,2021-06-03T05:11:27Z,OWNER,"Another potential way to implement this would be to hold the SQLite connection open and execute the full query there.

I've avoided this in the past due to concerns of resource exhaustion - if multiple requests attempt this at the same time all of the connections in the pool will become tied up and the site will be unable to respond to further requests.

But... now that Datasette has authentication there's the possibility of making this feature only available to specific authenticated users - the `--root` user for example. Which avoids the danger while unlocking a super-useful feature.

Not to mention people who are running Datasette privately on their own laptop, or the proposed `--query` CLI feature in #1356.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1356#issuecomment-853566337,https://api.github.com/repos/simonw/datasette/issues/1356,853566337,MDEyOklzc3VlQ29tbWVudDg1MzU2NjMzNw==,9599,simonw,2021-06-03T05:08:32Z,2021-06-03T05:08:32Z,OWNER,Also relevant: CSV streaming for canned queries in #526 - even better if we could stream ANY SQL query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1062#issuecomment-853566011,https://api.github.com/repos/simonw/datasette/issues/1062,853566011,MDEyOklzc3VlQ29tbWVudDg1MzU2NjAxMQ==,9599,simonw,2021-06-03T05:07:42Z,2021-06-03T05:07:42Z,OWNER,Implementing this would make #1356 a whole lot more interesting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1356#issuecomment-853565850,https://api.github.com/repos/simonw/datasette/issues/1356,853565850,MDEyOklzc3VlQ29tbWVudDg1MzU2NTg1MA==,9599,simonw,2021-06-03T05:07:21Z,2021-06-03T05:07:21Z,OWNER,"A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177.

So there's not much point implementing this unless we first make plugins able to add custom streaming formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853562891,https://api.github.com/repos/simonw/datasette/issues/1356,853562891,MDEyOklzc3VlQ29tbWVudDg1MzU2Mjg5MQ==,9599,simonw,2021-06-03T04:59:40Z,2021-06-03T04:59:40Z,OWNER,"It's weird that `--get` is documented on this page right now: https://docs.datasette.io/en/stable/getting_started.html#datasette-get

If I implement this I should build a new ""Datasette on the command-line"" documentation page to cover both `--get` and `--query`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560870,https://api.github.com/repos/simonw/datasette/issues/1356,853560870,MDEyOklzc3VlQ29tbWVudDg1MzU2MDg3MA==,9599,simonw,2021-06-03T04:53:47Z,2021-06-03T04:53:56Z,OWNER,"This is also interesting when used in conjunction with the proposed `datasette insert` command from #1163 - Datasette would become a plugin-driven CLI tool for both ingesting and outputting data, as a side-effect of its web features.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560389,https://api.github.com/repos/simonw/datasette/issues/1356,853560389,MDEyOklzc3VlQ29tbWVudDg1MzU2MDM4OQ==,9599,simonw,2021-06-03T04:52:13Z,2021-06-03T04:52:13Z,OWNER,I should implement #1355 for more efficient `--csv` streaming as part of this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560237,https://api.github.com/repos/simonw/datasette/issues/1356,853560237,MDEyOklzc3VlQ29tbWVudDg1MzU2MDIzNw==,9599,simonw,2021-06-03T04:51:49Z,2021-06-03T04:51:49Z,OWNER,This feels like a relatively simple feature to implement that unlocks a whole new set of possible uses for Datasette - as described by @eyeseast in https://github.com/simonw/sqlite-utils/issues/264#issue-907642546.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853559915,https://api.github.com/repos/simonw/datasette/issues/1356,853559915,MDEyOklzc3VlQ29tbWVudDg1MzU1OTkxNQ==,9599,simonw,2021-06-03T04:50:52Z,2021-06-03T04:50:52Z,OWNER,"What happens if you pass multiple databases? The `--query` would be executed against the first one. And if you pass `--crossdb` it would be executed against the `/_memory` database and would support cross-database joins.

Key thing here is that output plugins are supported (also plugins that add new SQL functions), making many Datasette plugins usable from the command-line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853558741,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853558741,MDEyOklzc3VlQ29tbWVudDg1MzU1ODc0MQ==,9599,simonw,2021-06-03T04:47:19Z,2021-06-03T04:47:19Z,OWNER,"This inspired me to re-examine how `--get` works, hence this issue: https://github.com/simonw/datasette/issues/1355","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/datasette/issues/1355#issuecomment-853557439,https://api.github.com/repos/simonw/datasette/issues/1355,853557439,MDEyOklzc3VlQ29tbWVudDg1MzU1NzQzOQ==,9599,simonw,2021-06-03T04:43:14Z,2021-06-03T04:43:14Z,OWNER,"It's using `TestClient` at the moment which is a wrapper around `httpx` (as of ) that uses the `@async_to_sync` decorator to hide the async nature.

https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/utils/testing.py#L102-L156

Maybe the fix here is to switch the `--get` implementation to using `httpx` directly with https://www.python-httpx.org/async/#streaming-responses","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910088936,datasette --get should efficiently handle streaming CSV,
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853553754,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853553754,MDEyOklzc3VlQ29tbWVudDg1MzU1Mzc1NA==,9599,simonw,2021-06-03T04:32:42Z,2021-06-03T04:36:36Z,OWNER,"This is a really interesting thought. I've so far resisted the temptation to add plugins to `sqlite-utils`, partly to avoid overlap with Datasette - but I'm open to discussing it.

There's actually a way for you to do what you're describing using `datasette` on the command-line, though it's a little obscure - also Datasette doesn't yet have a GeoJSON output extension, though it really should have one.

Here's an example using [datasette-yaml](https://datasette.io/plugins/datasette-yaml):

```
datasette /tmp/covid.db --get='/covid/ny_times_us_counties.yaml'
- rowid: 1
  date: '2020-01-21'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
- rowid: 2
  date: '2020-01-22'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
```
It even works with arbitrary SQL queries, though you might have to apply URL encoding to the `--get` string (this seems to work though):
```
datasette /tmp/covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 2' 
- date: '2020-01-21'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
- date: '2020-01-22'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
```
Here's the documentation for `--get`: https://docs.datasette.io/en/latest/getting_started.html#datasette-get","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853554550,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853554550,MDEyOklzc3VlQ29tbWVudDg1MzU1NDU1MA==,9599,simonw,2021-06-03T04:34:38Z,2021-06-03T04:34:38Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-indexes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853548442,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853548442,MDEyOklzc3VlQ29tbWVudDg1MzU0ODQ0Mg==,9599,simonw,2021-06-03T04:16:49Z,2021-06-03T04:16:49Z,OWNER,Needs to show the table each index applies to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853547681,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853547681,MDEyOklzc3VlQ29tbWVudDg1MzU0NzY4MQ==,9599,simonw,2021-06-03T04:14:28Z,2021-06-03T04:14:28Z,OWNER,"This looks good:
```
% sqlite-utils indexes /tmp/covid.db -t      
index_name                                                seqno    cid  name                 desc  coll      key
------------------------------------------------------  -------  -----  -----------------  ------  ------  -----
idx_johns_hopkins_csse_daily_reports_combined_key             0     12  combined_key            0  BINARY      1
idx_johns_hopkins_csse_daily_reports_country_or_region        0      1  country_or_region       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_province_or_state        0      2  province_or_state       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_day                      0      0  day                     0  BINARY      1
idx_ny_times_us_counties_date                                 0      0  date                    1  BINARY      1
idx_ny_times_us_counties_fips                                 0      3  fips                    0  BINARY      1
idx_ny_times_us_counties_county                               0      1  county                  0  BINARY      1
idx_ny_times_us_counties_state                                0      2  state                   0  BINARY      1

% sqlite-utils indexes /tmp/covid.db -t --aux
index_name                                                seqno    cid  name                 desc  coll      key
------------------------------------------------------  -------  -----  -----------------  ------  ------  -----
idx_johns_hopkins_csse_daily_reports_combined_key             0     12  combined_key            0  BINARY      1
idx_johns_hopkins_csse_daily_reports_combined_key             1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_country_or_region        0      1  country_or_region       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_country_or_region        1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_province_or_state        0      2  province_or_state       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_province_or_state        1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_day                      0      0  day                     0  BINARY      1
idx_johns_hopkins_csse_daily_reports_day                      1     -1                          0  BINARY      0
idx_ny_times_us_counties_date                                 0      0  date                    1  BINARY      1
idx_ny_times_us_counties_date                                 1     -1                          0  BINARY      0
idx_ny_times_us_counties_fips                                 0      3  fips                    0  BINARY      1
idx_ny_times_us_counties_fips                                 1     -1                          0  BINARY      0
idx_ny_times_us_counties_county                               0      1  county                  0  BINARY      1
idx_ny_times_us_counties_county                               1     -1                          0  BINARY      0
idx_ny_times_us_counties_state                                0      2  state                   0  BINARY      1
idx_ny_times_us_counties_state                                1     -1                          0  BINARY      0
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853546818,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853546818,MDEyOklzc3VlQ29tbWVudDg1MzU0NjgxOA==,9599,simonw,2021-06-03T04:11:46Z,2021-06-03T04:11:46Z,OWNER,"By default I won't return auxiliary columns, but I'll offer a `--aux` option to return them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853545743,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853545743,MDEyOklzc3VlQ29tbWVudDg1MzU0NTc0Mw==,9599,simonw,2021-06-03T04:08:04Z,2021-06-03T04:08:04Z,OWNER,"Figuring out the right queries:

https://covid-19.datasettes.com/covid?sql=select+sqlite_master.name%2C+i.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Awhere+type+%3D+%27table%27

This query shows all columns across all indexes across all tables:

```sql
select i.name as index_name, xinfo.* from sqlite_master
join pragma_index_list(sqlite_master.name) i
join pragma_index_xinfo(index_name) xinfo
where sqlite_master.type = 'table'
```
https://covid-19.datasettes.com/covid?sql=select+i.name+as+index_name%2C+xinfo.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Ajoin+pragma_index_xinfo%28index_name%29+xinfo%0D%0Awhere+sqlite_master.type+%3D+%27table%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853544493,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853544493,MDEyOklzc3VlQ29tbWVudDg1MzU0NDQ5Mw==,9599,simonw,2021-06-03T04:03:59Z,2021-06-03T04:03:59Z,OWNER,"Here's how `sqlite-utils triggers` works: https://github.com/simonw/sqlite-utils/blob/9c67cb925253cd5ef54a1fe0496e0ff9caeacfd6/sqlite_utils/cli.py#L1266-L1277

Running it from a SQL query makes it easy to support modifiers like `--csv` and `-t`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853541869,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853541869,MDEyOklzc3VlQ29tbWVudDg1MzU0MTg2OQ==,9599,simonw,2021-06-03T03:54:14Z,2021-06-03T03:54:14Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#xindexes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853535559,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853535559,MDEyOklzc3VlQ29tbWVudDg1MzUzNTU1OQ==,9599,simonw,2021-06-03T03:32:47Z,2021-06-03T03:32:47Z,OWNER,"New design:
```python
def test_xindexes(fresh_db):
    fresh_db.executescript(
        """"""
        create table Gosh (c1 text, c2 text, c3 text);
        create index Gosh_c1 on Gosh(c1);
        create index Gosh_c2c3 on Gosh(c2, c3 desc);
    """"""
    )
    assert fresh_db[""Gosh""].xindexes == [
        XIndex(
            name=""Gosh_c2c3"",
            columns=[
                XIndexColumn(seqno=0, cid=1, name=""c2"", desc=0, coll=""BINARY"", key=1),
                XIndexColumn(seqno=1, cid=2, name=""c3"", desc=1, coll=""BINARY"", key=1),
                XIndexColumn(seqno=2, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
        XIndex(
            name=""Gosh_c1"",
            columns=[
                XIndexColumn(seqno=0, cid=0, name=""c1"", desc=0, coll=""BINARY"", key=1),
                XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853534732,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853534732,MDEyOklzc3VlQ29tbWVudDg1MzUzNDczMg==,9599,simonw,2021-06-03T03:30:10Z,2021-06-03T03:30:10Z,OWNER,"I'm going to return `XIndex(name, columns)`  - where `columns` is a list of `XIndexColumn`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853530348,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853530348,MDEyOklzc3VlQ29tbWVudDg1MzUzMDM0OA==,9599,simonw,2021-06-03T03:16:33Z,2021-06-03T03:16:33Z,OWNER,"In prototyping this out I realize that I actually want to get back the name of each index, then for each of them the detailed list of index columns. Here's the test from my initial prototype:
```python
def test_xindexes(fresh_db):
    fresh_db.executescript(
        """"""
        create table Gosh (c1 text, c2 text, c3 text);
        create index Gosh_c1 on Gosh(c1);
        create index Gosh_c2c3 on Gosh(c2, c3 desc);
    """"""
    )
    assert fresh_db[""Gosh""].xindexes == [
        (
            ""Gosh_c2c3"",
            [
                XIndex(seqno=0, cid=1, name=""c2"", desc=0, coll=""BINARY"", key=1),
                XIndex(seqno=1, cid=2, name=""c3"", desc=1, coll=""BINARY"", key=1),
                XIndex(seqno=2, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
        (
            ""Gosh_c1"",
            [
                XIndex(seqno=0, cid=0, name=""c1"", desc=0, coll=""BINARY"", key=1),
                XIndex(seqno=1, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853525036,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853525036,MDEyOklzc3VlQ29tbWVudDg1MzUyNTAzNg==,9599,simonw,2021-06-03T03:02:22Z,2021-06-03T03:02:22Z,OWNER,"This would be a breaking change - and the fact that it returns auxiliary columns isn't particularly useful for most cases - ""Auxiliary columns are additional columns needed to locate the table entry that corresponds to each index entry"".

Instead, I'm going to add a new property `table.xindexes` which exposes this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/datasette/issues/893#issuecomment-852712469,https://api.github.com/repos/simonw/datasette/issues/893,852712469,MDEyOklzc3VlQ29tbWVudDg1MjcxMjQ2OQ==,9599,simonw,2021-06-02T04:29:56Z,2021-06-02T04:29:56Z,OWNER,"I think I fixed this a while ago, though I can't find the commit now. Please re-open if the bug still happens!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",656959584,pip3 install datasette not serving static on linuxbrew.,
https://github.com/simonw/datasette/issues/1127#issuecomment-852712106,https://api.github.com/repos/simonw/datasette/issues/1127,852712106,MDEyOklzc3VlQ29tbWVudDg1MjcxMjEwNg==,9599,simonw,2021-06-02T04:28:55Z,2021-06-02T04:28:55Z,OWNER,This became resizable in #1236.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756818250,Make the custom SQL query text box larger or resizable,
https://github.com/simonw/datasette/pull/1306#issuecomment-852710117,https://api.github.com/repos/simonw/datasette/issues/1306,852710117,MDEyOklzc3VlQ29tbWVudDg1MjcxMDExNw==,9599,simonw,2021-06-02T04:24:33Z,2021-06-02T04:24:33Z,OWNER,I have a test - I'll merge this and then add that test.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/pull/1306#issuecomment-852708574,https://api.github.com/repos/simonw/datasette/issues/1306,852708574,MDEyOklzc3VlQ29tbWVudDg1MjcwODU3NA==,9599,simonw,2021-06-02T04:20:18Z,2021-06-02T04:20:18Z,OWNER,https://github.com/simonw/datasette/blob/115332ce76c0e867d9936406aaf4bcee6b1ef3cb/datasette/views/index.py#L77 helps here - looks like we can pass `?_sort=relationships` to trigger the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/pull/1306#issuecomment-852707676,https://api.github.com/repos/simonw/datasette/issues/1306,852707676,MDEyOklzc3VlQ29tbWVudDg1MjcwNzY3Ng==,9599,simonw,2021-06-02T04:17:41Z,2021-06-02T04:17:41Z,OWNER,We need to add a test that illustrates the bug in #1305 so we can prove that this definitely fixes it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/issues/619#issuecomment-852706970,https://api.github.com/repos/simonw/datasette/issues/619,852706970,MDEyOklzc3VlQ29tbWVudDg1MjcwNjk3MA==,9599,simonw,2021-06-02T04:15:54Z,2021-06-02T04:15:54Z,OWNER,"<img width=""1009"" alt=""fixtures__select___from__foo__and_Facets_should_not_execute_for__shape_array_object_·_Issue__263_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/120423073-8a338700-c31e-11eb-8c8f-affc24708b32.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-852703357,https://api.github.com/repos/simonw/datasette/issues/619,852703357,MDEyOklzc3VlQ29tbWVudDg1MjcwMzM1Nw==,9599,simonw,2021-06-02T04:08:03Z,2021-06-02T04:08:03Z,OWNER,"A SQL error now looks like this: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D

<img width=""1023"" alt=""fixtures__select___from__foo_"" src=""https://user-images.githubusercontent.com/9599/120422579-691e6680-c31d-11eb-99ea-655bdc2a401d.png"">

I'm going to get rid of that ""0 results"" message if an error is shown.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/1257#issuecomment-852686827,https://api.github.com/repos/simonw/datasette/issues/1257,852686827,MDEyOklzc3VlQ29tbWVudDg1MjY4NjgyNw==,9599,simonw,2021-06-02T03:26:27Z,2021-06-02T03:26:27Z,OWNER,I tried and failed to get this fix working for tables with double quotes in their name - I couldn't figure out what the double-quote-in-a-table-name version of this code would look like: https://github.com/simonw/datasette/blob/807de378d08752a0f05bb1b980a0a62620a70520/datasette/utils/__init__.py#L538-L548,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828811618,Table names containing single quotes break things,
https://github.com/simonw/datasette/issues/1257#issuecomment-852681622,https://api.github.com/repos/simonw/datasette/issues/1257,852681622,MDEyOklzc3VlQ29tbWVudDg1MjY4MTYyMg==,9599,simonw,2021-06-02T03:12:18Z,2021-06-02T03:12:18Z,OWNER,"Created a test database like this:
```
% sqlite-utils create-table quote-in-name.db ""this'hasquoteinname"" id integer name text --pk id
% datasette quote-in-name.db -p 8025 --pdb
INFO:     Started server process [86046]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8025 (Press CTRL+C to quit)
> /Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py(530)detect_fts()
-> rows = conn.execute(detect_fts_sql(table)).fetchall()
(Pdb) c
Traceback (most recent call last):
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 1124, in route_path
    response = await view(request, send)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 147, in view
    return await self.dispatch_request(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 122, in dispatch_request
    return await handler(request, *args, **kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/index.py"", line 72, in get
    ""fts_table"": await db.fts_table(table),
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 279, in fts_table
    return await self.execute_fn(lambda conn: detect_fts(conn, table))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 155, in execute_fn
    return await asyncio.get_event_loop().run_in_executor(
  File ""/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/concurrent/futures/thread.py"", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 153, in in_thread
    return fn(conn)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 279, in <lambda>
    return await self.execute_fn(lambda conn: detect_fts(conn, table))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 530, in detect_fts
    rows = conn.execute(detect_fts_sql(table)).fetchall()
sqlite3.OperationalError: near ""hasquoteinname"": syntax error
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828811618,Table names containing single quotes break things,
https://github.com/simonw/datasette/pull/1352#issuecomment-852673695,https://api.github.com/repos/simonw/datasette/issues/1352,852673695,MDEyOklzc3VlQ29tbWVudDg1MjY3MzY5NQ==,9599,simonw,2021-06-02T02:52:26Z,2021-06-02T02:52:26Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908276134,Bump black from 21.5b1 to 21.5b2,
https://github.com/simonw/datasette/issues/263#issuecomment-852256784,https://api.github.com/repos/simonw/datasette/issues/263,852256784,MDEyOklzc3VlQ29tbWVudDg1MjI1Njc4NA==,9599,simonw,2021-06-01T16:20:09Z,2021-06-01T16:20:09Z,OWNER,This is a lot easier to implement now that we have a `?_nofacet=1` option from #1350.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323671577,Facets should not execute for ?shape=array|object,
https://github.com/simonw/datasette/issues/1350#issuecomment-852256454,https://api.github.com/repos/simonw/datasette/issues/1350,852256454,MDEyOklzc3VlQ29tbWVudDg1MjI1NjQ1NA==,9599,simonw,2021-06-01T16:19:38Z,2021-06-01T16:19:38Z,OWNER,I renamed this to `?_nofacet=1` in #1353.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906977719,?_nofacets=1 query string argument for disabling facets and suggested facets,
https://github.com/simonw/datasette/issues/1353#issuecomment-852255951,https://api.github.com/repos/simonw/datasette/issues/1353,852255951,MDEyOklzc3VlQ29tbWVudDg1MjI1NTk1MQ==,9599,simonw,2021-06-01T16:18:54Z,2021-06-01T16:18:54Z,OWNER,Documented here: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908446997,?_nocount=1 for opting out of table counts,
https://github.com/simonw/datasette/issues/1354#issuecomment-852254298,https://api.github.com/repos/simonw/datasette/issues/1354,852254298,MDEyOklzc3VlQ29tbWVudDg1MjI1NDI5OA==,9599,simonw,2021-06-01T16:16:32Z,2021-06-01T16:16:32Z,OWNER,Running `python update-docs-help.py` helps fix this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908465747,Update help in tests for latest Click,
https://github.com/simonw/datasette/issues/1353#issuecomment-852238201,https://api.github.com/repos/simonw/datasette/issues/1353,852238201,MDEyOklzc3VlQ29tbWVudDg1MjIzODIwMQ==,9599,simonw,2021-06-01T15:53:57Z,2021-06-01T15:53:57Z,OWNER,I'm going to rename `?_nofacets=1` to `?_nofacet=1` to keep it consistent with the new `?_nocount=1` option (and because I don't like `?_nocounts=1`).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908446997,?_nocount=1 for opting out of table counts,
https://github.com/simonw/datasette/issues/1349#issuecomment-852237347,https://api.github.com/repos/simonw/datasette/issues/1349,852237347,MDEyOklzc3VlQ29tbWVudDg1MjIzNzM0Nw==,9599,simonw,2021-06-01T15:52:50Z,2021-06-01T15:52:50Z,OWNER,Fixed in https://github.com/simonw/datasette/commit/d1d06ace49606da790a765689b4fbffa4c6deecb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1351#issuecomment-851133471,https://api.github.com/repos/simonw/datasette/issues/1351,851133471,MDEyOklzc3VlQ29tbWVudDg1MTEzMzQ3MQ==,9599,simonw,2021-05-31T03:02:59Z,2021-05-31T03:02:59Z,OWNER,"Since traces only work with `text/html` and JSON at the moment, the easiest way to do this will be to wrap generated CSV in a HTML page in a textarea if the user specified `?_trace=1`:

https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906993731,Get `?_trace=1` working with CSV and streaming CSVs,
https://github.com/simonw/datasette/issues/1349#issuecomment-851133125,https://api.github.com/repos/simonw/datasette/issues/1349,851133125,MDEyOklzc3VlQ29tbWVudDg1MTEzMzEyNQ==,9599,simonw,2021-05-31T03:01:48Z,2021-05-31T03:01:48Z,OWNER,I think it's worth getting `?_trace=1` to work with streaming CSV - this would have helped me spot this issue a long time ago.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851131999,https://api.github.com/repos/simonw/datasette/issues/1349,851131999,MDEyOklzc3VlQ29tbWVudDg1MTEzMTk5OQ==,9599,simonw,2021-05-31T02:57:25Z,2021-05-31T02:57:25Z,OWNER,"I'm having a really hard time figuring out how to unit test this - ideally I'd monitor which SQL queries are executed using the tracing mechanism, but that's not set up to work with anything other than HTML or JSON outputs: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851129464,https://api.github.com/repos/simonw/datasette/issues/1349,851129464,MDEyOklzc3VlQ29tbWVudDg1MTEyOTQ2NA==,9599,simonw,2021-05-31T02:48:06Z,2021-05-31T02:48:06Z,OWNER,"Actually there is precedent for swapping out `request.scope` for a new scope, as seen here in the routing code: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/app.py#L1117-L1122","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851129019,https://api.github.com/repos/simonw/datasette/issues/1349,851129019,MDEyOklzc3VlQ29tbWVudDg1MTEyOTAxOQ==,9599,simonw,2021-05-31T02:46:38Z,2021-05-31T02:46:38Z,OWNER,I think the right way to do this is to construct a new `Request` with a modified ASGI scope: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/utils/asgi.py#L88-L99,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851128524,https://api.github.com/repos/simonw/datasette/issues/1349,851128524,MDEyOklzc3VlQ29tbWVudDg1MTEyODUyNA==,9599,simonw,2021-05-31T02:44:44Z,2021-05-31T02:44:44Z,OWNER,"Now that I have `?_nofacets=1` I can use that to fix this. The challenge is that in this block of code I need to modify the incoming request's query string arguments, which isn't something I've done before: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L270
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1345#issuecomment-851127314,https://api.github.com/repos/simonw/datasette/issues/1345,851127314,MDEyOklzc3VlQ29tbWVudDg1MTEyNzMxNA==,9599,simonw,2021-05-31T02:40:26Z,2021-05-31T02:40:44Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state - the state column is not selected but facet by state still works: 

<img width=""1134"" alt=""fixtures__facetable__15_rows_and_Why_Russians_do_not_smile"" src=""https://user-images.githubusercontent.com/9599/120132273-110a2780-c198-11eb-91a5-cafb77a78c97.png"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1350#issuecomment-851122934,https://api.github.com/repos/simonw/datasette/issues/1350,851122934,MDEyOklzc3VlQ29tbWVudDg1MTEyMjkzNA==,9599,simonw,2021-05-31T02:23:47Z,2021-05-31T02:23:47Z,OWNER,"I think `?_facets_off=1` is a good design for this.

I considered `?_facet=` with an empty string but that's a bit weird, and it's not clear what that would do if used like `?_facet=state&_facet=`.

So it definitely needs to be a separate named option.

Actually, since I have `?_nocol=` I'm going to call this `?_nofacets=1` instead as that's a little more consistent.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906977719,?_nofacets=1 query string argument for disabling facets and suggested facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851120306,https://api.github.com/repos/simonw/datasette/issues/1345,851120306,MDEyOklzc3VlQ29tbWVudDg1MTEyMDMwNg==,9599,simonw,2021-05-31T02:14:36Z,2021-05-31T02:14:36Z,OWNER,"Yes! This was easier than I thought. I'm going with that solution - where facets are calculated against all columns, ignoring `?_col=` and `?_nocol=` entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851117233,https://api.github.com/repos/simonw/datasette/issues/1345,851117233,MDEyOklzc3VlQ29tbWVudDg1MTExNzIzMw==,9599,simonw,2021-05-31T02:04:35Z,2021-05-31T02:04:35Z,OWNER,"That long-term solution may not be too difficult. The facets are calculated against `sql_no_limit` which is constructed here:

https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L659-L665

And used here:

https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L706-L718

Crucially, `sql_no_limit` is ONLY used for faceting - nothing else uses it anywhere. So constructing it before constructing `sql` and taking `?_col=` and `?_nocol=` into account may not be a complex change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851116105,https://api.github.com/repos/simonw/datasette/issues/1345,851116105,MDEyOklzc3VlQ29tbWVudDg1MTExNjEwNQ==,9599,simonw,2021-05-31T02:00:44Z,2021-05-31T02:00:44Z,OWNER,"Maybe there's a short-term and longer-term solution for this - where the long-term solution is to use different columns in the faceting selects, while the short-term solution is to disable ""Hide this column"" for certain things.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851115805,https://api.github.com/repos/simonw/datasette/issues/1345,851115805,MDEyOklzc3VlQ29tbWVudDg1MTExNTgwNQ==,9599,simonw,2021-05-31T01:59:39Z,2021-05-31T01:59:39Z,OWNER,"Maybe there's a concept here of the columns that are required by a selected facet? Those can then be included as `data-` attributes on the page, which will then impact which ""Hide this column"" options are available.

I can also use them to provide a better error message than ""no such column: state"" - I can verify that `?_nocol` and `?_col` have not been used to disable the required columns.

There is one other option here: I could still include the columns that are known to be needed for faceting in the faceting SQL queries, but leave them out of the query that is used to return the results! That's actually a pretty tempting (albeit more complex) option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851114806,https://api.github.com/repos/simonw/datasette/issues/1345,851114806,MDEyOklzc3VlQ29tbWVudDg1MTExNDgwNg==,9599,simonw,2021-05-31T01:55:56Z,2021-05-31T01:55:56Z,OWNER,"Two options here:

- Don't provide users with options that will lead to this situation - so no ""Hide this column"" option on pages that are already faceted by that column
- Ignore facet selections for columns which are no longer visible

I think I like the first option more.

I could partially implement that in the `table.js` JavaScript by looking at the `?_facet=` parameters... but that won't cover the case where the facet is happening because of default facets configured in `metadata.yml`.

Instead the JavaScript should look for evidence in the DOM that specific facets are enabled. This could also help me cover other types of faceting, such as `?_facet_array=` or even custom facets provided by plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851113886,https://api.github.com/repos/simonw/datasette/issues/1345,851113886,MDEyOklzc3VlQ29tbWVudDg1MTExMzg4Ng==,9599,simonw,2021-05-31T01:52:27Z,2021-05-31T01:52:27Z,OWNER,"Related issue: visit https://latest.datasette.io/fixtures/facetable?_facet=state and click ""Hide this column"" on the ""state"" cog menu and you get https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state which shows an error:

> ### Invalid SQL
> no such column: state
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778311,https://api.github.com/repos/simonw/datasette/issues/1349,850778311,MDEyOklzc3VlQ29tbWVudDg1MDc3ODMxMQ==,9599,simonw,2021-05-29T06:12:21Z,2021-05-29T06:12:21Z,OWNER,"It's not just facets, I think it's trying to execute suggested facets too!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778273,https://api.github.com/repos/simonw/datasette/issues/1349,850778273,MDEyOklzc3VlQ29tbWVudDg1MDc3ODI3Mw==,9599,simonw,2021-05-29T06:11:59Z,2021-05-29T06:11:59Z,OWNER,"Related issue: https://github.com/simonw/datasette/issues/263 - ""Facets should not execute for ?shape=array|object""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778230,https://api.github.com/repos/simonw/datasette/issues/1349,850778230,MDEyOklzc3VlQ29tbWVudDg1MDc3ODIzMA==,9599,simonw,2021-05-29T06:11:28Z,2021-05-29T06:11:28Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L315,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850772428,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850772428,MDEyOklzc3VlQ29tbWVudDg1MDc3MjQyOA==,9599,simonw,2021-05-29T05:16:02Z,2021-05-29T05:28:57Z,OWNER,"I needed to find some CSV files on my computer with a BOM at the beginning - I figured out this recipe:
```
% rg -U -E none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .
```
TIL here: https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771264,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850771264,MDEyOklzc3VlQ29tbWVudDg1MDc3MTI2NA==,9599,simonw,2021-05-29T05:06:13Z,2021-05-29T05:06:13Z,OWNER,The other option is to check if the file starts with `codecs.BOM_UTF8` - which is `b'\xef\xbb\xbf'`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771017,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850771017,MDEyOklzc3VlQ29tbWVudDg1MDc3MTAxNw==,9599,simonw,2021-05-29T05:04:28Z,2021-05-29T05:04:28Z,OWNER,"https://stackoverflow.com/a/44573867/6083 says:

> There is no reason to check if a BOM exists or not, utf-8-sig manages that for you and behaves exactly as utf-8 if the BOM does not exist

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850769067,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850769067,MDEyOklzc3VlQ29tbWVudDg1MDc2OTA2Nw==,9599,simonw,2021-05-29T04:48:10Z,2021-05-29T04:48:10Z,OWNER,"I confirmed and it's possible to have a SQLite column with a hyphen at the start, confirmed using:
```
% sqlite-utils create-table demo.db demo -- id integer name text -blah integer
% sqlite-utils tables --schema demo.db -t
table    schema
-------  ---------------------
demo     CREATE TABLE [demo] (
            [id] INTEGER,
            [name] TEXT,
            [-blah] INTEGER
         )
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850768315,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850768315,MDEyOklzc3VlQ29tbWVudDg1MDc2ODMxNQ==,9599,simonw,2021-05-29T04:39:33Z,2021-05-29T04:39:33Z,OWNER,"This doesn't work:
```
sqlite-utils create-index mydb.db dogs ""-age"" name
```
But this does:
```
sqlite-utils create-index mydb.db dogs -- -age name
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850767210,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850767210,MDEyOklzc3VlQ29tbWVudDg1MDc2NzIxMA==,9599,simonw,2021-05-29T04:26:26Z,2021-05-29T04:28:31Z,OWNER,It's weird having to use `Database.DescIndex` - I'm going to put `DescIndex` in `sqlite_utils.db` directly and let people import it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766552,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850766552,MDEyOklzc3VlQ29tbWVudDg1MDc2NjU1Mg==,9599,simonw,2021-05-29T04:20:40Z,2021-05-29T04:24:01Z,OWNER,"`PRAGMA index_xinfo(table)` DOES return that data:
```
(Pdb) [c[0] for c in fresh_db.execute(""PRAGMA index_xinfo('idx_dogs_age_name')"").description]
['seqno', 'cid', 'name', 'desc', 'coll', 'key']
(Pdb) fresh_db.execute(""PRAGMA index_xinfo('idx_dogs_age_name')"").fetchall()
[(0, 2, 'age', 1, 'BINARY', 1), (1, 0, 'name', 0, 'BINARY', 1), (2, -1, None, 0, 'BINARY', 0)]
```
See https://sqlite.org/pragma.html#pragma_index_xinfo

Example output: https://covid-19.datasettes.com/covid?sql=select+*+from+pragma_index_xinfo%28%27idx_ny_times_us_counties_date%27%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766335,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850766335,MDEyOklzc3VlQ29tbWVudDg1MDc2NjMzNQ==,9599,simonw,2021-05-29T04:18:19Z,2021-05-29T04:18:19Z,OWNER,"Annoyingly the `table.indexes` property won't indicate if an index is in regular or reverse order - because the SQLite `PRAGMA index_info(table)` statement doesn't indicate that either. You have to look at the `sqlite_master` index definition to tell if any of the columns are in reverse order:
```
(Pdb) fresh_db.execute(""select * from sqlite_master where type = 'index'"").fetchall()
[('index', 'idx_dogs_age_name', 'dogs', 3, 'CREATE INDEX [idx_dogs_age_name]\n    ON [dogs] ([age] desc, [name])')]
(Pdb) fresh_db.execute(""PRAGMA index_info('idx_dogs_age_name')"").fetchall()
[(0, 2, 'age'), (1, 0, 'name')]
(Pdb) fresh_db.execute(""PRAGMA index_info('idx_dogs_age_name')"").description
(('seqno', None, None, None, None, None, None), ('cid', None, None, None, None, None, None), ('name', None, None, None, None, None, None))
(Pdb) dogs.indexes
[Index(seq=0, name='idx_dogs_age_name', unique=0, origin='c', partial=0, columns=['age', 'name'])]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765450,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765450,MDEyOklzc3VlQ29tbWVudDg1MDc2NTQ1MA==,9599,simonw,2021-05-29T04:09:13Z,2021-05-29T04:09:13Z,OWNER,"Decisions: for the Python API I'm going with `db.DescIndex(""column"")` as the way to do this.

For the CLI I'm going to do the ""-age"" thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765291,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765291,MDEyOklzc3VlQ29tbWVudDg1MDc2NTI5MQ==,9599,simonw,2021-05-29T04:07:48Z,2021-05-29T04:08:21Z,OWNER,"For the CLI version I could say that you can use a `-` prefix to specify reverse direction:
```sh
sqlite-utils create-index mydb.db dogs -age name
```
No, that doesn't work - it could get confused with a command-line flag. I guess you could do this:
```
sqlite-utils create-index mydb.db dogs ""-age"" name
```
This does mean that if any of your column names begin with a hyphen you can't use the CLI to add indexes to them. Is that an acceptable limitation? Users can always use `sqlite-utils mydb.db ""create index ...""` in that case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765050,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765050,MDEyOklzc3VlQ29tbWVudDg1MDc2NTA1MA==,9599,simonw,2021-05-29T04:05:24Z,2021-05-29T04:05:40Z,OWNER,"Need to solve this for the CLI tool too. Currently that works like this: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes
```sh
sqlite-utils create-index mydb.db mytable col1 [col2...]
```
Even harder to decide how to add a descending option to this. Maybe like this?
```sh
sqlite-utils create-index mydb.db mytable --direction col1 asc --direction col2 desc
```
It's a bit gross though! We're saying here that if a single one of the columns you are creating an index for is in reverse direction you have to use `--direction` to specify each end every other index.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764700,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764700,MDEyOklzc3VlQ29tbWVudDg1MDc2NDcwMA==,9599,simonw,2021-05-29T04:02:10Z,2021-05-29T04:02:10Z,OWNER,"I could use `db.desc_index(""age"")` to match SQLite SQL syntax, which uses `desc` and not `descending`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764655,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764655,MDEyOklzc3VlQ29tbWVudDg1MDc2NDY1NQ==,9599,simonw,2021-05-29T04:01:41Z,2021-05-29T04:01:41Z,OWNER,"Maybe:
```python
db[""dogs""].create_index([db.descending_index(""age""), ""name""])
```
It's a little verbose but it's for a relatively rare activity and it does make it very clear what is going on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764594,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764594,MDEyOklzc3VlQ29tbWVudDg1MDc2NDU5NA==,9599,simonw,2021-05-29T04:00:54Z,2021-05-29T04:00:54Z,OWNER,"A few options:
```python
db[""dogs""].create_index([(""age"", ""desc""), ""name""])

db[""dogs""].create_index([desc(""age""), ""name""])

db[""dogs""].create_index([db.desc(""age""), ""name""])
```
The first option uses an optional tuple. The second two use a `desc()` function - the question is where should that live? 

`sqlite_utils.desc(column)` or `db.desc(column)`  are both options.

I don't like using the term `desc()` for ""descending index"" though - it feels like it should mean something more broad.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764253,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764253,MDEyOklzc3VlQ29tbWVudDg1MDc2NDI1Mw==,9599,simonw,2021-05-29T03:57:54Z,2021-05-29T03:57:54Z,OWNER,"The problem here is differentiating between a column with the name `date desc` and wanting to create a descending index on a column called `date`.

This won't work:
```python
db[""ny_times_us_counties""].create_index([""date desc""], desc=True)
```
Because we need to be able to create compound indexes with columns with different directions - for example:
```sql
create index idx_age_desc_name on dogs (age desc, name)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/datasette/pull/1296#issuecomment-850583584,https://api.github.com/repos/simonw/datasette/issues/1296,850583584,MDEyOklzc3VlQ29tbWVudDg1MDU4MzU4NA==,9599,simonw,2021-05-28T18:06:11Z,2021-05-28T18:06:11Z,OWNER,"> As a bonus, the Docker image becomes smaller

That's a huge surprise to me! And most welcome.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855446829,Dockerfile: use Ubuntu 20.10 as base,
https://github.com/simonw/datasette/pull/1346#issuecomment-850059328,https://api.github.com/repos/simonw/datasette/issues/1346,850059328,MDEyOklzc3VlQ29tbWVudDg1MDA1OTMyOA==,9599,simonw,2021-05-28T02:08:10Z,2021-05-28T02:09:25Z,OWNER,"This is the current test failure, but it actually another problem that we don't have tests in place for errors with different formats and shapes:

```
    def test_magic_parameters_cannot_be_used_in_arbitrary_queries(magic_parameters_client):
        response = magic_parameters_client.get(
            ""/data.json?sql=select+:_header_host&_shape=array""
        )
        assert 400 == response.status
>       assert ""You did not supply a value for binding 1."" == response.json[""error""]
E       TypeError: list indices must be integers or slices, not str
```

The test fails because `response.json` here is the empty list `[]`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904537568,Re-display user's query with an error message if an error occurs,
https://github.com/simonw/datasette/issues/619#issuecomment-850059607,https://api.github.com/repos/simonw/datasette/issues/619,850059607,MDEyOklzc3VlQ29tbWVudDg1MDA1OTYwNw==,9599,simonw,2021-05-28T02:08:58Z,2021-05-28T02:08:58Z,OWNER,Remaining work will happen in #1346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/pull/1346#issuecomment-850058851,https://api.github.com/repos/simonw/datasette/issues/1346,850058851,MDEyOklzc3VlQ29tbWVudDg1MDA1ODg1MQ==,9599,simonw,2021-05-28T02:06:46Z,2021-05-28T02:06:46Z,OWNER,"The one test failure here actually illustrates a larger problem: if the user specifies `.json?_shape=array` but an error occurs, what should we do?

Prior to this change we return the following JSON:
```json
{
    ""ok"": false,
    ""error"": ""You did not supply a value for binding 1."",
    ""status"": 500,
    ""title"": null
}
```
But this comes from the `handle_500` higher level code here: https://github.com/simonw/datasette/blob/eae3084b46e2c3931db12cdef79093ad0e644bce/datasette/app.py#L1251-L1263","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904537568,Re-display user's query with an error message if an error occurs,
https://github.com/simonw/datasette/issues/619#issuecomment-850057694,https://api.github.com/repos/simonw/datasette/issues/619,850057694,MDEyOklzc3VlQ29tbWVudDg1MDA1NzY5NA==,9599,simonw,2021-05-28T02:03:05Z,2021-05-28T02:03:05Z,OWNER,"I nearly got this working, but I ran into one last problem: the code path for when an error is raised but the user specified `?_shape=array`. I'll open a draft PR with where I've got to so far.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-850025106,https://api.github.com/repos/simonw/datasette/issues/619,850025106,MDEyOklzc3VlQ29tbWVudDg1MDAyNTEwNg==,9599,simonw,2021-05-28T00:18:30Z,2021-05-28T00:18:30Z,OWNER,"I'm going to return a 400 HTTP status code for ""bad request"", under the assumption that the client sent bad SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-850019486,https://api.github.com/repos/simonw/datasette/issues/619,850019486,MDEyOklzc3VlQ29tbWVudDg1MDAxOTQ4Ng==,9599,simonw,2021-05-28T00:05:50Z,2021-05-28T00:05:50Z,OWNER,Came up on discussions here: https://github.com/simonw/datasette/discussions/1334,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/1344#issuecomment-849820019,https://api.github.com/repos/simonw/datasette/issues/1344,849820019,MDEyOklzc3VlQ29tbWVudDg0OTgyMDAxOQ==,9599,simonw,2021-05-27T17:44:39Z,2021-05-27T17:52:58Z,OWNER,"This pattern appears to work, executed at the root of a checkout of `datasette`:
```
docker run -it -v `pwd`:/mnt --platform linux/amd64 \
  datasetteproject/datasette:0.57a1 bash -c '
    pip install ""pytest>=5.2.2,<6.3.0"" \
      ""pytest-xdist>=2.2.1,<2.3"" \
      ""pytest-asyncio>=0.10,<0.16"" \
      ""beautifulsoup4>=4.8.1,<4.10.0"" \
      ""black==21.5b1"" \
      ""pytest-timeout>=1.4.2,<1.5"" \
      ""trustme>=0.7,<0.8"" \
    && cd /mnt && pytest'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849815361,https://api.github.com/repos/simonw/datasette/issues/1344,849815361,MDEyOklzc3VlQ29tbWVudDg0OTgxNTM2MQ==,9599,simonw,2021-05-27T17:36:51Z,2021-05-27T17:36:51Z,OWNER,"To run the test suite... need to install a few more dependencies: https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/setup.py#L71-L79

And then mount the `tests/` folder into the container and run `pytest`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849806581,https://api.github.com/repos/simonw/datasette/issues/1344,849806581,MDEyOklzc3VlQ29tbWVudDg0OTgwNjU4MQ==,9599,simonw,2021-05-27T17:22:37Z,2021-05-27T17:34:27Z,OWNER,"```
~ % docker run -it --platform linux/arm/v7 -v `pwd`:/mnt -p 8001:8001 datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
Unable to find image 'datasetteproject/datasette:0.57a1' locally
0.57a1: Pulling from datasetteproject/datasette
ee0a2cc24f29: Pull complete 
ca231153300d: Pull complete 
6073fc53d406: Pull complete 
bb2a6a6421dd: Pull complete 
14c15f441034: Pull complete 
28d113f72ba5: Pull complete 
40516c2c3785: Pull complete 
Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
Status: Downloaded newer image for datasetteproject/datasette:0.57a1
uname_result(system='Linux', node='c2e1a05732ed', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='armv7l')
```
Shorter version:
```
~ % docker run -it --platform linux/ppc64le datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
Unable to find image 'datasetteproject/datasette:0.57a1' locally
0.57a1: Pulling from datasetteproject/datasette
c840eb5e9aed: Pull complete 
fc17d8b577ec: Pull complete 
712a0ad995f8: Pull complete 
56101149914c: Pull complete 
e6241dfe9d38: Pull complete 
41ed85039888: Pull complete 
26beeabe9213: Pull complete 
Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
Status: Downloaded newer image for datasetteproject/datasette:0.57a1
uname_result(system='Linux', node='0fe85887fa30', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='ppc64le')
~ % 
```

OK! Now I need to figure out how to run the full test suite on each one, then I can call this done.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849805778,https://api.github.com/repos/simonw/datasette/issues/1344,849805778,MDEyOklzc3VlQ29tbWVudDg0OTgwNTc3OA==,9599,simonw,2021-05-27T17:21:11Z,2021-05-27T17:21:22Z,OWNER,"This helps:
```
~ % docker run -it --platform linux/s390x -v `pwd`:/mnt \
  -p 8001:8001 datasetteproject/datasette:0.57a1 \
  python -c 'import platform; print(platform.uname())'
```
`uname_result(system='Linux', node='d14916ca91df', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='s390x')`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849804684,https://api.github.com/repos/simonw/datasette/issues/1344,849804684,MDEyOklzc3VlQ29tbWVudDg0OTgwNDY4NA==,9599,simonw,2021-05-27T17:19:16Z,2021-05-27T17:19:16Z,OWNER,"Having pushed that to Docker Hub the following _seems_ to work on my laptop:

    docker run --platform linux/s390x \
      -v `pwd`:/mnt -p 8001:8001 \
      datasetteproject/datasette:0.57a1 datasette -p 8001 -h 0.0.0.0

I'd like to confirm that this is correctly emulating the architecture and running the correct published image though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849801969,https://api.github.com/repos/simonw/datasette/issues/1344,849801969,MDEyOklzc3VlQ29tbWVudDg0OTgwMTk2OQ==,9599,simonw,2021-05-27T17:14:49Z,2021-05-27T17:14:49Z,OWNER,"Job completed in 13m38s - results are here: https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated&name=0.57a1

<img width=""1286"" alt=""datasetteproject_datasette_Tags"" src=""https://user-images.githubusercontent.com/9599/119868791-5c8caf00-bed4-11eb-9e9f-ebca557621c0.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849797767,https://api.github.com/repos/simonw/datasette/issues/1344,849797767,MDEyOklzc3VlQ29tbWVudDg0OTc5Nzc2Nw==,9599,simonw,2021-05-27T17:08:13Z,2021-05-27T17:08:13Z,OWNER,Related issue: #1272 - being able to run the test suite inside Docker inside GitHub Actions would be incredibly helpful here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849793312,https://api.github.com/repos/simonw/datasette/issues/1344,849793312,MDEyOklzc3VlQ29tbWVudDg0OTc5MzMxMg==,9599,simonw,2021-05-27T17:01:16Z,2021-05-27T17:01:16Z,OWNER,https://github.com/simonw/datasette/runs/2687196350 is a run of https://github.com/simonw/datasette/blob/main/.github/workflows/push_docker_tag.yml to publish the new `0.57a1` alpha to Docker Hub.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849788412,https://api.github.com/repos/simonw/datasette/issues/1344,849788412,MDEyOklzc3VlQ29tbWVudDg0OTc4ODQxMg==,9599,simonw,2021-05-27T16:53:28Z,2021-05-27T16:53:28Z,OWNER,(Should also update https://docs.datasette.io/en/stable/contributing.html#release-process with notes on how this works),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849788159,https://api.github.com/repos/simonw/datasette/issues/1344,849788159,MDEyOklzc3VlQ29tbWVudDg0OTc4ODE1OQ==,9599,simonw,2021-05-27T16:53:04Z,2021-05-27T16:53:04Z,OWNER,I'm going to release a `0.57a1` alpha to Docker Hub using the new buildx mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/pull/1319#issuecomment-849785098,https://api.github.com/repos/simonw/datasette/issues/1319,849785098,MDEyOklzc3VlQ29tbWVudDg0OTc4NTA5OA==,9599,simonw,2021-05-27T16:48:21Z,2021-05-27T16:48:21Z,OWNER,"OK, since I have a mechanism for deploying alpha versions now I'm going to merge this and use it to publish 0.57a1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/issues/1343#issuecomment-849784005,https://api.github.com/repos/simonw/datasette/issues/1343,849784005,MDEyOklzc3VlQ29tbWVudDg0OTc4NDAwNQ==,9599,simonw,2021-05-27T16:46:37Z,2021-05-27T16:46:37Z,OWNER,"That worked:
```
 % docker run -it datasetteproject/datasette:0.57a0
Unable to find image 'datasetteproject/datasette:0.57a0' locally
0.57a0: Pulling from datasetteproject/datasette
75646c2fb410: Pull complete 
edf8b70d3cc4: Pull complete 
229daa3ebd94: Pull complete 
65d21ac50a56: Pull complete 
e083ad7cc3ca: Pull complete 
ed4de1431432: Pull complete 
1da5e8b27cca: Pull complete 
Digest: sha256:4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db
Status: Downloaded newer image for datasetteproject/datasette:0.57a0
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849783398,https://api.github.com/repos/simonw/datasette/issues/1343,849783398,MDEyOklzc3VlQ29tbWVudDg0OTc4MzM5OA==,9599,simonw,2021-05-27T16:45:41Z,2021-05-27T16:45:41Z,OWNER,https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now shows this at the top: https://hub.docker.com/layers/datasetteproject/datasette/0.57a0/images/sha256-4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db?context=explore,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849782098,https://api.github.com/repos/simonw/datasette/issues/1343,849782098,MDEyOklzc3VlQ29tbWVudDg0OTc4MjA5OA==,9599,simonw,2021-05-27T16:43:40Z,2021-05-27T16:43:55Z,OWNER,"https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml

<img width=""944"" alt=""Actions_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/119864935-01f15400-bed0-11eb-8979-0a5cc1b6bcb4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849781747,https://api.github.com/repos/simonw/datasette/issues/1343,849781747,MDEyOklzc3VlQ29tbWVudDg0OTc4MTc0Nw==,9599,simonw,2021-05-27T16:43:04Z,2021-05-27T16:43:04Z,OWNER,I'm going to try pushing an alpha of https://github.com/simonw/datasette/releases/tag/0.57a0 using this workflow I built a few months ago: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/push_docker_tag.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/pull/1319#issuecomment-849780481,https://api.github.com/repos/simonw/datasette/issues/1319,849780481,MDEyOklzc3VlQ29tbWVudDg0OTc4MDQ4MQ==,9599,simonw,2021-05-27T16:41:03Z,2021-05-27T16:41:03Z,OWNER,It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the `latest` tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/pull/1319#issuecomment-849778373,https://api.github.com/repos/simonw/datasette/issues/1319,849778373,MDEyOklzc3VlQ29tbWVudDg0OTc3ODM3Mw==,9599,simonw,2021-05-27T16:37:50Z,2021-05-27T16:37:50Z,OWNER,Started [a conversation](https://twitter.com/simonw/status/1397953338407223303) about this on Twitter. Ideally I'd like to push an alpha release to Docker Hub so people can start testing these out before they make it into an official non-alpha release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/issues/1337#issuecomment-849762735,https://api.github.com/repos/simonw/datasette/issues/1337,849762735,MDEyOklzc3VlQ29tbWVudDg0OTc2MjczNQ==,9599,simonw,2021-05-27T16:14:14Z,2021-05-27T16:14:14Z,OWNER,"Here's a demo showing that once you get above 1,000 results you still get the ""..."" but it no longer links to anything: https://fivethirtyeight.datasettes.com/fivethirtyeight/births%2FUS_births_2000-2014_SSA?_facet=births&_facet_size=max","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1332#issuecomment-849761894,https://api.github.com/repos/simonw/datasette/issues/1332,849761894,MDEyOklzc3VlQ29tbWVudDg0OTc2MTg5NA==,9599,simonw,2021-05-27T16:13:07Z,2021-05-27T16:13:07Z,OWNER,Added `?_facet_size=max` in #1337 and made the `...` truncated note link to that - here's a demo: https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act?_facet=states&_facet=current_agency&_facet=pres_or_congress,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1337#issuecomment-849761508,https://api.github.com/repos/simonw/datasette/issues/1337,849761508,MDEyOklzc3VlQ29tbWVudDg0OTc2MTUwOA==,9599,simonw,2021-05-27T16:12:31Z,2021-05-27T16:12:31Z,OWNER,More exciting demo: https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act?_facet=states&_facet=current_agency&_facet=pres_or_congress,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1337#issuecomment-849759440,https://api.github.com/repos/simonw/datasette/issues/1337,849759440,MDEyOklzc3VlQ29tbWVudDg0OTc1OTQ0MA==,9599,simonw,2021-05-27T16:09:34Z,2021-05-27T16:09:34Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet_size=2&_facet=state&_facet=city_id

![demo-max](https://user-images.githubusercontent.com/9599/119860507-3f071780-becb-11eb-8698-64c8c6112983.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1337#issuecomment-849725291,https://api.github.com/repos/simonw/datasette/issues/1337,849725291,MDEyOklzc3VlQ29tbWVudDg0OTcyNTI5MQ==,9599,simonw,2021-05-27T15:23:33Z,2021-05-27T15:23:33Z,OWNER,Turns out `path_with_replaced_args()` is already exposed to the `table.html` template context: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1342#issuecomment-849724915,https://api.github.com/repos/simonw/datasette/issues/1342,849724915,MDEyOklzc3VlQ29tbWVudDg0OTcyNDkxNQ==,9599,simonw,2021-05-27T15:23:01Z,2021-05-27T15:23:01Z,OWNER,Since they are already exposed (but undocumented) this isn't actually a blocker on #1337 like I thought it was. Should still clean this up in the future though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,
https://github.com/simonw/datasette/issues/1342#issuecomment-849724108,https://api.github.com/repos/simonw/datasette/issues/1342,849724108,MDEyOklzc3VlQ29tbWVudDg0OTcyNDEwOA==,9599,simonw,2021-05-27T15:21:54Z,2021-05-27T15:21:54Z,OWNER,"Turns out some of these are already being exposed to the template context on an ad-hoc basis.

In `QueryView`: https://github.com/simonw/datasette/blob/2bd9d54b2762c991e11950c22c88c0336158d49b/datasette/views/database.py#L295-L304

In `TableView`: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,
https://github.com/simonw/datasette/issues/1342#issuecomment-849722218,https://api.github.com/repos/simonw/datasette/issues/1342,849722218,MDEyOklzc3VlQ29tbWVudDg0OTcyMjIxOA==,9599,simonw,2021-05-27T15:19:23Z,2021-05-27T15:19:23Z,OWNER,Current implementations: https://github.com/simonw/datasette/blob/51d788114035458d8f860d9ea6d74078e6c0ea0d/datasette/utils/__init__.py#L214-L272,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,
https://github.com/simonw/datasette/issues/1337#issuecomment-849721280,https://api.github.com/repos/simonw/datasette/issues/1337,849721280,MDEyOklzc3VlQ29tbWVudDg0OTcyMTI4MA==,9599,simonw,2021-05-27T15:18:08Z,2021-05-27T15:18:08Z,OWNER,In order to cleanly implement this I need to expose the `path_with_replaced_args` utility function to Datasette's template engine. This is the first time this will become an exposed (and hence should-by-documented) API and I don't like its shape much.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1337#issuecomment-849695117,https://api.github.com/repos/simonw/datasette/issues/1337,849695117,MDEyOklzc3VlQ29tbWVudDg0OTY5NTExNw==,9599,simonw,2021-05-27T14:44:13Z,2021-05-27T14:44:13Z,OWNER,"Here's a much easier way to implement this: implement `?_facet_size=max` (sets it to `max_returned_rows`, default 1000) and then link to that from the ellipsis.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/pull/1319#issuecomment-849311884,https://api.github.com/repos/simonw/datasette/issues/1319,849311884,MDEyOklzc3VlQ29tbWVudDg0OTMxMTg4NA==,9599,simonw,2021-05-27T04:33:51Z,2021-05-27T04:33:51Z,OWNER,This is so cool! I wonder how I can best test it? I have a Raspberry Pi I should dig out to try this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/issues/1341#issuecomment-849310294,https://api.github.com/repos/simonw/datasette/issues/1341,849310294,MDEyOklzc3VlQ29tbWVudDg0OTMxMDI5NA==,9599,simonw,2021-05-27T04:29:04Z,2021-05-27T04:29:04Z,OWNER,For implementation simplicity I'm going to show the item even if all of the columns are visible because `?_col=` has selected all of them - selecting it will still reset the view to having no `?_col=` or `?_nocol=` parameters.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903200328,"""Show all columns"" cog menu item should show if ?_col= is used",
https://github.com/simonw/datasette/issues/615#issuecomment-849309330,https://api.github.com/repos/simonw/datasette/issues/615,849309330,MDEyOklzc3VlQ29tbWVudDg0OTMwOTMzMA==,9599,simonw,2021-05-27T04:26:10Z,2021-05-27T04:26:10Z,OWNER,"Demos:

- https://latest.datasette.io/fixtures/sortable?_nocol=sortable
- https://latest.datasette.io/fixtures/sortable?_col=sortable&_col=text

Also try the column cog menu on that page.

Documentation: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/292#issuecomment-849308907,https://api.github.com/repos/simonw/datasette/issues/292,849308907,MDEyOklzc3VlQ29tbWVudDg0OTMwODkwNw==,9599,simonw,2021-05-27T04:25:01Z,2021-05-27T04:25:01Z,OWNER,Now that `?_col=` and `?_nocol=` are implemented I'm closing this ticket - other customizations can already be handled by defining SQL views or creating canned SQL queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",326800219,Mechanism for customizing the SQL used to select specific columns in the table view,
https://github.com/simonw/datasette/issues/670#issuecomment-849022714,https://api.github.com/repos/simonw/datasette/issues/670,849022714,MDEyOklzc3VlQ29tbWVudDg0OTAyMjcxNA==,9599,simonw,2021-05-26T18:33:47Z,2021-05-26T18:33:58Z,OWNER,Worth mentioning here: I've been doing a tun of research around running Datasette-like functionality against PostgreSQL in my https://github.com/simonw/django-sql-dashboard project - which will definitely inform the Datasette implementation.,"{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL,
https://github.com/simonw/datasette/issues/1340#issuecomment-847987346,https://api.github.com/repos/simonw/datasette/issues/1340,847987346,MDEyOklzc3VlQ29tbWVudDg0Nzk4NzM0Ng==,9599,simonw,2021-05-25T15:49:55Z,2021-05-25T15:49:55Z,OWNER,"Usual problem with this kind of UI: what would it do on mobile where hover isn't a thing? Plenty of other questions: if it opens a menu, what options might I put in there other than ""Show more like this""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",901009787,Research: Cell action menu (like column action but for individual cells),
https://github.com/simonw/datasette/issues/1327#issuecomment-847240470,https://api.github.com/repos/simonw/datasette/issues/1327,847240470,MDEyOklzc3VlQ29tbWVudDg0NzI0MDQ3MA==,9599,simonw,2021-05-24T18:21:44Z,2021-05-24T18:21:44Z,OWNER,"I'm having trouble replicating this - I tried the following `metadata.json` in my local environment and it displayed correctly:
```json
{
    ""title"": ""မြန်မာစာ (""
}
```
What operating system are you using here? Is there a chance that your `metadata.json` file is saved in a text encoding other than utf-8?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",892457208,Support Unicode characters in metadata.json,
https://github.com/simonw/datasette/issues/1320#issuecomment-847237524,https://api.github.com/repos/simonw/datasette/issues/1320,847237524,MDEyOklzc3VlQ29tbWVudDg0NzIzNzUyNA==,9599,simonw,2021-05-24T18:15:56Z,2021-05-24T18:15:56Z,OWNER,Added some new documentation about that here: https://github.com/simonw/datasette/blob/c0a748e5c3f498fa8c139b420d07dd3dea612379/docs/installation.rst#installing-plugins,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",884952179,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,
https://github.com/simonw/datasette/issues/1320#issuecomment-847232547,https://api.github.com/repos/simonw/datasette/issues/1320,847232547,MDEyOklzc3VlQ29tbWVudDg0NzIzMjU0Nw==,9599,simonw,2021-05-24T18:06:32Z,2021-05-24T18:06:32Z,OWNER,"I'm testing tweaks to the Dockerfile by editing it and running:
```
docker build . -t datasette-057a0 --build-arg VERSION=0.57a0
```
Then running this to inspect the new image:
```
docker run -it -p 8001:8001 datasette-057a0 bash
```
After changing that line to `rm -rf /var/lib/dpkg/info/*` I can run `apt-get` successfully from inside that bash prompt:
```
# apt-get update
# apt-get install ripgrep
# rg --version
ripgrep 12.1.1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",884952179,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,
https://github.com/simonw/datasette/issues/1320#issuecomment-847230866,https://api.github.com/repos/simonw/datasette/issues/1320,847230866,MDEyOklzc3VlQ29tbWVudDg0NzIzMDg2Ng==,9599,simonw,2021-05-24T18:03:29Z,2021-05-24T18:03:29Z,OWNER,"```
root@33c269900821:/# du -h /var/lib/dpkg
6.3M	/var/lib/dpkg/info
24K	/var/lib/dpkg/triggers
4.0K	/var/lib/dpkg/updates
20K	/var/lib/dpkg/alternatives
4.0K	/var/lib/dpkg/parts
6.7M	/var/lib/dpkg
```
So `/var/lib/dpkg/info` is most of the space, I'll try just deleting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",884952179,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,
https://github.com/simonw/datasette/issues/1320#issuecomment-847223853,https://api.github.com/repos/simonw/datasette/issues/1320,847223853,MDEyOklzc3VlQ29tbWVudDg0NzIyMzg1Mw==,9599,simonw,2021-05-24T17:51:12Z,2021-05-24T17:51:12Z,OWNER,"This is a good catch, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",884952179,Can't use apt-get in Dockerfile when using datasetteproj/datasette as base,
https://github.com/simonw/datasette/issues/615#issuecomment-846592583,https://api.github.com/repos/simonw/datasette/issues/615,846592583,MDEyOklzc3VlQ29tbWVudDg0NjU5MjU4Mw==,9599,simonw,2021-05-23T16:50:45Z,2021-05-24T17:15:48Z,OWNER,"Still needed:

- Unit tests
- Documentation

And maybe a UI mechanism for this? I could at least add a ""remove this column"" item to the cog menu on columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846668419,https://api.github.com/repos/simonw/datasette/issues/615,846668419,MDEyOklzc3VlQ29tbWVudDg0NjY2ODQxOQ==,9599,simonw,2021-05-24T01:14:27Z,2021-05-24T01:14:27Z,OWNER,"I tried allowing the removal of `rowid` and got this exception:
```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 831, in extra_template
    display_columns, display_rows = await self.display_columns_and_rows(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 163, in display_columns_and_rows
    pk_path = path_from_row_pks(row, pks, not pks, False)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 82, in path_from_row_pks
    bits = [row[""rowid""]]
IndexError: No item with that key
```
I'm going to disable the removal of `rowid` - or indeed any of the primary keys, since they are needed to construct the row permalink.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846660103,https://api.github.com/repos/simonw/datasette/issues/615,846660103,MDEyOklzc3VlQ29tbWVudDg0NjY2MDEwMw==,9599,simonw,2021-05-24T00:47:00Z,2021-05-24T00:47:00Z,OWNER,"> Here's a bug: removing the `rowid` column returns an error.

Removing the `rowid` column should work. We can continue to show the `Link` column, ensuring users can still navigate to the row page for each row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846659272,https://api.github.com/repos/simonw/datasette/issues/615,846659272,MDEyOklzc3VlQ29tbWVudDg0NjY1OTI3Mg==,9599,simonw,2021-05-24T00:44:47Z,2021-05-24T00:44:47Z,OWNER,I can't think of any good reasons to support passing the same column twice `?col=airline&_col=airline` so I'm going to de-duplicate and silently ignore the second one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846627322,https://api.github.com/repos/simonw/datasette/issues/615,846627322,MDEyOklzc3VlQ29tbWVudDg0NjYyNzMyMg==,9599,simonw,2021-05-23T21:31:22Z,2021-05-23T21:31:22Z,OWNER,"Would it be useful to allow this mechanism to alias columns, for example supporting one of the following:

- `?_col=airline as name_of_airline`
- `?_col=airline:name_of_airline`

This could be handy for renaming columns to match a specific expected JSON output.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846627163,https://api.github.com/repos/simonw/datasette/issues/615,846627163,MDEyOklzc3VlQ29tbWVudDg0NjYyNzE2Mw==,9599,simonw,2021-05-23T21:30:01Z,2021-05-23T21:30:01Z,OWNER,"Interesting side-effect of this implementation is that you can both control column order and request the same column multiple times:

`/fivethirtyeight/airline-safety%2Fairline-safety?_col=fatal_accidents_00_14&_col=fatalities_00_14&_col=airline&_col=airline`

<img width=""930"" alt=""fivethirtyeight__airline-safety_airline-safety__56_rows"" src=""https://user-images.githubusercontent.com/9599/119277243-58f6e080-bbd3-11eb-98be-69787cd9192d.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846626871,https://api.github.com/repos/simonw/datasette/issues/615,846626871,MDEyOklzc3VlQ29tbWVudDg0NjYyNjg3MQ==,9599,simonw,2021-05-23T21:27:36Z,2021-05-23T21:27:36Z,OWNER,A better interface for this would be a full list of columns each with a checkbox for making it visible on invisible - this could then be used to apply a bulk change (rather than refreshing the interface after every removed column) and it could also be easily designed to work on narrow mobile screens where the cog icon is not visible.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846626567,https://api.github.com/repos/simonw/datasette/issues/615,846626567,MDEyOklzc3VlQ29tbWVudDg0NjYyNjU2Nw==,9599,simonw,2021-05-23T21:25:19Z,2021-05-23T21:25:19Z,OWNER,"Here's a bug: removing the `rowid` column returns an error.

![rowid-bug](https://user-images.githubusercontent.com/9599/119277132-b179ae00-bbd2-11eb-8f2f-e228c7d7c264.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846599732,https://api.github.com/repos/simonw/datasette/issues/615,846599732,MDEyOklzc3VlQ29tbWVudDg0NjU5OTczMg==,9599,simonw,2021-05-23T17:46:45Z,2021-05-23T17:46:45Z,OWNER,I've changed my mind about forbidding `?_col=` and `?_nocol=` from being applied at the same time - I'm going to come up with a simple resolution rule instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846595473,https://api.github.com/repos/simonw/datasette/issues/615,846595473,MDEyOklzc3VlQ29tbWVudDg0NjU5NTQ3Mw==,9599,simonw,2021-05-23T17:13:09Z,2021-05-23T17:13:09Z,OWNER,"Natalie suggests a quick way to implement ""undo"" would be to add a ""Show all columns"" item to that menu which only appears when at least one column is hidden.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846595091,https://api.github.com/repos/simonw/datasette/issues/615,846595091,MDEyOklzc3VlQ29tbWVudDg0NjU5NTA5MQ==,9599,simonw,2021-05-23T17:09:59Z,2021-05-23T17:09:59Z,OWNER,"Here's that prototype of a ""Hide this column"" cog menu button:

![hide](https://user-images.githubusercontent.com/9599/119269916-effd7180-bbae-11eb-92fa-0ef374bda4b8.gif)

Need a way to undo that once you've hidden a column - maybe a list of currently hidden columns that lets  you un-hide them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/615#issuecomment-846592392,https://api.github.com/repos/simonw/datasette/issues/615,846592392,MDEyOklzc3VlQ29tbWVudDg0NjU5MjM5Mg==,9599,simonw,2021-05-23T16:49:30Z,2021-05-23T16:49:44Z,OWNER,"I started looking at this again, inspired by #1326. I have a new diff that works against the latest `main` branch.

```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 4879228..f4b2ee2 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -64,6 +64,36 @@ class Row:
 
 
 class RowTableShared(DataView):
+    async def columns_to_select(self, db, table, request):
+        table_columns = await db.table_columns(table)
+        if ""_col"" in request.args and ""_nocol"" in request.args:
+            raise DatasetteError(""Cannot use _col and _nocol at the same time"")
+        if ""_col"" in request.args:
+            new_columns = []
+            for column in request.args.getlist(""_col""):
+                if column not in table_columns:
+                    raise DatasetteError(""_col={} is an invalid column"".format(column))
+                new_columns.append(column)
+            return new_columns
+        elif ""_nocol"" in request.args:
+            # Return all columns EXCEPT these
+            bad_columns = [
+                column
+                for column in request.args.getlist(""_nocol"")
+                if column not in table_columns
+            ]
+            if bad_columns:
+                raise DatasetteError(
+                    ""_nocol={} - invalid columns"".format("", "".join(bad_columns))
+                )
+            return [
+                column
+                for column in table_columns
+                if column not in request.args.getlist(""_nocol"")
+            ]
+        else:
+            return table_columns
+
     async def sortable_columns_for_table(self, database, table, use_rowid):
         db = self.ds.databases[database]
         table_metadata = self.ds.table_metadata(database, table)
@@ -321,18 +351,16 @@ class TableView(RowTableShared):
         )
 
         pks = await db.primary_keys(table)
-        table_column_details = await db.table_column_details(table)
-        table_columns = [column.name for column in table_column_details]
-
-        select_columns = "", "".join(escape_sqlite(t) for t in table_columns)
+        table_columns = await self.columns_to_select(db, table, request)
+        select_clause = "", "".join(escape_sqlite(t) for t in table_columns)
 
         use_rowid = not pks and not is_view
         if use_rowid:
-            select = f""rowid, {select_columns}""
+            select = f""rowid, {select_clause}""
             order_by = ""rowid""
             order_by_pks = ""rowid""
         else:
-            select = select_columns
+            select = select_clause
             order_by_pks = "", "".join([escape_sqlite(pk) for pk in pks])
             order_by = order_by_pks
 
@@ -715,6 +743,8 @@ class TableView(RowTableShared):
                 column = fk[""column""]
                 if column not in columns_to_expand:
                     continue
+                if column not in columns:
+                    continue
                 expanded_columns.append(column)
                 # Gather the values
                 column_index = columns.index(column)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",517451234,?_col= and ?_nocol= support for toggling columns on table view,
https://github.com/simonw/datasette/issues/1326#issuecomment-846493370,https://api.github.com/repos/simonw/datasette/issues/1326,846493370,MDEyOklzc3VlQ29tbWVudDg0NjQ5MzM3MA==,9599,simonw,2021-05-23T02:55:00Z,2021-05-23T02:55:00Z,OWNER,"Currently you can do this with a custom SQL query, but as you've noticed this means you don't get automatic pagination: https://latest.datasette.io/fixtures.json?sql=select+pk%2C+created%2C+state+from+facetable+order+by+pk+limit+10&_shape=array

I have an open issue to allow selecting specific columns on the table page: #615

A workaround you can use today is to define a SQL view that selects just the tables you want. View pages behave exactly like tables in that they provide pagination by default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",891969037,How to limit fields returned from the JSON API?,
https://github.com/simonw/datasette/issues/1331#issuecomment-846483426,https://api.github.com/repos/simonw/datasette/issues/1331,846483426,MDEyOklzc3VlQ29tbWVudDg0NjQ4MzQyNg==,9599,simonw,2021-05-23T00:57:30Z,2021-05-23T00:57:30Z,OWNER,Here's an alpha release with updated dependencies: https://pypi.org/project/datasette/0.57a0/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1331#issuecomment-846482057,https://api.github.com/repos/simonw/datasette/issues/1331,846482057,MDEyOklzc3VlQ29tbWVudDg0NjQ4MjA1Nw==,9599,simonw,2021-05-23T00:39:55Z,2021-05-23T00:39:55Z,OWNER,"> I'm stuck also because datasette wants itsdangerous~=1.1 instead of allowing itsdangerous-2.0.0

Bumped that dependency in b64d87204612a84663616e075f542499a5d82a03","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1331#issuecomment-846481937,https://api.github.com/repos/simonw/datasette/issues/1331,846481937,MDEyOklzc3VlQ29tbWVudDg0NjQ4MTkzNw==,9599,simonw,2021-05-23T00:38:28Z,2021-05-23T00:38:28Z,OWNER,I'm going to push an alpha release with the new dependencies shortly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1337#issuecomment-846481896,https://api.github.com/repos/simonw/datasette/issues/1337,846481896,MDEyOklzc3VlQ29tbWVudDg0NjQ4MTg5Ng==,9599,simonw,2021-05-23T00:37:52Z,2021-05-23T00:37:52Z,OWNER,"The tricky thing about this is that ideally I'd like this to link to a paginated set of results - but arbitrary SQL queries do not yet support pagination, see #856.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1332#issuecomment-846481801,https://api.github.com/repos/simonw/datasette/issues/1332,846481801,MDEyOklzc3VlQ29tbWVudDg0NjQ4MTgwMQ==,9599,simonw,2021-05-23T00:36:20Z,2021-05-23T00:36:20Z,OWNER,"Documentation:

- https://docs.datasette.io/en/latest/facets.html#facets-in-query-strings at the bottom of the section
- https://docs.datasette.io/en/latest/json_api.html#special-table-arguments at the bottom of the list of arguments","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1332#issuecomment-846479730,https://api.github.com/repos/simonw/datasette/issues/1332,846479730,MDEyOklzc3VlQ29tbWVudDg0NjQ3OTczMA==,9599,simonw,2021-05-23T00:11:31Z,2021-05-23T00:11:31Z,OWNER,"Here's the code in question: https://github.com/simonw/datasette/blob/5e9672c9bb33e41686472db4aa427168f9e67dbe/datasette/facets.py#L187-L210

It has access to the current request object as `self.request`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1332#issuecomment-846479212,https://api.github.com/repos/simonw/datasette/issues/1332,846479212,MDEyOklzc3VlQ29tbWVudDg0NjQ3OTIxMg==,9599,simonw,2021-05-23T00:07:46Z,2021-05-23T00:07:46Z,OWNER,`?_facet_size=50` seems like a very reasonable thing to support - I'll cap it at whatever the `max_returned_rows` setting is (which defaults to 1000).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1332#issuecomment-846479062,https://api.github.com/repos/simonw/datasette/issues/1332,846479062,MDEyOklzc3VlQ29tbWVudDg0NjQ3OTA2Mg==,9599,simonw,2021-05-23T00:06:34Z,2021-05-23T00:06:34Z,OWNER,"A URL parameter to modify that facet size is a really good idea.

I thought I had an issue open for ""..."" linking to more results but I can't find it now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-844250232,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59,844250232,MDEyOklzc3VlQ29tbWVudDg0NDI1MDIzMg==,9599,simonw,2021-05-19T16:08:10Z,2021-05-19T16:08:10Z,MEMBER,Thanks for catching this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771872303,Remove unneeded exists=True for -a/--auth flag.,
https://github.com/dogsheep/github-to-sqlite/pull/61#issuecomment-844249385,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61,844249385,MDEyOklzc3VlQ29tbWVudDg0NDI0OTM4NQ==,9599,simonw,2021-05-19T16:07:06Z,2021-05-19T16:07:06Z,MEMBER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797108702,fixing typo in get cli help text,
https://github.com/simonw/sqlite-utils/issues/253#issuecomment-843718859,https://api.github.com/repos/simonw/sqlite-utils/issues/253,843718859,MDEyOklzc3VlQ29tbWVudDg0MzcxODg1OQ==,9599,simonw,2021-05-19T03:31:47Z,2021-05-19T03:31:47Z,OWNER,Fixed: https://simonwillison.net/2020/Sep/23/sqlite-advanced-alter-table/,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",847423559,fixtures.db example error in sql-utils blog post,
https://github.com/simonw/sqlite-utils/issues/253#issuecomment-843716042,https://api.github.com/repos/simonw/sqlite-utils/issues/253,843716042,MDEyOklzc3VlQ29tbWVudDg0MzcxNjA0Mg==,9599,simonw,2021-05-19T03:29:04Z,2021-05-19T03:29:04Z,OWNER,"Thanks for pointing this out - the problem here is that these days the `https://latest.datasette.io/fixtures.db` example database includes a test for generated columns, which are a brand new feature in SQLite. Your local SQLite is an older version.

I'm going to update the blog post to instead use a copy of `fixtures.db` that doesn't use that feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",847423559,fixtures.db example error in sql-utils blog post,
https://github.com/simonw/sqlite-utils/issues/259#issuecomment-843715502,https://api.github.com/repos/simonw/sqlite-utils/issues/259,843715502,MDEyOklzc3VlQ29tbWVudDg0MzcxNTUwMg==,9599,simonw,2021-05-19T03:27:33Z,2021-05-19T03:27:33Z,OWNER,"```
 % sqlite-utils insert test-big.db text big.json 
Error: table text has no column named colun

Try using --alter to add additional columns
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",894948100,Suggest the --alter option if a new column cannot be added,
https://github.com/simonw/sqlite-utils/issues/256#issuecomment-843715308,https://api.github.com/repos/simonw/sqlite-utils/issues/256,843715308,MDEyOklzc3VlQ29tbWVudDg0MzcxNTMwOA==,9599,simonw,2021-05-19T03:26:54Z,2021-05-19T03:26:54Z,OWNER,"I decided to treat this as a usability bug: there's a workaround, which is adding `--alter` - but it's not at all obvious that you should do that. I addressed this in #259.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",861622839,inserting with --nl errors with: sqlite3.OperationalError: table <table> has no column named <column>,
https://github.com/simonw/sqlite-utils/issues/256#issuecomment-843708246,https://api.github.com/repos/simonw/sqlite-utils/issues/256,843708246,MDEyOklzc3VlQ29tbWVudDg0MzcwODI0Ng==,9599,simonw,2021-05-19T03:06:02Z,2021-05-19T03:06:02Z,OWNER,Did you try running this with the `--alter` command-line option? That seems to work when I try it on a sample file here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",861622839,inserting with --nl errors with: sqlite3.OperationalError: table <table> has no column named <column>,
https://github.com/simonw/sqlite-utils/pull/244#issuecomment-843705806,https://api.github.com/repos/simonw/sqlite-utils/issues/244,843705806,MDEyOklzc3VlQ29tbWVudDg0MzcwNTgwNg==,9599,simonw,2021-05-19T02:58:18Z,2021-05-19T02:58:18Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",820468864,Typo in upsert example,
https://github.com/simonw/sqlite-utils/pull/245#issuecomment-843705721,https://api.github.com/repos/simonw/sqlite-utils/issues/245,843705721,MDEyOklzc3VlQ29tbWVudDg0MzcwNTcyMQ==,9599,simonw,2021-05-19T02:58:01Z,2021-05-19T02:58:01Z,OWNER,Thanks very much.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830803173,Correct some typos,
https://github.com/simonw/sqlite-utils/pull/254#issuecomment-843705533,https://api.github.com/repos/simonw/sqlite-utils/issues/254,843705533,MDEyOklzc3VlQ29tbWVudDg0MzcwNTUzMw==,9599,simonw,2021-05-19T02:57:22Z,2021-05-19T02:57:22Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",857280617,Fix incorrect create-table cli description,
https://github.com/simonw/sqlite-utils/pull/258#issuecomment-843702392,https://api.github.com/repos/simonw/sqlite-utils/issues/258,843702392,MDEyOklzc3VlQ29tbWVudDg0MzcwMjM5Mg==,9599,simonw,2021-05-19T02:47:37Z,2021-05-19T02:47:37Z,OWNER,I'm going to merge this and add a test - thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",868191959,Fixing insert from JSON containing strings with non-ascii characters …,
https://github.com/simonw/datasette/issues/856#issuecomment-843291675,https://api.github.com/repos/simonw/datasette/issues/856,843291675,MDEyOklzc3VlQ29tbWVudDg0MzI5MTY3NQ==,9599,simonw,2021-05-18T15:56:45Z,2021-05-18T15:56:45Z,OWNER,"Tables and views get ""stream all rows"" at the moment, so one workaround is to define a SQL view for your query - this only works for queries that don't take any parameters though (although you may be able to define a view and then pass it extra fields using the Datasette table interface, like on https://latest.datasette.io/fixtures/paginated_view?content_extra__contains=9)

I've explored this problem in a bit more detail in https://githu.com/simonw/django-sql-dashboard and I think I have a pattern that could work.

For your canned query, you could implement the pattern yourself by setting up two canned queries that look something like this:

https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+sha%2C+author_date+from+commits+order+by+rowid+limit+1000

```sql
select rowid, sha, author_date from commits order by rowid limit 1000
```
That gets you the first set of 1,000 results. The important thing here is to order by a unique column, in this case `rowid` - because then subsequent pages can be loaded by a separate canned query that looks like this:
```sql
select rowid, sha, author_date from commits where rowid > :after order by rowid limit 1000
```
https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+sha%2C+author_date+from+commits+where+rowid+%3E+%3Aafter+order+by+rowid+limit+1000&after=1000

You then need to write code which knows how to generate these queries - start with the first query with no `where` clause (or if you are using `rowid` you can just use the second query and pass it `?after=0` for the first call) - then keep calling the query passing in the last rowid you recieved as the `after` parameter.

Basically this is an implementation of keyset pagination with a smart client. When Datasette grows the ability to do this itself it will work by executing this mechanism inside the Python code, which is how the ""stream all rows"" option for tables works at the moment.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",642296989,Consider pagination of canned queries,
https://github.com/simonw/datasette/issues/1331#issuecomment-842497125,https://api.github.com/repos/simonw/datasette/issues/1331,842497125,MDEyOklzc3VlQ29tbWVudDg0MjQ5NzEyNQ==,9599,simonw,2021-05-17T17:20:15Z,2021-05-17T17:20:15Z,OWNER,I wonder if there are any new 3.0 features we should be taking advantage of here that would justify pinning to 3.0 minimum?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1331#issuecomment-842495346,https://api.github.com/repos/simonw/datasette/issues/1331,842495346,MDEyOklzc3VlQ29tbWVudDg0MjQ5NTM0Ng==,9599,simonw,2021-05-17T17:17:19Z,2021-05-17T17:17:19Z,OWNER,The tests are passing on #1324 so this may not require any code changes at all!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/pull/1324#issuecomment-842494844,https://api.github.com/repos/simonw/datasette/issues/1324,842494844,MDEyOklzc3VlQ29tbWVudDg0MjQ5NDg0NA==,9599,simonw,2021-05-17T17:16:36Z,2021-05-17T17:16:36Z,OWNER,Refs #1331.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",890073940,"Update jinja2 requirement from <2.12.0,>=2.10.3 to >=2.10.3,<3.1.0",
https://github.com/simonw/datasette/issues/1331#issuecomment-842494332,https://api.github.com/repos/simonw/datasette/issues/1331,842494332,MDEyOklzc3VlQ29tbWVudDg0MjQ5NDMzMg==,9599,simonw,2021-05-17T17:15:45Z,2021-05-17T17:16:04Z,OWNER,Datasette uses async templates which evidently have been improved in Jinja 2 3.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893537744,Add support for Jinja2 version 3.0,
https://github.com/simonw/datasette/issues/1168#issuecomment-834636796,https://api.github.com/repos/simonw/datasette/issues/1168,834636796,MDEyOklzc3VlQ29tbWVudDgzNDYzNjc5Ng==,9599,simonw,2021-05-07T17:22:52Z,2021-05-07T17:22:52Z,OWNER,Related: Here's an implementation of a `get_metadata()` plugin hook by @brandonrobertz https://github.com/next-LI/datasette/commit/3fd8ce91f3108c82227bf65ff033929426c60437,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1308#issuecomment-826041458,https://api.github.com/repos/simonw/datasette/issues/1308,826041458,MDEyOklzc3VlQ29tbWVudDgyNjA0MTQ1OA==,9599,simonw,2021-04-24T06:07:01Z,2021-04-24T06:07:01Z,OWNER,"I can use `td.type-pk` instead - here's the existing HTML:
```html
<td class=""col-Link type-pk""><a href=""/link/link/1"">1</a></td>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",866668415,"Columns named ""link"" display in bold",
https://github.com/simonw/datasette/issues/1308#issuecomment-826040909,https://api.github.com/repos/simonw/datasette/issues/1308,826040909,MDEyOklzc3VlQ29tbWVudDgyNjA0MDkwOQ==,9599,simonw,2021-04-24T06:01:21Z,2021-04-24T06:01:21Z,OWNER,"Demo:
```
echo '{""link"": ""https://example.com/""}' | sqlite-utils insert link.db link -
```
<img width=""467"" alt=""link__link__1_row"" src=""https://user-images.githubusercontent.com/9599/115949159-cfdf7300-a487-11eb-84d1-437d954d1841.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",866668415,"Columns named ""link"" display in bold",
https://github.com/simonw/datasette/issues/1308#issuecomment-826040676,https://api.github.com/repos/simonw/datasette/issues/1308,826040676,MDEyOklzc3VlQ29tbWVudDgyNjA0MDY3Ng==,9599,simonw,2021-04-24T05:58:35Z,2021-04-24T05:58:35Z,OWNER,Here's why: https://github.com/simonw/datasette/blob/6ed9238178a56da5fb019f37fb1e1e15886be1d1/datasette/static/app.css#L435-L437,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",866668415,"Columns named ""link"" display in bold",
https://github.com/simonw/datasette/issues/1295#issuecomment-817301355,https://api.github.com/repos/simonw/datasette/issues/1295,817301355,MDEyOklzc3VlQ29tbWVudDgxNzMwMTM1NQ==,9599,simonw,2021-04-11T12:40:25Z,2021-04-11T12:41:06Z,OWNER,"I could have a page about error codes in the docs, then have `https://datasette.io/E123` style URLs for each error core which are shown when that error occurs and redirect to the corresponding documentation section.

Can enforce these with a documentation unit test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855296937,Errors should have links to further information,
https://github.com/simonw/datasette/issues/1293#issuecomment-813480043,https://api.github.com/repos/simonw/datasette/issues/1293,813480043,MDEyOklzc3VlQ29tbWVudDgxMzQ4MDA0Mw==,9599,simonw,2021-04-05T16:16:17Z,2021-04-05T16:16:17Z,OWNER,"https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:

```sql
CREATE VIEW paginated_view AS
    SELECT
        content,
        '- ' || content || ' -' AS content_extra
    FROM no_primary_key;
```
So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813445512,https://api.github.com/repos/simonw/datasette/issues/1293,813445512,MDEyOklzc3VlQ29tbWVudDgxMzQ0NTUxMg==,9599,simonw,2021-04-05T15:11:40Z,2021-04-05T15:11:40Z,OWNER,"Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813438771,https://api.github.com/repos/simonw/datasette/issues/1293,813438771,MDEyOklzc3VlQ29tbWVudDgxMzQzODc3MQ==,9599,simonw,2021-04-05T14:58:48Z,2021-04-05T14:58:48Z,OWNER,I may need to do something special for rowid columns - there is a `RowId` opcode that might come into play here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/620#issuecomment-813167335,https://api.github.com/repos/simonw/datasette/issues/620,813167335,MDEyOklzc3VlQ29tbWVudDgxMzE2NzMzNQ==,9599,simonw,2021-04-05T03:57:22Z,2021-04-05T03:57:22Z,OWNER,This may be obsoleted by #1293 - it looks like I may be able to auto-detect these foreign keys for arbitrary queries after all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520667773,Mechanism for indicating foreign key relationships in the table and query page URLs,
https://github.com/simonw/datasette/issues/1293#issuecomment-813164282,https://api.github.com/repos/simonw/datasette/issues/1293,813164282,MDEyOklzc3VlQ29tbWVudDgxMzE2NDI4Mg==,9599,simonw,2021-04-05T03:42:26Z,2021-04-05T03:42:36Z,OWNER,"Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the `explain select...` query. Using `defaultdict` seems to work there:

```pycon
>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))
>>> [dict(r) for r in rows if r['opcode'] == 'Variable']
[{'addr': 2,
  'opcode': 'Variable',
  'p1': 1,
  'p2': 1,
  'p3': 0,
  'p4': ':id',
  'p5': 0,
  'comment': None}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813162622,https://api.github.com/repos/simonw/datasette/issues/1293,813162622,MDEyOklzc3VlQ29tbWVudDgxMzE2MjYyMg==,9599,simonw,2021-04-05T03:34:24Z,2021-04-05T03:40:35Z,OWNER,"This almost works, but throws errors with some queries (anything with a `rowid` column for example) - it needs a bunch of test coverage.
```python
def columns_for_query(conn, sql):
    rows = conn.execute('explain ' + sql).fetchall()
    table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
    names_by_rootpage = dict(
        conn.execute(
            'select rootpage, name from sqlite_master where rootpage in ({})'.format(
                ', '.join(map(str, table_rootpage_by_register.values()))
            )
        )
    )
    columns_by_column_register = {}
    for row in rows:
        if row['opcode'] == 'Column':
            addr, opcode, table_id, cid, column_register, p4, p5, comment = row
            table = names_by_rootpage[table_rootpage_by_register[table_id]]
            columns_by_column_register[column_register] = (table, cid)
    result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
    registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""] - 1))
    all_column_names = {}
    for table in names_by_rootpage.values():
        table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
        for row in table_xinfo:
            all_column_names[(table, row[""cid""])] = row[""name""]
    final_output = []
    for r in registers:
        try:
            table, cid = columns_by_column_register[r]
            final_output.append((table, all_column_names[table, cid]))
        except KeyError:
            final_output.append((None, None))
    return final_output
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134637,https://api.github.com/repos/simonw/datasette/issues/1293,813134637,MDEyOklzc3VlQ29tbWVudDgxMzEzNDYzNw==,9599,simonw,2021-04-05T01:21:59Z,2021-04-05T01:21:59Z,OWNER,"http://www.sqlite.org/draft/lang_explain.html says:

> Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.

I'm going to keep exploring this though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134227,https://api.github.com/repos/simonw/datasette/issues/1293,813134227,MDEyOklzc3VlQ29tbWVudDgxMzEzNDIyNw==,9599,simonw,2021-04-05T01:19:31Z,2021-04-05T01:19:31Z,OWNER,"|   addr | opcode        |   p1 |   p2 |   p3 | p4                    |   p5 | comment   |
|--------|---------------|------|------|------|-----------------------|------|-----------|
|      0 | Init          |    0 |   47 |    0 |                       |   00 |           |
|      1 | OpenRead      |    0 |   51 |    0 | 15                    |   00 |           |
|      2 | Integer       |   15 |    2 |    0 |                       |   00 |           |
|      3 | Once          |    0 |   15 |    0 |                       |   00 |           |
|      4 | OpenEphemeral |    2 |    1 |    0 | k(1,)                 |   00 |           |
|      5 | VOpen         |    1 |    0 |    0 | vtab:3E692C362158     |   00 |           |
|      6 | String8       |    0 |    5 |    0 | CPAD_2020a_SuperUnits |   00 |           |
|      7 | SCopy         |    7 |    6 |    0 |                       |   00 |           |
|      8 | Integer       |    2 |    3 |    0 |                       |   00 |           |
|      9 | Integer       |    2 |    4 |    0 |                       |   00 |           |
|     10 | VFilter       |    1 |   15 |    3 |                       |   00 |           |
|     11 | Rowid         |    1 |    8 |    0 |                       |   00 |           |
|     12 | MakeRecord    |    8 |    1 |    9 | C                     |   00 |           |
|     13 | IdxInsert     |    2 |    9 |    8 | 1                     |   00 |           |
|     14 | VNext         |    1 |   11 |    0 |                       |   00 |           |
|     15 | Return        |    2 |    0 |    0 |                       |   00 |           |
|     16 | Rewind        |    2 |   46 |    0 |                       |   00 |           |
|     17 | Column        |    2 |    0 |    1 |                       |   00 |           |
|     18 | IsNull        |    1 |   45 |    0 |                       |   00 |           |
|     19 | SeekRowid     |    0 |   45 |    1 |                       |   00 |           |
|     20 | Column        |    0 |    2 |   11 |                       |   00 |           |
|     21 | Function0     |    1 |   10 |    9 | like(2)               |   02 |           |
|     22 | IfNot         |    9 |   45 |    1 |                       |   00 |           |
|     23 | Column        |    0 |   14 |   13 |                       |   00 |           |
|     24 | Function0     |    1 |   12 |    9 | intersects(2)         |   02 |           |
|     25 | Ne            |   14 |   45 |    9 |                       |   51 |           |
|     26 | Column        |    0 |   14 |    9 |                       |   00 |           |
|     27 | Function0     |    0 |    9 |   15 | asgeojson(1)          |   01 |           |
|     28 | Rowid         |    0 |   16 |    0 |                       |   00 |           |
|     29 | Column        |    0 |    1 |   17 |                       |   00 |           |
|     30 | Column        |    0 |    2 |   18 |                       |   00 |           |
|     31 | Column        |    0 |    3 |   19 |                       |   00 |           |
|     32 | Column        |    0 |    4 |   20 |                       |   00 |           |
|     33 | Column        |    0 |    5 |   21 |                       |   00 |           |
|     34 | Column        |    0 |    6 |   22 |                       |   00 |           |
|     35 | Column        |    0 |    7 |   23 |                       |   00 |           |
|     36 | Column        |    0 |    8 |   24 |                       |   00 |           |
|     37 | Column        |    0 |    9 |   25 |                       |   00 |           |
|     38 | Column        |    0 |   10 |   26 |                       |   00 |           |
|     39 | Column        |    0 |   11 |   27 |                       |   00 |           |
|     40 | RealAffinity  |   27 |    0 |    0 |                       |   00 |           |
|     41 | Column        |    0 |   12 |   28 |                       |   00 |           |
|     42 | Column        |    0 |   13 |   29 |                       |   00 |           |
|     43 | Column        |    0 |   14 |   30 |                       |   00 |           |
|     44 | ResultRow     |   15 |   16 |    0 |                       |   00 |           |
|     45 | Next          |    2 |   17 |    0 |                       |   00 |           |
|     46 | Halt          |    0 |    0 |    0 |                       |   00 |           |
|     47 | Transaction   |    0 |    0 |  265 | 0                     |   01 |           |
|     48 | Variable      |    1 |   31 |    0 | :freedraw             |   00 |           |
|     49 | Function0     |    1 |   31 |    7 | geomfromgeojson(1)    |   01 |           |
|     50 | String8       |    0 |   10 |    0 | %mini%                |   00 |           |
|     51 | Variable      |    1 |   32 |    0 | :freedraw             |   00 |           |
|     52 | Function0     |    1 |   32 |   12 | geomfromgeojson(1)    |   01 |           |
|     53 | Integer       |    1 |   14 |    0 |                       |   00 |           |
|     54 | Goto          |    0 |    1 |    0 |                       |   00 |           |

Essential documentation for understanding that output: https://www.sqlite.org/opcode.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134072,https://api.github.com/repos/simonw/datasette/issues/1293,813134072,MDEyOklzc3VlQ29tbWVudDgxMzEzNDA3Mg==,9599,simonw,2021-04-05T01:18:37Z,2021-04-05T01:18:37Z,OWNER,"Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by `explain select ...`.

Copying the reply I posted to this thread:

That's really useful, thanks! It looks like it _might_ be possible for me to reconstruct where each column came from using the `explain select` output.

Here's a complex example: <https://calands.datasettes.com/calands?sql=explain+select%0D%0A++AsGeoJSON%28geometry%29%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+%27%25mini%25%27+and%0D%0A++Intersects%28GeomFromGeoJSON%28%3Afreedraw%29%2C+geometry%29+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+%28%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+%27CPAD_2020a_SuperUnits%27%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON%28%3Afreedraw%29%0D%0A++%29&freedraw=%7B%22type%22%3A%22MultiPolygon%22%2C%22coordinates%22%3A%5B%5B%5B%5B-122.42202758789064%2C37.82280243352759%5D%2C%5B-122.39868164062501%2C37.823887203271454%5D%2C%5B-122.38220214843751%2C37.81846319511331%5D%2C%5B-122.35061645507814%2C37.77071473849611%5D%2C%5B-122.34924316406251%2C37.74465712069939%5D%2C%5B-122.37258911132814%2C37.703380457832374%5D%2C%5B-122.39044189453125%2C37.690340943717715%5D%2C%5B-122.41241455078126%2C37.680559803205135%5D%2C%5B-122.44262695312501%2C37.67295135774715%5D%2C%5B-122.47283935546876%2C37.67295135774715%5D%2C%5B-122.52502441406251%2C37.68382032669382%5D%2C%5B-122.53463745117189%2C37.6892542140253%5D%2C%5B-122.54699707031251%2C37.690340943717715%5D%2C%5B-122.55798339843751%2C37.72945260537781%5D%2C%5B-122.54287719726564%2C37.77831314799672%5D%2C%5B-122.49893188476564%2C37.81303878836991%5D%2C%5B-122.46185302734376%2C37.82822612280363%5D%2C%5B-122.42889404296876%2C37.82822612280363%5D%2C%5B-122.42202758789064%2C37.82280243352759%5D%5D%5D%5D%7D>

It looks like the opcodes I need to inspect are `OpenRead`, `Column` and `ResultRow`.

`OpenRead` tells me which tables are being opened - the `p2` value (in this case 51) corresponds to the `rootpage` column in `sqlite_master` here: <https://calands.datasettes.com/calands?sql=select+*+from+sqlite_master> - it gets assigned to the register in `p1`.

The `Column` opcodes tell me which columns are being read - `p1` is that table reference, and `p2` is the `cid` of the column within that table.

The `ResultRow` opcode then tells me which columns are used in the results. `15 16` means start at the 15th and then read the next 16 columns.

I think this might work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813116177,https://api.github.com/repos/simonw/datasette/issues/1293,813116177,MDEyOklzc3VlQ29tbWVudDgxMzExNjE3Nw==,9599,simonw,2021-04-04T23:31:00Z,2021-04-04T23:31:00Z,OWNER,"Sadly it doesn't do what I need. This query should only return one column, but instead I get back every column that was consulted by the query:

<img width=""597"" alt=""sql-metadata_-_Jupyter_Notebook"" src=""https://user-images.githubusercontent.com/9599/113524385-216ca000-9563-11eb-8a7e-cbbe5dfc02f6.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813115607,https://api.github.com/repos/simonw/datasette/issues/1293,813115607,MDEyOklzc3VlQ29tbWVudDgxMzExNTYwNw==,9599,simonw,2021-04-04T23:25:15Z,2021-04-04T23:25:15Z,OWNER,"Oh wow, I just spotted https://github.com/macbre/sql-metadata

> Uses tokenized query returned by python-sqlparse and generates query metadata. Extracts column names and tables used by the query. Provides a helper for normalization of SQL queries and tables aliases resolving.

It's for MySQL, PostgreSQL and Hive right now but maybe getting it working with SQLite wouldn't be too hard?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813115414,https://api.github.com/repos/simonw/datasette/issues/1293,813115414,MDEyOklzc3VlQ29tbWVudDgxMzExNTQxNA==,9599,simonw,2021-04-04T23:23:34Z,2021-04-04T23:23:34Z,OWNER,"The other approach I considered for this was to have my own SQL query parser running in Python, which could pick apart a complex query and figure out which column was sourced from which table. I dropped this idea because it felt that the moment `select *` came into play a pure parsing approach wouldn't work - I'd need knowledge of the schema in order to resolve the `*`.

A Python parser approach might be good enough to handle a subset of queries - those that don't use `select *` for example - and maybe that would be worth shipping? The feature doesn't have to be perfect for it to be useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813114933,https://api.github.com/repos/simonw/datasette/issues/1293,813114933,MDEyOklzc3VlQ29tbWVudDgxMzExNDkzMw==,9599,simonw,2021-04-04T23:19:22Z,2021-04-04T23:19:22Z,OWNER,I asked about this on the SQLite forum: https://sqlite.org/forum/forumpost/0180277fb7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813113653,https://api.github.com/repos/simonw/datasette/issues/1293,813113653,MDEyOklzc3VlQ29tbWVudDgxMzExMzY1Mw==,9599,simonw,2021-04-04T23:10:49Z,2021-04-04T23:10:49Z,OWNER,"One option I've not fully explored yet: could I write my own custom SQLite C extension which exposes this functionality as a callable function?

Then I could load that extension and run a SQL query something like this:

```
select database, table, column from analyze_query(:sql_query)
```
Where `analyze_query(...)` would be a fancy virtual table function of some sort that uses the underlying `sqlite3_column_database_name()` C functions to analyze the SQL query and return details of what it would return.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813113403,https://api.github.com/repos/simonw/datasette/issues/1293,813113403,MDEyOklzc3VlQ29tbWVudDgxMzExMzQwMw==,9599,simonw,2021-04-04T23:08:48Z,2021-04-04T23:08:48Z,OWNER,"Worth noting that adding `limit 0` to the query still causes it to conduct the permission checks, hopefully while avoiding doing any of the actual work of executing the query:
```pycon
In [20]: db.execute('select * from compound_primary_key join facetable on facetable.rowid = compound_primary_key.rowid limit 0').fetchall()
    ...: 
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'ROWID', 'main', None) kwargs {}
Out[20]: []
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813113218,https://api.github.com/repos/simonw/datasette/issues/1293,813113218,MDEyOklzc3VlQ29tbWVudDgxMzExMzIxOA==,9599,simonw,2021-04-04T23:07:25Z,2021-04-04T23:07:25Z,OWNER,"Here are all of the available constants:
```pycon
In [3]: for k in dir(sqlite3):
   ...:     if k.startswith(""SQLITE_""):
   ...:         print(k, getattr(sqlite3, k))
   ...: 
SQLITE_ALTER_TABLE 26
SQLITE_ANALYZE 28
SQLITE_ATTACH 24
SQLITE_CREATE_INDEX 1
SQLITE_CREATE_TABLE 2
SQLITE_CREATE_TEMP_INDEX 3
SQLITE_CREATE_TEMP_TABLE 4
SQLITE_CREATE_TEMP_TRIGGER 5
SQLITE_CREATE_TEMP_VIEW 6
SQLITE_CREATE_TRIGGER 7
SQLITE_CREATE_VIEW 8
SQLITE_CREATE_VTABLE 29
SQLITE_DELETE 9
SQLITE_DENY 1
SQLITE_DETACH 25
SQLITE_DONE 101
SQLITE_DROP_INDEX 10
SQLITE_DROP_TABLE 11
SQLITE_DROP_TEMP_INDEX 12
SQLITE_DROP_TEMP_TABLE 13
SQLITE_DROP_TEMP_TRIGGER 14
SQLITE_DROP_TEMP_VIEW 15
SQLITE_DROP_TRIGGER 16
SQLITE_DROP_VIEW 17
SQLITE_DROP_VTABLE 30
SQLITE_FUNCTION 31
SQLITE_IGNORE 2
SQLITE_INSERT 18
SQLITE_OK 0
SQLITE_PRAGMA 19
SQLITE_READ 20
SQLITE_RECURSIVE 33
SQLITE_REINDEX 27
SQLITE_SAVEPOINT 32
SQLITE_SELECT 21
SQLITE_TRANSACTION 22
SQLITE_UPDATE 23
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813113175,https://api.github.com/repos/simonw/datasette/issues/1293,813113175,MDEyOklzc3VlQ29tbWVudDgxMzExMzE3NQ==,9599,simonw,2021-04-04T23:07:01Z,2021-04-04T23:07:01Z,OWNER,"A more promising route I found involved the `db.set_authorizer` method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!

```pycon
>>> def print_args(*args, **kwargs):
...    print(""args"", args, ""kwargs"", kwargs)
...    return sqlite3.SQLITE_OK

>>> db = sqlite3.connect(""fixtures.db"")
>>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
```
Those `20` values (where 20 is `SQLITE_READ`) looked like they were checking permissions for the columns in the order they would be returned!

Then I found a snag:

```pycon
In [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')
args (21, None, None, None, None) kwargs {}
args (31, None, 'max', None, None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (21, None, None, None, None) kwargs {}
args (20, 'facetable', '', None, None) kwargs {}
```
Once a subselect is involved the order of the `20` checks no longer matches the order in which the columns are returned from the query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-813112546,https://api.github.com/repos/simonw/datasette/issues/1293,813112546,MDEyOklzc3VlQ29tbWVudDgxMzExMjU0Ng==,9599,simonw,2021-04-04T23:02:45Z,2021-04-04T23:02:45Z,OWNER,"I've done various pieces of research into this over the past few years. Capturing what I've discovered in this ticket.

The SQLite C API has functions that can help with this: https://www.sqlite.org/c3ref/column_database_name.html details those. But they're not exposed in the Python SQLite library.

Maybe it would be possible to use them via `ctypes`? My hunch is that I would have to re-implement the full `sqlite3` module with `ctypes`, which sounds daunting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1292#issuecomment-813109789,https://api.github.com/repos/simonw/datasette/issues/1292,813109789,MDEyOklzc3VlQ29tbWVudDgxMzEwOTc4OQ==,9599,simonw,2021-04-04T22:37:47Z,2021-04-04T22:37:47Z,OWNER,Could maybe replace this code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/utils/__init__.py#L1021-L1026,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849975810,Research ctypes.util.find_library('spatialite'),
https://github.com/simonw/datasette/issues/1273#issuecomment-813061516,https://api.github.com/repos/simonw/datasette/issues/1273,813061516,MDEyOklzc3VlQ29tbWVudDgxMzA2MTUxNg==,9599,simonw,2021-04-04T16:32:40Z,2021-04-04T16:32:40Z,OWNER,Useful tutorial series from 2012: https://northredoubt.com/n/2012/01/20/spatialite-speed-test/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890,Refresh SpatiaLite documentation,
https://github.com/simonw/datasette/issues/916#issuecomment-812941818,https://api.github.com/repos/simonw/datasette/issues/916,812941818,MDEyOklzc3VlQ29tbWVudDgxMjk0MTgxOA==,9599,simonw,2021-04-03T23:43:11Z,2021-04-03T23:43:11Z,OWNER,"Relevant code is some of the most complex in all of Datasette.

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L530-L594

And

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L743-L771

I'll need to think hard about how to refactor this out into something more understandable before implementing previous links.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,"Support reverse pagination (previous page, has-previous-items)",
https://github.com/simonw/datasette/issues/916#issuecomment-812941340,https://api.github.com/repos/simonw/datasette/issues/916,812941340,MDEyOklzc3VlQ29tbWVudDgxMjk0MTM0MA==,9599,simonw,2021-04-03T23:38:37Z,2021-04-03T23:38:37Z,OWNER,"Same query again with `a, d, v` returns 0 results, which is also as we would want: it signifies that we are back to the very first page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=d&p2=v","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,"Support reverse pagination (previous page, has-previous-items)",
https://github.com/simonw/datasette/issues/916#issuecomment-812941112,https://api.github.com/repos/simonw/datasette/issues/916,812941112,MDEyOklzc3VlQ29tbWVudDgxMjk0MTExMg==,9599,simonw,2021-04-03T23:35:55Z,2021-04-03T23:35:55Z,OWNER,"I tried flipping the direction of the sort and the comparison operators and got this: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=h&p2=r

```sql
select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 < :p0)
  or
(pk1 = :p0 and pk2 < :p1)
  or
(pk1 = :p0 and pk2 = :p1 and pk3 < :p2)) order by pk1 desc, pk2 desc, pk3 desc limit 1 offset 99
```
Which returned `a-d-v` as desired. I messed around with it to find the `limit 1 offset 99` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,"Support reverse pagination (previous page, has-previous-items)",
https://github.com/simonw/datasette/issues/916#issuecomment-812940907,https://api.github.com/repos/simonw/datasette/issues/916,812940907,MDEyOklzc3VlQ29tbWVudDgxMjk0MDkwNw==,9599,simonw,2021-04-03T23:33:41Z,2021-04-03T23:33:41Z,OWNER,"Let's figure out the SQL for this. The most complex case is probably this one: https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Ch%2Cr

Here's the SQL for that page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3E+%3Ap2%29%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=a&p1=h&p2=r

```sql
select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1)
  or
(pk1 = :p0 and pk2 = :p1 and pk3 > :p2)) order by pk1, pk2, pk3 limit 101
```
Where `p0` is `a`, `p1` is `h` and `p2` is `r`.

Given the above, how would I figure out the correct previous link? It should be https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv - `a`, `d`, `v`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,"Support reverse pagination (previous page, has-previous-items)",
https://github.com/simonw/datasette/issues/916#issuecomment-812940457,https://api.github.com/repos/simonw/datasette/issues/916,812940457,MDEyOklzc3VlQ29tbWVudDgxMjk0MDQ1Nw==,9599,simonw,2021-04-03T23:28:40Z,2021-04-03T23:28:40Z,OWNER,"I think my ideal implementation for this would be to reverse the order, grab the previous page-size-plus-one items, then return a `?_next=x` token that would provide the previous page sorted back in the expected default order.

The alternative would be to have a `?_previous=x` token which can be used to paginate backwards in reverse order, but I think this would be confusing as it would result in ""hit next page, then hit previous page"" returning you to a new state which features rows in the reverse order.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,"Support reverse pagination (previous page, has-previous-items)",
https://github.com/simonw/datasette/issues/1287#issuecomment-812935384,https://api.github.com/repos/simonw/datasette/issues/1287,812935384,MDEyOklzc3VlQ29tbWVudDgxMjkzNTM4NA==,9599,simonw,2021-04-03T22:38:33Z,2021-04-03T22:38:33Z,OWNER,"https://twitter.com/llanga/status/1378431719934681094 looks like I should wait for 3.9.4, out in a few days.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,Upgrade to Python 3.9.4,
https://github.com/simonw/datasette/issues/916#issuecomment-812804998,https://api.github.com/repos/simonw/datasette/issues/916,812804998,MDEyOklzc3VlQ29tbWVudDgxMjgwNDk5OA==,9599,simonw,2021-04-03T03:47:45Z,2021-04-03T03:47:45Z,OWNER,I found one example of an implementation of reversed keyset pagination here: https://github.com/tvainika/objection-keyset-pagination/blob/cb21a493c96daa6e63c302efae6718d09aa11661/index.js#L74-L79,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",672421411,"Support reverse pagination (previous page, has-previous-items)",
https://github.com/simonw/datasette/issues/1289#issuecomment-812803256,https://api.github.com/repos/simonw/datasette/issues/1289,812803256,MDEyOklzc3VlQ29tbWVudDgxMjgwMzI1Ng==,9599,simonw,2021-04-03T03:29:25Z,2021-04-03T03:29:25Z,OWNER,"https://github.com/simonw/datasette/actions/runs/713207828 ran with `pytest-xdist` in 4m22s:

<img width=""1202"" alt=""Run_tests_in_CI_using_pytest-xdist__refs__1289_·_simonw_datasette_d63fc61"" src=""https://user-images.githubusercontent.com/9599/113466592-f151bf80-93f1-11eb-9d74-b830e96c19c8.png"">

Here's the test suite running on regular `pytest` in 5m13s:

<img width=""1162"" alt=""©_2017-2021_·_simonw_datasette_59ef4a2"" src=""https://user-images.githubusercontent.com/9599/113466606-00387200-93f2-11eb-95a3-24521c628cbe.png"">

Not a huge speed-up because there are only 2 available cores in the GitHub Actions environment, but still worthwhile - especially since this lets people run in parallel on their own laptops.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849543502,Speed up tests with pytest-xdist,
https://github.com/simonw/datasette/issues/1289#issuecomment-812768915,https://api.github.com/repos/simonw/datasette/issues/1289,812768915,MDEyOklzc3VlQ29tbWVudDgxMjc2ODkxNQ==,9599,simonw,2021-04-03T00:59:15Z,2021-04-03T00:59:26Z,OWNER,"Looks like `-n auto` only detected two cores on GitHub Actions: https://github.com/simonw/datasette/runs/2257597137?check_suite_focus=true
```
============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
SQLite: 3.31.1
rootdir: /home/runner/work/datasette/datasette, configfile: pytest.ini
plugins: xdist-2.2.1, timeout-1.4.2, forked-1.3.0, asyncio-0.14.0
gw0 I / gw1 I
gw0 [878] / gw1 [878]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849543502,Speed up tests with pytest-xdist,
https://github.com/simonw/datasette/issues/1289#issuecomment-812767460,https://api.github.com/repos/simonw/datasette/issues/1289,812767460,MDEyOklzc3VlQ29tbWVudDgxMjc2NzQ2MA==,9599,simonw,2021-04-03T00:48:26Z,2021-04-03T00:48:26Z,OWNER,"On my Mac `pytest-xdist` ran the test suite (minus two tests) in 59s, as opposed to 2m23s without xdist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849543502,Speed up tests with pytest-xdist,
https://github.com/simonw/datasette/issues/1287#issuecomment-812665092,https://api.github.com/repos/simonw/datasette/issues/1287,812665092,MDEyOklzc3VlQ29tbWVudDgxMjY2NTA5Mg==,9599,simonw,2021-04-02T18:54:29Z,2021-04-02T18:54:29Z,OWNER,`python:3.9.3-slim-buster` isn't on Docker Hub yet either.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,Upgrade to Python 3.9.4,
https://github.com/simonw/datasette/issues/1286#issuecomment-812664443,https://api.github.com/repos/simonw/datasette/issues/1286,812664443,MDEyOklzc3VlQ29tbWVudDgxMjY2NDQ0Mw==,9599,simonw,2021-04-02T18:52:45Z,2021-04-02T18:52:51Z,OWNER,"Idea: default to displaying single-dimension JSON arrays of strings as a comma-separated list but show the comma in a different colour - something like this:

<img width=""561"" alt=""fixtures__facetable__15_rows"" src=""https://user-images.githubusercontent.com/9599/113445005-dce9d480-93a9-11eb-901c-0c2fb079c150.png"">

I used this HTML for the prototype (re-using `.type-int` just to get the colour):
```html
<td class=""col-tags type-str"">tag1<span class=""type-int"">, </span>tag2</td>
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,Better default display of arrays of items,
https://github.com/simonw/datasette/issues/1286#issuecomment-812663107,https://api.github.com/repos/simonw/datasette/issues/1286,812663107,MDEyOklzc3VlQ29tbWVudDgxMjY2MzEwNw==,9599,simonw,2021-04-02T18:49:22Z,2021-04-02T18:49:22Z,OWNER,"This makes senses - showing an array as `[""blah"", ""blah2"", ""blah3""]` isn't particularly human-friendly!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,Better default display of arrays of items,
https://github.com/simonw/datasette/issues/1287#issuecomment-812662583,https://api.github.com/repos/simonw/datasette/issues/1287,812662583,MDEyOklzc3VlQ29tbWVudDgxMjY2MjU4Mw==,9599,simonw,2021-04-02T18:47:59Z,2021-04-02T18:47:59Z,OWNER,Once again having tests for the Dockerfile as seen in #1272 would be useful here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,Upgrade to Python 3.9.4,
https://github.com/simonw/datasette/issues/1287#issuecomment-812662026,https://api.github.com/repos/simonw/datasette/issues/1287,812662026,MDEyOklzc3VlQ29tbWVudDgxMjY2MjAyNg==,9599,simonw,2021-04-02T18:46:20Z,2021-04-02T18:46:20Z,OWNER,https://devcenter.heroku.com/articles/python-support#supported-runtimes looks like Heroku still only have 3.9.2 for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,Upgrade to Python 3.9.4,
https://github.com/simonw/datasette/issues/1287#issuecomment-812661269,https://api.github.com/repos/simonw/datasette/issues/1287,812661269,MDEyOklzc3VlQ29tbWVudDgxMjY2MTI2OQ==,9599,simonw,2021-04-02T18:45:08Z,2021-04-02T18:45:19Z,OWNER,"A few places:

https://github.com/simonw/datasette/blob/7b1a9a1999eb9326ce8ec830d75ac200e5279c46/Dockerfile#L1

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L350

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/tests/test_package.py#L14

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/publish/heroku.py#L177-L178","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849396758,Upgrade to Python 3.9.4,
https://github.com/simonw/datasette/issues/1284#issuecomment-810740486,https://api.github.com/repos/simonw/datasette/issues/1284,810740486,MDEyOklzc3VlQ29tbWVudDgxMDc0MDQ4Ng==,9599,simonw,2021-03-31T03:57:55Z,2021-03-31T03:57:55Z,OWNER,"You're right, doing this is really hard at the moment - I'm not sure I know how I would tackle this either, and it's something I've wanted in the past!

I'll have a think about this one.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",845794436,Feature or Documentation Request: Individual table as home page template,
https://github.com/simonw/datasette/pull/1282#issuecomment-809670294,https://api.github.com/repos/simonw/datasette/issues/1282,809670294,MDEyOklzc3VlQ29tbWVudDgwOTY3MDI5NA==,9599,simonw,2021-03-29T19:57:29Z,2021-03-29T19:57:29Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",843739658,Fix little typo,
https://github.com/simonw/datasette/issues/696#issuecomment-809548363,https://api.github.com/repos/simonw/datasette/issues/696,809548363,MDEyOklzc3VlQ29tbWVudDgwOTU0ODM2Mw==,9599,simonw,2021-03-29T17:04:19Z,2021-03-29T17:04:19Z,OWNER,I tried this just now against Datasette 0.56 with the new Dockerfile from #1249 (that uses SQLite and SpatiaLite installed with `apt-get install`) and the tests all passed.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",576722115,Single failing unit test when run inside the Docker image,
https://github.com/simonw/datasette/pull/1031#issuecomment-809010713,https://api.github.com/repos/simonw/datasette/issues/1031,809010713,MDEyOklzc3VlQ29tbWVudDgwOTAxMDcxMw==,9599,simonw,2021-03-29T01:46:45Z,2021-03-29T01:46:45Z,OWNER,Sorry I didn't get to this PR sooner. I've joint-credited you in the release notes for this fix: https://docs.datasette.io/en/stable/changelog.html#v0-56,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed,
https://github.com/simonw/datasette/issues/1281#issuecomment-809009580,https://api.github.com/repos/simonw/datasette/issues/1281,809009580,MDEyOklzc3VlQ29tbWVudDgwOTAwOTU4MA==,9599,simonw,2021-03-29T01:41:48Z,2021-03-29T01:41:48Z,OWNER,"https://github.com/simonw/datasette/runs/2214871602?check_suite_focus=true worked:

<img width=""1001"" alt=""Banners_and_Alerts_and_Actions_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/112776676-270f3680-8ff5-11eb-90d1-533d4e82f677.png"">

Here's the 0.56 image on Docker Hub: https://hub.docker.com/layers/datasetteproject/datasette/0.56/images/sha256-701fc0f299a0ea79434a4852c46dab351254b9ac25dbe3c5f36fd5360caf52f9?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,Latest Datasette tags missing from Docker Hub,
https://github.com/simonw/datasette/issues/1281#issuecomment-809008760,https://api.github.com/repos/simonw/datasette/issues/1281,809008760,MDEyOklzc3VlQ29tbWVudDgwOTAwODc2MA==,9599,simonw,2021-03-29T01:38:21Z,2021-03-29T01:38:21Z,OWNER,"Got this error:
```
""docker tag"" requires exactly 2 arguments.
See 'docker tag --help'.

Usage:  docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,Latest Datasette tags missing from Docker Hub,
https://github.com/simonw/datasette/issues/1281#issuecomment-809007255,https://api.github.com/repos/simonw/datasette/issues/1281,809007255,MDEyOklzc3VlQ29tbWVudDgwOTAwNzI1NQ==,9599,simonw,2021-03-29T01:32:18Z,2021-03-29T01:32:18Z,OWNER,I'm going to build a new GitHub Actions workflow for this that lets me manually specify a tag to build and push as a Docker image.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,Latest Datasette tags missing from Docker Hub,
https://github.com/simonw/datasette/issues/1281#issuecomment-809001653,https://api.github.com/repos/simonw/datasette/issues/1281,809001653,MDEyOklzc3VlQ29tbWVudDgwOTAwMTY1Mw==,9599,simonw,2021-03-29T01:08:31Z,2021-03-29T01:08:31Z,OWNER,"I'm going to attempt to fix this manually for the 0.56 release, by building and tagging it by hand and then pushing the 0.56 tag to Docker Hub.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,Latest Datasette tags missing from Docker Hub,
https://github.com/simonw/datasette/issues/1281#issuecomment-809001273,https://api.github.com/repos/simonw/datasette/issues/1281,809001273,MDEyOklzc3VlQ29tbWVudDgwOTAwMTI3Mw==,9599,simonw,2021-03-29T01:06:45Z,2021-03-29T01:06:45Z,OWNER,"https://docs.docker.com/engine/reference/commandline/push/#push-all-tags-of-an-image

> Use the `-a` (or `--all-tags`) option to push all tags of a local image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,Latest Datasette tags missing from Docker Hub,
https://github.com/simonw/datasette/issues/1281#issuecomment-809000903,https://api.github.com/repos/simonw/datasette/issues/1281,809000903,MDEyOklzc3VlQ29tbWVudDgwOTAwMDkwMw==,9599,simonw,2021-03-29T01:05:10Z,2021-03-29T01:05:10Z,OWNER,"https://github.com/simonw/datasette/runs/1763835467?check_suite_focus=true for Datasette 0.54 worked, and the output included this:

```
Successfully tagged ***/datasette:0.54
The push refers to repository [docker.io/***/datasette]
aedd33c6b161: Preparing
...
aedd33c6b161: Pushed
0.54: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
aedd33c6b161: Preparing
...
5dacd731af1b: Layer already exists
latest: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
```
Here's that same section of output from the 0.56 release:
```
Successfully tagged ***/datasette:0.56
Using default tag: latest
The push refers to repository [docker.io/***/datasette]
4d4a9976adcc: Preparing
...
9b2132a0d5cf: Pushed
latest: digest: sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a size: 1793
```
The difference here is the ""Using default tag: latest"" bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,Latest Datasette tags missing from Docker Hub,
https://github.com/simonw/datasette/issues/1281#issuecomment-808999525,https://api.github.com/repos/simonw/datasette/issues/1281,808999525,MDEyOklzc3VlQ29tbWVudDgwODk5OTUyNQ==,9599,simonw,2021-03-29T01:00:38Z,2021-03-29T01:00:38Z,OWNER,"Here's the diff between `Dockerfile` in 0.54.1 and 0.56: https://github.com/simonw/datasette/compare/0.54.1...0.56#diff-551d1fcf87f78cc3bc18a7b332a4dc5d8773a512062df881c5aba28a6f5c48d7

<img width=""1357"" alt=""Banners_and_Alerts_and_Comparing_0_54_1___0_56_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/112774760-79e5ef80-8fef-11eb-886a-5d546d6d2f3d.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842881221,Latest Datasette tags missing from Docker Hub,
https://github.com/simonw/datasette/issues/1249#issuecomment-808998719,https://api.github.com/repos/simonw/datasette/issues/1249,808998719,MDEyOklzc3VlQ29tbWVudDgwODk5ODcxOQ==,9599,simonw,2021-03-29T00:57:13Z,2021-03-29T00:57:13Z,OWNER,"I just shipped Datasette 0.56 - here's the CI run: https://github.com/simonw/datasette/runs/2214701802?check_suite_focus=true

It pushed a new `latest` tag to https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated

    docker pull datasetteproject/datasette:latest

And then:

    docker run datasetteproject/datasette:latest datasette \
      --load-extension=spatialite \
      --get /-/versions.json | jq .sqlite

Outputs:
```json
{
  ""version"": ""3.27.2"",
  ""fts_versions"": [
    ""FTS5"",
    ""FTS4"",
    ""FTS3""
  ],
  ""extensions"": {
    ""json1"": null,
    ""spatialite"": ""5.0.1""
  },
  ""compile_options"": [
    ""COMPILER=gcc-8.3.0"",
    ""ENABLE_COLUMN_METADATA"",
    ""ENABLE_DBSTAT_VTAB"",
    ""ENABLE_FTS3"",
    ""ENABLE_FTS3_PARENTHESIS"",
    ""ENABLE_FTS3_TOKENIZER"",
    ""ENABLE_FTS4"",
    ""ENABLE_FTS5"",
    ""ENABLE_JSON1"",
    ""ENABLE_LOAD_EXTENSION"",
    ""ENABLE_PREUPDATE_HOOK"",
    ""ENABLE_RTREE"",
    ""ENABLE_SESSION"",
    ""ENABLE_STMTVTAB"",
    ""ENABLE_UNLOCK_NOTIFY"",
    ""ENABLE_UPDATE_DELETE_LIMIT"",
    ""HAVE_ISNAN"",
    ""LIKE_DOESNT_MATCH_BLOBS"",
    ""MAX_SCHEMA_RETRY=25"",
    ""MAX_VARIABLE_NUMBER=250000"",
    ""OMIT_LOOKASIDE"",
    ""SECURE_DELETE"",
    ""SOUNDEX"",
    ""TEMP_STORE=1"",
    ""THREADSAFE=1"",
    ""USE_URI""
  ]
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/pull/1031#issuecomment-808989067,https://api.github.com/repos/simonw/datasette/issues/1031,808989067,MDEyOklzc3VlQ29tbWVudDgwODk4OTA2Nw==,9599,simonw,2021-03-29T00:23:41Z,2021-03-29T00:23:41Z,OWNER,"This bug should have been fixed in #1229 - let me know if that's not the case!

Thanks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed,
https://github.com/simonw/datasette/pull/1260#issuecomment-808988697,https://api.github.com/repos/simonw/datasette/issues/1260,808988697,MDEyOklzc3VlQ29tbWVudDgwODk4ODY5Nw==,9599,simonw,2021-03-29T00:22:21Z,2021-03-29T00:22:21Z,OWNER,"This is interesting!

I've decided to apply a subset of these - the `if` and `elif` blocks are a deliberate style choice from me, because I find code clearer when it has if/else as opposed to relying on early termination. Likewise the iteration against `.keys()` on dictionaries.

I like the other fixes though, I'm about to land them in a separate commit that credits you.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831163537,Fix: code quality issues,
https://github.com/simonw/datasette/pull/1229#issuecomment-808987304,https://api.github.com/repos/simonw/datasette/issues/1229,808987304,MDEyOklzc3VlQ29tbWVudDgwODk4NzMwNA==,9599,simonw,2021-03-29T00:17:13Z,2021-03-29T00:17:13Z,OWNER,Thanks for figuring this out!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810507413,ensure immutable databses when starting in configuration directory mode with,
https://github.com/simonw/datasette/pull/1252#issuecomment-808986495,https://api.github.com/repos/simonw/datasette/issues/1252,808986495,MDEyOklzc3VlQ29tbWVudDgwODk4NjQ5NQ==,9599,simonw,2021-03-29T00:13:59Z,2021-03-29T00:13:59Z,OWNER,"Neat fix, thank you!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",825217564,Add back styling to lists within table cells (fixes #1141),
https://github.com/simonw/datasette/pull/1279#issuecomment-808986036,https://api.github.com/repos/simonw/datasette/issues/1279,808986036,MDEyOklzc3VlQ29tbWVudDgwODk4NjAzNg==,9599,simonw,2021-03-29T00:11:50Z,2021-03-29T00:11:50Z,OWNER,Thanks for the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842556944,Minor Docs Update. Added `--app` to fly install command.,
https://github.com/simonw/datasette/issues/1280#issuecomment-808983160,https://api.github.com/repos/simonw/datasette/issues/1280,808983160,MDEyOklzc3VlQ29tbWVudDgwODk4MzE2MA==,9599,simonw,2021-03-28T23:59:28Z,2021-03-29T00:10:05Z,OWNER,"Might be easier to do this using https://github.com/coleifer/pysqlite3 rather than try to replace the system `sqlite3` on the Ubuntu GitHub Actions instances.

These instructions should help: https://github.com/coleifer/pysqlite3#building-a-statically-linked-library","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842862708,Ability to run CI against multiple SQLite versions,
https://github.com/simonw/datasette/issues/1276#issuecomment-808981968,https://api.github.com/repos/simonw/datasette/issues/1276,808981968,MDEyOklzc3VlQ29tbWVudDgwODk4MTk2OA==,9599,simonw,2021-03-28T23:52:31Z,2021-03-28T23:52:31Z,OWNER,Testing this on Glitch by adding `https://github.com/simonw/datasette/archive/48d5e0e6ac8975cfd869d4e8c69c64ca0c65e29e.zip` as a dependency. That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1276#issuecomment-808979608,https://api.github.com/repos/simonw/datasette/issues/1276,808979608,MDEyOklzc3VlQ29tbWVudDgwODk3OTYwOA==,9599,simonw,2021-03-28T23:38:34Z,2021-03-28T23:38:34Z,OWNER,"Aha! https://www.sqlite.org/pragma.html says:

>  The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1276#issuecomment-808979218,https://api.github.com/repos/simonw/datasette/issues/1276,808979218,MDEyOklzc3VlQ29tbWVudDgwODk3OTIxOA==,9599,simonw,2021-03-28T23:35:53Z,2021-03-28T23:36:26Z,OWNER,"Here's where I run that: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L249-L253

That's from when I added the `--crossdb` option in #1232: https://github.com/simonw/datasette/commit/6f41c8a2bef309a66588b2875c3e24d26adb4850#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9R249","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1276#issuecomment-808979049,https://api.github.com/repos/simonw/datasette/issues/1276,808979049,MDEyOklzc3VlQ29tbWVudDgwODk3OTA0OQ==,9599,simonw,2021-03-28T23:34:38Z,2021-03-28T23:34:38Z,OWNER,"The Glitch server logs showed:

> `ERROR: conn=<sqlite3.Connection object at 0x7f88ebadd8f0>, sql = 'select seq, name, file from pragma_database_list() where seq > 0', params = None: no such table: pragma_database_list`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1276#issuecomment-808978808,https://api.github.com/repos/simonw/datasette/issues/1276,808978808,MDEyOklzc3VlQ29tbWVudDgwODk3ODgwOA==,9599,simonw,2021-03-28T23:32:58Z,2021-03-28T23:33:58Z,OWNER,"I just managed to replicate this bug on Glitch: https://nicar-2020.glitch.me/data

> Invalid SQL
> no such table: pragma_database_list

https://nicar-2020.glitch.me/-/versions says:

```json
{
    ""python"": {
        ""version"": ""3.7.10"",
        ""full"": ""3.7.10 (default, Feb 20 2021, 21:21:24) \n[GCC 5.4.0 20160609]""
    },
    ""datasette"": {
        ""version"": ""0.55""
    },
    ""asgi"": ""3.0"",
    ""uvicorn"": ""0.13.4"",
    ""sqlite"": {
        ""version"": ""3.11.0"",
        ""fts_versions"": [
            ""FTS4"",
            ""FTS3""
        ],
        ""extensions"": {
            ""json1"": null
        },
        ""compile_options"": [
            ""ENABLE_COLUMN_METADATA"",
            ""ENABLE_DBSTAT_VTAB"",
            ""ENABLE_FTS3"",
            ""ENABLE_FTS3_PARENTHESIS"",
            ""ENABLE_JSON1"",
            ""ENABLE_LOAD_EXTENSION"",
            ""ENABLE_RTREE"",
            ""ENABLE_UNLOCK_NOTIFY"",
            ""ENABLE_UPDATE_DELETE_LIMIT"",
            ""HAVE_ISNAN"",
            ""LIKE_DOESNT_MATCH_BLOBS"",
            ""MAX_SCHEMA_RETRY=25"",
            ""OMIT_LOOKASIDE"",
            ""SECURE_DELETE"",
            ""SOUNDEX"",
            ""SYSTEM_MALLOC"",
            ""TEMP_STORE=1"",
            ""THREADSAFE=1""
        ]
    }
}
```
That's [SQLite 3.11.0 from 2016-02-15](https://www.sqlite.org/releaselog/3_11_0.html) with no FTS5.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1273#issuecomment-808759984,https://api.github.com/repos/simonw/datasette/issues/1273,808759984,MDEyOklzc3VlQ29tbWVudDgwODc1OTk4NA==,9599,simonw,2021-03-27T16:43:17Z,2021-03-27T16:43:17Z,OWNER,That rivers example in the tutorial would work a lot better with a live demo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890,Refresh SpatiaLite documentation,
https://github.com/simonw/datasette/issues/1273#issuecomment-808756921,https://api.github.com/repos/simonw/datasette/issues/1273,808756921,MDEyOklzc3VlQ29tbWVudDgwODc1NjkyMQ==,9599,simonw,2021-03-27T16:19:45Z,2021-03-27T16:26:28Z,OWNER,"I have a better recipe for using spatial indexes now on https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/
```sql
select
  AsGeoJSON(geometry), *
from
  CPAD_2020a_SuperUnits
where
  PARK_NAME like '%mini%' and
  Intersects(GeomFromGeoJSON(:freedraw), geometry) = 1
  and CPAD_2020a_SuperUnits.rowid in (
    select
      rowid
    from
      SpatialIndex
    where
      f_table_name = 'CPAD_2020a_SuperUnits'
      and search_frame = GeomFromGeoJSON(:freedraw)
  )
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890,Refresh SpatiaLite documentation,
https://github.com/simonw/datasette/issues/1273#issuecomment-808757721,https://api.github.com/repos/simonw/datasette/issues/1273,808757721,MDEyOklzc3VlQ29tbWVudDgwODc1NzcyMQ==,9599,simonw,2021-03-27T16:25:48Z,2021-03-27T16:25:48Z,OWNER,"> This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map.

That should promote `datasette-leaflet-geojson` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838382890,Refresh SpatiaLite documentation,
https://github.com/simonw/datasette/issues/1090#issuecomment-808757659,https://api.github.com/repos/simonw/datasette/issues/1090,808757659,MDEyOklzc3VlQ29tbWVudDgwODc1NzY1OQ==,9599,simonw,2021-03-27T16:25:25Z,2021-03-27T16:25:25Z,OWNER,Related feature request: ability to set default values for canned queries: #1258,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1090#issuecomment-808757155,https://api.github.com/repos/simonw/datasette/issues/1090,808757155,MDEyOklzc3VlQ29tbWVudDgwODc1NzE1NQ==,9599,simonw,2021-03-27T16:21:43Z,2021-03-27T16:21:43Z,OWNER,"Idea for these: imitate https://django-sql-dashboard.readthedocs.io/en/latest/widgets.html#custom-widgets and drive them with templates.

So a custom widget type of `textarea` would look for a template called `widgets/textarea.html` - which means users could define brand new custom widgets just by creating their own template files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1278#issuecomment-808756366,https://api.github.com/repos/simonw/datasette/issues/1278,808756366,MDEyOklzc3VlQ29tbWVudDgwODc1NjM2Ng==,9599,simonw,2021-03-27T16:15:47Z,2021-03-27T16:15:47Z,OWNER,https://timezones-api.datasette.io/ is now up and running on Cloud Run.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842416110,SpatiaLite timezones demo is broken,
https://github.com/simonw/datasette/issues/1278#issuecomment-808652008,https://api.github.com/repos/simonw/datasette/issues/1278,808652008,MDEyOklzc3VlQ29tbWVudDgwODY1MjAwOA==,9599,simonw,2021-03-27T04:47:17Z,2021-03-27T04:47:17Z,OWNER,"https://github.com/simonw/timezones-api is that project, it's pretty old now. I'll try to get it running on Cloud Run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842416110,SpatiaLite timezones demo is broken,
https://github.com/simonw/datasette/issues/1258#issuecomment-808651088,https://api.github.com/repos/simonw/datasette/issues/1258,808651088,MDEyOklzc3VlQ29tbWVudDgwODY1MTA4OA==,9599,simonw,2021-03-27T04:41:52Z,2021-03-27T04:42:14Z,OWNER,"Right now they look like this:
```yaml
databases:
  fixtures:
    queries:
      neighborhood_search:
        params:
        - text
```
In addition to being able to specify defaults, I'd also like to add other things in the future - most significantly the ability to specify a different input widget (e.g. textarea v.s. single-line input)

So maybe this looks like:
```yaml
params:
- name: text
  default: """"
- name: age
  widget: number
```","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/1258#issuecomment-808650266,https://api.github.com/repos/simonw/datasette/issues/1258,808650266,MDEyOklzc3VlQ29tbWVudDgwODY1MDI2Ng==,9599,simonw,2021-03-27T04:37:07Z,2021-03-27T04:37:07Z,OWNER,I like that idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/1249#issuecomment-808649480,https://api.github.com/repos/simonw/datasette/issues/1249,808649480,MDEyOklzc3VlQ29tbWVudDgwODY0OTQ4MA==,9599,simonw,2021-03-27T04:32:10Z,2021-03-27T04:32:10Z,OWNER,I'll close this issue after I ship Datasette 0.56 and confirm that the Dockerfile was correctly built and published to Docker Hub.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-808649322,https://api.github.com/repos/simonw/datasette/issues/1249,808649322,MDEyOklzc3VlQ29tbWVudDgwODY0OTMyMg==,9599,simonw,2021-03-27T04:31:28Z,2021-03-27T04:31:28Z,OWNER,"One last test of that Dockerfile:
```
(datasette) datasette % docker build -f Dockerfile -t datasetteproject/datasette:0.55a --build-arg VERSION=0.55 .
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' | jq
{
  ""python"": {
    ""version"": ""3.9.2"",
    ""full"": ""3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]""
  },
  ""datasette"": {
    ""version"": ""0.55""
  },
  ""asgi"": ""3.0"",
  ""uvicorn"": ""0.13.4"",
  ""sqlite"": {
    ""version"": ""3.27.2"",
    ""fts_versions"": [
      ""FTS5"",
      ""FTS4"",
      ""FTS3""
    ],
    ""extensions"": {
      ""json1"": null
    },
    ""compile_options"": [
      ""COMPILER=gcc-8.3.0"",
      ""ENABLE_COLUMN_METADATA"",
      ""ENABLE_DBSTAT_VTAB"",
      ""ENABLE_FTS3"",
      ""ENABLE_FTS3_PARENTHESIS"",
      ""ENABLE_FTS3_TOKENIZER"",
      ""ENABLE_FTS4"",
      ""ENABLE_FTS5"",
      ""ENABLE_JSON1"",
      ""ENABLE_LOAD_EXTENSION"",
      ""ENABLE_PREUPDATE_HOOK"",
      ""ENABLE_RTREE"",
      ""ENABLE_SESSION"",
      ""ENABLE_STMTVTAB"",
      ""ENABLE_UNLOCK_NOTIFY"",
      ""ENABLE_UPDATE_DELETE_LIMIT"",
      ""HAVE_ISNAN"",
      ""LIKE_DOESNT_MATCH_BLOBS"",
      ""MAX_SCHEMA_RETRY=25"",
      ""MAX_VARIABLE_NUMBER=250000"",
      ""OMIT_LOOKASIDE"",
      ""SECURE_DELETE"",
      ""SOUNDEX"",
      ""TEMP_STORE=1"",
      ""THREADSAFE=1"",
      ""USE_URI""
    ]
  }
}
(datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' --load-extension=spatialite | jq
{
  ""python"": {
    ""version"": ""3.9.2"",
    ""full"": ""3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]""
  },
  ""datasette"": {
    ""version"": ""0.55""
  },
  ""asgi"": ""3.0"",
  ""uvicorn"": ""0.13.4"",
  ""sqlite"": {
    ""version"": ""3.27.2"",
    ""fts_versions"": [
      ""FTS5"",
      ""FTS4"",
      ""FTS3""
    ],
    ""extensions"": {
      ""json1"": null,
      ""spatialite"": ""5.0.1""
    },
    ""compile_options"": [
      ""COMPILER=gcc-8.3.0"",
      ""ENABLE_COLUMN_METADATA"",
      ""ENABLE_DBSTAT_VTAB"",
      ""ENABLE_FTS3"",
      ""ENABLE_FTS3_PARENTHESIS"",
      ""ENABLE_FTS3_TOKENIZER"",
      ""ENABLE_FTS4"",
      ""ENABLE_FTS5"",
      ""ENABLE_JSON1"",
      ""ENABLE_LOAD_EXTENSION"",
      ""ENABLE_PREUPDATE_HOOK"",
      ""ENABLE_RTREE"",
      ""ENABLE_SESSION"",
      ""ENABLE_STMTVTAB"",
      ""ENABLE_UNLOCK_NOTIFY"",
      ""ENABLE_UPDATE_DELETE_LIMIT"",
      ""HAVE_ISNAN"",
      ""LIKE_DOESNT_MATCH_BLOBS"",
      ""MAX_SCHEMA_RETRY=25"",
      ""MAX_VARIABLE_NUMBER=250000"",
      ""OMIT_LOOKASIDE"",
      ""SECURE_DELETE"",
      ""SOUNDEX"",
      ""TEMP_STORE=1"",
      ""THREADSAFE=1"",
      ""USE_URI""
    ]
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1272#issuecomment-808648974,https://api.github.com/repos/simonw/datasette/issues/1272,808648974,MDEyOklzc3VlQ29tbWVudDgwODY0ODk3NA==,9599,simonw,2021-03-27T04:29:42Z,2021-03-27T04:29:42Z,OWNER,"I'm skipping this for the moment because the new Dockerfile shape introduced in https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.

Will need to change that if I want to do this unit tests thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838245338,Unit tests for the Dockerfile,
https://github.com/simonw/datasette/issues/1272#issuecomment-808647937,https://api.github.com/repos/simonw/datasette/issues/1272,808647937,MDEyOklzc3VlQ29tbWVudDgwODY0NzkzNw==,9599,simonw,2021-03-27T04:23:19Z,2021-03-27T04:23:36Z,OWNER,"Part of the challenge here is only running if a Docker daemon is available. I think this pattern works, in `tests/test_dockerfile.py`:
```python
import httpx
import pathlib
import pytest
import subprocess

root = pathlib.Path(__file__).parent.parent

def docker_is_available():
    try:
        client = httpx.Client(
            transport=httpx.HTTPTransport(uds=""/var/run/docker.sock"")
        )
        client.get(""http://docker/info"")
        return True
    except httpx.ConnectError:
        return False


@pytest.fixture
def build_container():
    assert (root / ""Dockerfile"").exists()
    subprocess.check_call([
        ""docker"", ""build"", str(root), ""-t"", ""datasette-dockerfile-test""
    ])


@pytest.mark.skipif(not docker_is_available(),
    reason=""Docker is not available""
)
def test_dockerfile(build_container):
    output = subprocess.check_output([
        ""docker"", ""run"", ""datasette-dockerfile-test"", ""datasette"", ""--get"", ""/_memory?sql=select+1&shape=_array""
    ])
    assert False, ""Implement better assertion here""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838245338,Unit tests for the Dockerfile,
https://github.com/simonw/datasette/issues/1276#issuecomment-808642405,https://api.github.com/repos/simonw/datasette/issues/1276,808642405,MDEyOklzc3VlQ29tbWVudDgwODY0MjQwNQ==,9599,simonw,2021-03-27T03:53:18Z,2021-03-27T03:53:18Z,OWNER,That's really odd. What version of SQLite are you using on the server? You can tell by visiting `https://your-site/-/versions`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841456306,"Invalid SQL: ""no such table: pragma_database_list"" on database page",
https://github.com/simonw/datasette/issues/1277#issuecomment-808641846,https://api.github.com/repos/simonw/datasette/issues/1277,808641846,MDEyOklzc3VlQ29tbWVudDgwODY0MTg0Ng==,9599,simonw,2021-03-27T03:49:34Z,2021-03-27T03:49:34Z,OWNER,"I fixed this already, it's a duplicate of #1239","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842212586,Facet by array breaks if table name contains a space,
https://github.com/simonw/sqlite-utils/issues/252#issuecomment-808302971,https://api.github.com/repos/simonw/sqlite-utils/issues/252,808302971,MDEyOklzc3VlQ29tbWVudDgwODMwMjk3MQ==,9599,simonw,2021-03-26T15:21:38Z,2021-03-26T15:21:38Z,OWNER,Already got that! It's the `--nl` option - works for both importing and exporting data: https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842062949,Support json-line files,
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807647791,https://api.github.com/repos/simonw/sqlite-utils/issues/251,807647791,MDEyOklzc3VlQ29tbWVudDgwNzY0Nzc5MQ==,9599,simonw,2021-03-25T22:42:48Z,2021-03-25T22:44:31Z,OWNER,"Idea: enhance `lambda` to allow it to return a dictionary of values, which will then be used to populate new columns. Use a `--multicolumn` option to indicate this:

    sqlite-utils convert lambda mydb.db mytable mycolumn \
      --code '{""first_name"": value.split()[0], ""last_name"": value.split()[1]}' \
      --multicolumn --drop

The `--drop` means ""drop the `mycolumn` column after making this change"".

Maybe `--multi` is a better name than `--multicolumn` here, since either way it's going to need additional explanation somewhere.

Would this overlap with #239 at all?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807642041,https://api.github.com/repos/simonw/sqlite-utils/issues/251,807642041,MDEyOklzc3VlQ29tbWVudDgwNzY0MjA0MQ==,9599,simonw,2021-03-25T22:39:22Z,2021-03-25T22:39:22Z,OWNER,"Here's the full current implementation of that tool: https://github.com/simonw/sqlite-transform/blob/0.5/sqlite_transform/cli.py

My current plan is to make this functionality available as the following:

    sqlite-utils convert jsonsplit mydb.db mytable mycolumn
    sqlite-utils convert parsedatetime mydb.db mytable mycolumn
    sqlite-utils convert parsedate mydb.db mytable mycolumn
    sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/datasette/issues/741#issuecomment-806166575,https://api.github.com/repos/simonw/datasette/issues/741,806166575,MDEyOklzc3VlQ29tbWVudDgwNjE2NjU3NQ==,9599,simonw,2021-03-24T20:30:33Z,2021-03-24T20:30:33Z,OWNER,"`datasette package` is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects.

I'll make this change to `package` at the same time as I land it for `publish` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/1274#issuecomment-805216038,https://api.github.com/repos/simonw/datasette/issues/1274,805216038,MDEyOklzc3VlQ29tbWVudDgwNTIxNjAzOA==,9599,simonw,2021-03-23T20:14:53Z,2021-03-23T20:14:53Z,OWNER,"Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings.

See #1153 for YAML by default in the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",839008371,Might there be some way to comment metadata.json?,
https://github.com/simonw/datasette/issues/1153#issuecomment-805109341,https://api.github.com/repos/simonw/datasette/issues/1153,805109341,MDEyOklzc3VlQ29tbWVudDgwNTEwOTM0MQ==,9599,simonw,2021-03-23T17:55:48Z,2021-03-23T18:41:57Z,OWNER,"Beginnings of a UI element for switching between them:
```html
<div style=""border: 1px solid rgb(225, 228, 229);
  background-color: rgb(238, 255, 204);
  padding: 0.3em;
  position: relative;
  top: 3px;
  font-family: courier;"">
<a href=""#"" style=""display: inline-block; padding-left: 0px; padding-right: 2em;"">JSON</a>
<a href=""#"" style=""display: inline-block;"">YAML</a>
</div>
```
<img width=""646"" alt=""Metadata_—_Datasette_documentation"" src=""https://user-images.githubusercontent.com/9599/112194637-51f92500-8bc6-11eb-9662-3faa7ef37538.png"">

That `<pre>` has a padding of 12px, so using 12px padding on the tab links should get them to line up better.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805047117,https://api.github.com/repos/simonw/datasette/issues/1153,805047117,MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw==,9599,simonw,2021-03-23T16:30:15Z,2021-03-23T16:46:06Z,OWNER,"https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option.

https://github.com/nodeca/js-yaml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1270#issuecomment-805058241,https://api.github.com/repos/simonw/datasette/issues/1270,805058241,MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ==,9599,simonw,2021-03-23T16:45:39Z,2021-03-23T16:45:39Z,OWNER,"I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1153#issuecomment-805056806,https://api.github.com/repos/simonw/datasette/issues/1153,805056806,MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg==,9599,simonw,2021-03-23T16:43:38Z,2021-03-23T16:43:38Z,OWNER,"I used this code to get that:
```javascript
var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);
div.querySelector('.highlight pre').innerText = jsonVersion;
div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805055291,https://api.github.com/repos/simonw/datasette/issues/1153,805055291,MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ==,9599,simonw,2021-03-23T16:41:31Z,2021-03-23T16:41:31Z,OWNER,"One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:

<img width=""655"" alt=""Metadata_—_Datasette_documentation"" src=""https://user-images.githubusercontent.com/9599/112183909-ef9b2700-8bbb-11eb-882b-aeb387c29e61.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805050163,https://api.github.com/repos/simonw/datasette/issues/1153,805050163,MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw==,9599,simonw,2021-03-23T16:34:35Z,2021-03-23T16:35:32Z,OWNER,"https://docs.datasette.io/en/stable/metadata.html has this example:
```yaml
title: Demonstrating Metadata from YAML
description_html: |-
  <p>This description includes a long HTML string</p>
  <ul>
    <li>YAML is better for embedding HTML strings than JSON!</li>
  </ul>
license: ODbL
license_url: https://opendatacommons.org/licenses/odbl/
databases:
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search
```

I ran this in the browser dev tools:
```javascript
var s = document.createElement('script')
s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'
document.head.appendChild(s)
var yamlExample = document.querySelector('.highlight-yaml').textContent);
console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))
```
And got:
```json
{
    ""title"": ""Demonstrating Metadata from YAML"",
    ""description_html"": ""<p>This description includes a long HTML string</p>\n<ul>\n  <li>YAML is better for embedding HTML strings than JSON!</li>\n</ul>"",
    ""license"": ""ODbL"",
    ""license_url"": ""https://opendatacommons.org/licenses/odbl/"",
    ""databases"": {
        ""fixtures"": {
            ""tables"": {
                ""no_primary_key"": {
                    ""hidden"": true
                }
            },
            ""queries"": {
                ""neighborhood_search"": {
                    ""sql"": ""select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;"",
                    ""title"": ""Search neighborhoods"",
                    ""description_html"": ""<p>This demonstrates <em>basic</em> LIKE search""
                }
            }
        }
    }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805042880,https://api.github.com/repos/simonw/datasette/issues/1153,805042880,MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA==,9599,simonw,2021-03-23T16:24:32Z,2021-03-23T16:24:32Z,OWNER,... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805041522,https://api.github.com/repos/simonw/datasette/issues/1153,805041522,MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg==,9599,simonw,2021-03-23T16:22:46Z,2021-03-23T16:22:46Z,OWNER,"That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1249#issuecomment-805033155,https://api.github.com/repos/simonw/datasette/issues/1249,805033155,MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ==,9599,simonw,2021-03-23T16:12:13Z,2021-03-23T16:12:13Z,OWNER,"Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source

> The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful [VirtualKNN module](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN)), take a look at the [Datasette Dockerfile](https://github.com/simonw/datasette/blob/master/Dockerfile).

See also #1273","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/163#issuecomment-804540869,https://api.github.com/repos/simonw/datasette/issues/163,804540869,MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ==,9599,simonw,2021-03-23T02:44:33Z,2021-03-23T02:44:33Z,OWNER,Comments welcome!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,Document the querystring argument for setting a different time limit,
https://github.com/simonw/datasette/pull/1271#issuecomment-804299406,https://api.github.com/repos/simonw/datasette/issues/1271,804299406,MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg==,9599,simonw,2021-03-22T18:36:14Z,2021-03-22T21:49:27Z,OWNER,"This isn't actually working - the outer code attempts to send an `.interrupt()` call to the connection object via the `connections` thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.

Need to figure out how to communicate with that thread properly.

Also a test that fails in this particular case would be a good idea!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/issues/1249#issuecomment-804406675,https://api.github.com/repos/simonw/datasette/issues/1249,804406675,MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ==,9599,simonw,2021-03-22T21:26:27Z,2021-03-22T21:26:27Z,OWNER,(Without the `apt-get update ...` SpatiaLite line it's 125MB),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804404544,https://api.github.com/repos/simonw/datasette/issues/1249,804404544,MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA==,9599,simonw,2021-03-22T21:22:56Z,2021-03-22T21:24:24Z,OWNER,"Final version of Dockerfile which installs the specified version from GitHub:

    docker build . -t datasette-spatialite --build-arg VERSION=0.55

```dockerfile
FROM python:3.9.2-slim-buster as build

# Version of Datasette to install, e.g. 0.55
#   docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION

# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt-get remove -y software-properties-common && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD [""datasette""]
```
Run against 0.55 this produces an image of 262MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804384196,https://api.github.com/repos/simonw/datasette/issues/1249,804384196,MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng==,9599,simonw,2021-03-22T20:48:46Z,2021-03-22T20:48:46Z,OWNER,I think part of the reason it's smaller is that I ran `pip install datasette` instead of using `COPY . /datasette` followed by `pip install /datasette`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804380181,https://api.github.com/repos/simonw/datasette/issues/1249,804380181,MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ==,9599,simonw,2021-03-22T20:42:16Z,2021-03-22T20:42:16Z,OWNER,"Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804379644,https://api.github.com/repos/simonw/datasette/issues/1249,804379644,MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA==,9599,simonw,2021-03-22T20:41:23Z,2021-03-22T20:41:23Z,OWNER,I tried adding `apt-get remove -y software-properties-common &&` to remove `software-properties-common` but it made no difference to the image size.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804372977,https://api.github.com/repos/simonw/datasette/issues/1249,804372977,MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw==,9599,simonw,2021-03-22T20:30:37Z,2021-03-22T20:30:37Z,OWNER,"I tried copying just the `mod_spatialite.so` file into a second stage build but it failed. So I ran `bash` in a working image and used `ldd` to figure out what it was linked to:
```
root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
	linux-vdso.so.1 (0x00007ffd021f4000)
	libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
	libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
	librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
	libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
	libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
	libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
	libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
	libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
	libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
	libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
	libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
	libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
	libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
	libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
	libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
	libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
	libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
	libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
	libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
	librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
	libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
	libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
	libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
	libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
	libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
	libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
	libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804368372,https://api.github.com/repos/simonw/datasette/issues/1249,804368372,MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg==,9599,simonw,2021-03-22T20:22:43Z,2021-03-22T20:22:43Z,OWNER,"```dockerfile
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD [""datasette""]
```
262 MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804363687,https://api.github.com/repos/simonw/datasette/issues/1249,804363687,MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw==,9599,simonw,2021-03-22T20:15:00Z,2021-03-22T20:15:00Z,OWNER,"```
RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r
```
That dropped it to 265MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804360701,https://api.github.com/repos/simonw/datasette/issues/1249,804360701,MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ==,9599,simonw,2021-03-22T20:10:07Z,2021-03-22T20:10:07Z,OWNER,Adding `--no-install-recommends` dropped it to 275MB,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804347152,https://api.github.com/repos/simonw/datasette/issues/1249,804347152,MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg==,9599,simonw,2021-03-22T19:47:56Z,2021-03-22T19:48:03Z,OWNER,I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804344553,https://api.github.com/repos/simonw/datasette/issues/1249,804344553,MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw==,9599,simonw,2021-03-22T19:43:25Z,2021-03-22T19:43:25Z,OWNER,Does `--no-install-recommends` make a difference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804338678,https://api.github.com/repos/simonw/datasette/issues/1249,804338678,MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA==,9599,simonw,2021-03-22T19:33:43Z,2021-03-22T19:33:43Z,OWNER,"Replacing `rm -rf /var/lib/{apt,dpkg,cache,log}/` with 
```
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg
```
Got the size down to 305MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804318314,https://api.github.com/repos/simonw/datasette/issues/1249,804318314,MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA==,9599,simonw,2021-03-22T19:04:30Z,2021-03-22T19:04:30Z,OWNER,Considering the image on Docker Hub right now is `383MB` this is actually an improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804317545,https://api.github.com/repos/simonw/datasette/issues/1249,804317545,MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ==,9599,simonw,2021-03-22T19:03:22Z,2021-03-22T19:03:22Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

EXPOSE 8001
CMD [""datasette""]
```
Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804310353,https://api.github.com/repos/simonw/datasette/issues/1249,804310353,MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw==,9599,simonw,2021-03-22T18:52:12Z,2021-03-22T18:52:12Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build

# Setup build dependencies
RUN apt update \
    && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
    libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
    && apt clean

RUN wget ""https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz"" && tar xzf sqlite-autoconf-3340100.tar.gz \
    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
    CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
    && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz"" && tar zxf freexl-1.0.6.tar.gz \
    && cd freexl-1.0.6 && ./configure && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz"" && tar zxf libspatialite-5.0.1.tar.gz \
    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz"" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz"" && tar zxf spatialite-tools-5.0.0.tar.gz \
    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install

# Add local code to the image instead of fetching from pypi.
#COPY . /datasette
#RUN pip install /datasette
RUN pip install datasette

FROM python:3.9.2-slim-buster

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
Produced a 448MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804309510,https://api.github.com/repos/simonw/datasette/issues/1249,804309510,MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA==,9599,simonw,2021-03-22T18:50:50Z,2021-03-22T18:50:50Z,OWNER,"Ideally I'd like to use the Debian stable `python:3.9.2-slim-buster` base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7

This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/pull/1271#issuecomment-804265042,https://api.github.com/repos/simonw/datasette/issues/1271,804265042,MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg==,9599,simonw,2021-03-22T17:45:45Z,2021-03-22T17:45:45Z,OWNER,"I can remove this code too: 
https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L190-L192","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/issues/1249#issuecomment-804263434,https://api.github.com/repos/simonw/datasette/issues/1249,804263434,MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA==,9599,simonw,2021-03-22T17:43:25Z,2021-03-22T17:43:25Z,OWNER,I figured out the cause of the hang in #1268 - it was caused by `select count(*) from SpatialIndex` interacting badly with the `set_progress_handler()` mechanism I was using to implement query time limits. #1271 has a replacement for that using `asyncio.wait_for()` and `conn.interrupt()` which should resolve the SpatiaLite issue too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1268#issuecomment-804261915,https://api.github.com/repos/simonw/datasette/issues/1268,804261915,MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ==,9599,simonw,2021-03-22T17:41:12Z,2021-03-22T17:41:12Z,OWNER,"Closing this because I've figured out the root of the problem now, and I have a potential solution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1269#issuecomment-804261610,https://api.github.com/repos/simonw/datasette/issues/1269,804261610,MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA==,9599,simonw,2021-03-22T17:40:41Z,2021-03-22T17:40:41Z,OWNER,"#1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty `count(*)` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,Don't attempt to run count(*) against virtual tables,
https://github.com/simonw/datasette/issues/1270#issuecomment-804255633,https://api.github.com/repos/simonw/datasette/issues/1270,804255633,MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw==,9599,simonw,2021-03-22T17:32:02Z,2021-03-22T17:32:08Z,OWNER,Confirmed that the `interrupt()` based cancellation mechanism fixes the SpatiaLite issue in #1268!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1270#issuecomment-803834784,https://api.github.com/repos/simonw/datasette/issues/1270,803834784,MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA==,9599,simonw,2021-03-22T07:31:57Z,2021-03-22T16:22:19Z,OWNER,"I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157

I figured out a similar pattern in `datasette-ripgrep` here: https://github.com/simonw/datasette-ripgrep/blob/0.7/datasette_ripgrep/__init__.py#L63-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1268#issuecomment-803802957,https://api.github.com/repos/simonw/datasette/issues/1268,803802957,MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw==,9599,simonw,2021-03-22T06:38:14Z,2021-03-22T06:38:14Z,OWNER,"Also worth trying is to change this code:
```python
     n = 1000 
     if ms < 50: 
         n = 1 
```
What happens with `n = 10` instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1269#issuecomment-803785808,https://api.github.com/repos/simonw/datasette/issues/1269,803785808,MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA==,9599,simonw,2021-03-22T06:00:53Z,2021-03-22T06:00:53Z,OWNER,This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,Don't attempt to run count(*) against virtual tables,
https://github.com/simonw/datasette/issues/1268#issuecomment-803784902,https://api.github.com/repos/simonw/datasette/issues/1268,803784902,MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg==,9599,simonw,2021-03-22T05:59:06Z,2021-03-22T05:59:06Z,OWNER,"Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing `select count(*) from SpatialIndex`.

That `interrupt` timeout mechanism is worth digging into further.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803782705,https://api.github.com/repos/simonw/datasette/issues/1268,803782705,MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ==,9599,simonw,2021-03-22T05:54:19Z,2021-03-22T05:54:19Z,OWNER,"Got two new TILs out of this:

* [Tracing every executed Python statement](https://til.simonwillison.net/python/tracing-every-statement)
* [Running gdb against a Python process in a running Docker container](https://til.simonwillison.net/docker/gdb-python-docker)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803777724,https://api.github.com/repos/simonw/datasette/issues/1268,803777724,MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA==,9599,simonw,2021-03-22T05:42:50Z,2021-03-22T05:43:23Z,OWNER,"<img width=""1129"" alt=""tuscany_housenumbers__select___from_sqlite_master_where_sql_like__create_virtual_table__"" src=""https://user-images.githubusercontent.com/9599/111945249-99809380-8a96-11eb-871b-9a1509360bcf.png"">

If I want to avoid counting virtual tables, I need to detect which tables are virtual tables.

The safest way to do this is probably to pull the `sql` for every table and then, in Python, check for values that start with `create virtual table` after converting to lower case, using any number of spaces.

This would catch things like ` CREATE       virtual      TABLE` which might be missed by a SQL `like` query.   ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803775121,https://api.github.com/repos/simonw/datasette/issues/1268,803775121,MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ==,9599,simonw,2021-03-22T05:36:26Z,2021-03-22T05:36:26Z,OWNER,So one fix could be to avoid running counts for anything that turns out to be a virtual table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774926,https://api.github.com/repos/simonw/datasette/issues/1268,803774926,MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg==,9599,simonw,2021-03-22T05:35:56Z,2021-03-22T05:35:56Z,OWNER,That's in this code here: https://github.com/simonw/datasette/blob/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b/datasette/database.py#L221-L241,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774518,https://api.github.com/repos/simonw/datasette/issues/1268,803774518,MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA==,9599,simonw,2021-03-22T05:34:57Z,2021-03-22T05:34:57Z,OWNER,"... and sure enough, adding this code fixed the problem:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 3579cce..b466b12 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -224,6 +226,9 @@ class Database:
         # Try to get counts for each table, $limit timeout for each count
         counts = {}
         for table in await self.table_names():
+            if table == ""SpatialIndex"":
+                counts[table] = 0
+                continue
             try:
                 table_count = (
                     await self.execute(
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803773484,https://api.github.com/repos/simonw/datasette/issues/1268,803773484,MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA==,9599,simonw,2021-03-22T05:32:29Z,2021-03-22T05:32:29Z,OWNER,"To figure out which SQL query triggers the problem I added this code to write to a log file:
```python
            with sqlite_timelimit(conn, time_limit_ms):
                try:
                    cursor = conn.cursor()
                    with open(""/tmp/sql.log"", ""ab"", buffering=0) as fp:
                        fp.write((""{}: {}\n"".format(sql, params)).encode(""utf-8""))
                    cursor.execute(sql, params if params is not None else {})
```
I had to use `ab` binary mode because Python doesn't allow `buffering=0` for non-binary file operations.

With the log enabled, I used `docker exec -it 589ae68de943 bash` to attach to the running container and `tail -f /tmp/sql.log` to see the logs. Here's where it broke:

```
select count(*) from [idx_civici_geom_parent]: None
select count(*) from [sqlite_stat1]: None
select count(*) from [sqlite_stat3]: None
select count(*) from [SpatialIndex]: None
```
So attempting to run a `count(*)` against the `SpatialIndex` virtual table is the thing that triggers the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764919,https://api.github.com/repos/simonw/datasette/issues/1268,803764919,MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ==,9599,simonw,2021-03-22T05:11:11Z,2021-03-22T05:11:11Z,OWNER,"Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?

https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764200,https://api.github.com/repos/simonw/datasette/issues/1268,803764200,MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA==,9599,simonw,2021-03-22T05:09:13Z,2021-03-22T05:09:13Z,OWNER,"I tried building a container where the `conn.set_progress_handler(handler, n)` line was commented out... and it fixed the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762969,https://api.github.com/repos/simonw/datasette/issues/1268,803762969,MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ==,9599,simonw,2021-03-22T05:05:51Z,2021-03-22T05:05:51Z,OWNER,I had to run `docker kill 16197781a7b5` to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762609,https://api.github.com/repos/simonw/datasette/issues/1268,803762609,MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ==,9599,simonw,2021-03-22T05:05:00Z,2021-03-22T05:05:00Z,OWNER,"Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run `gdb`. I had to use `--privileged` here because otherwise `gdb` showed a ""Could not attach to process"" error.
```
docker exec --privileged -it 16197781a7b5 bash
# apt-get install gdb python3-dbg
# gdb /usr/bin/python3 -p 20
```
This paused the process. I tried running this:
```
(gdb) py-bt
Traceback (most recent call first):
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 1845, in _run_once
    if handle._cancelled:
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 570, in run_forever
    self._run_once()
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 603, in run_until_complete
    self.run_forever()
  File ""/usr/local/lib/python3.8/dist-packages/uvicorn/server.py"", line 49, in run
    loop.run_until_complete(self.serve(sockets=sockets))
  File ""/usr/local/lib/python3.8/dist-packages/uvicorn/main.py"", line 386, in run
    server.run()
  File ""/usr/local/lib/python3.8/dist-packages/datasette/cli.py"", line 575, in serve
    uvicorn.run(ds.app(), **uvicorn_kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/bin/datasette"", line 8, in <module>
    sys.exit(cli())
  <built-in method exec of module object at remote 0x7f0981a280e0>
  File ""/usr/lib/python3.8/trace.py"", line 450, in runctx
    exec(cmd, globals, locals)
  File ""/usr/lib/python3.8/trace.py"", line 6632, in main
  File ""/usr/lib/python3.8/trace.py"", line 756, in <module>
    main()
  <built-in method exec of module object at remote 0x7f0981a280e0>
  File ""/usr/lib/python3.8/runpy.py"", line 343, in _run_code
  File ""/usr/lib/python3.8/runpy.py"", line 450, in _run_module_as_main
```
Not sure if that's useful or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803759051,https://api.github.com/repos/simonw/datasette/issues/1268,803759051,MDEyOklzc3VlQ29tbWVudDgwMzc1OTA1MQ==,9599,simonw,2021-03-22T04:55:22Z,2021-03-22T04:55:22Z,OWNER,So I think there's a bug in the way the `set_progress_handler()` mechanism works when used in conjunction with SpatiaLite 5.0 on Linux.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758793,https://api.github.com/repos/simonw/datasette/issues/1268,803758793,MDEyOklzc3VlQ29tbWVudDgwMzc1ODc5Mw==,9599,simonw,2021-03-22T04:54:32Z,2021-03-22T04:54:32Z,OWNER,"Hitting http://localhost:8001/tuscany_housenumbers triggers the bug. It gets stuck in a loop that looks like this:

<img width=""381"" alt=""datasette_—_root_16197781a7b5____—_com_docker_cli_◂_docker_run_-it_-p_8001_8001_-v___Dropbox_Development_datasette__mnt_datasette-spatialite_latest_bash_—_195×48_and_getIncidentsGit_—_-zsh_—_162×60"" src=""https://user-images.githubusercontent.com/9599/111942115-e0b75600-8a8f-11eb-8b6d-5c37240ce2f8.png"">

Which looks to me like this code: https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L139-L158","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758182,https://api.github.com/repos/simonw/datasette/issues/1268,803758182,MDEyOklzc3VlQ29tbWVudDgwMzc1ODE4Mg==,9599,simonw,2021-03-22T04:52:15Z,2021-03-22T04:52:15Z,OWNER,Hitting http://localhost:8001/ successfully shows the homepage (after a lot more scrolling).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803757746,https://api.github.com/repos/simonw/datasette/issues/1268,803757746,MDEyOklzc3VlQ29tbWVudDgwMzc1Nzc0Ng==,9599,simonw,2021-03-22T04:50:40Z,2021-03-22T04:51:52Z,OWNER,"Here's a fun debugging trick:

    docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
    root@16197781a7b5:/# python3 -m trace --trace $(which datasette) \
      -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite \
      --load-extension=spatialite

A huge amount of stuff scrolls past as Datasette starts up, since we are tracing every executed line of Python.

After about a minute it's finished starting and gets to this point:

```
selectors.py(452):             if timeout is None:
selectors.py(454):             elif timeout <= 0:
selectors.py(459):                 timeout = math.ceil(timeout * 1e3) * 1e-3
selectors.py(464):             max_ev = max(len(self._fd_to_key), 1)
selectors.py(466):             ready = []
selectors.py(467):             try:
selectors.py(468):                 fd_event_list = self._selector.poll(timeout, max_ev)
```
Now  I can make some HTTP requests against it.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803756495,https://api.github.com/repos/simonw/datasette/issues/1268,803756495,MDEyOklzc3VlQ29tbWVudDgwMzc1NjQ5NQ==,9599,simonw,2021-03-22T04:46:04Z,2021-03-22T04:46:04Z,OWNER,`gdb` may be able to help debug this: https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1249#issuecomment-803755698,https://api.github.com/repos/simonw/datasette/issues/1249,803755698,MDEyOklzc3VlQ29tbWVudDgwMzc1NTY5OA==,9599,simonw,2021-03-22T04:43:02Z,2021-03-22T04:43:02Z,OWNER,I'll spin off a separate ticket to investigate the hang.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1267#issuecomment-803754226,https://api.github.com/repos/simonw/datasette/issues/1267,803754226,MDEyOklzc3VlQ29tbWVudDgwMzc1NDIyNg==,9599,simonw,2021-03-22T04:37:26Z,2021-03-22T04:37:26Z,OWNER,"Thanks for doing this - I've used alternativeto.net a bunch in the past, it's great to see Datasette listed there.

This does raise some interesting philosophical questions: three years into the project I'm still not entirely sure what Datasette competes with! Could be SQLite desktop packages, could be visualization software like Tableau, could even be something like Airtable (given a few more plugins).

It will be interesting to see how the alternativeto listing evolves, maybe it will help me answer that question!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837208901,Update Datasette alternativeto listening with details,
https://github.com/simonw/datasette/issues/1249#issuecomment-803753388,https://api.github.com/repos/simonw/datasette/issues/1249,803753388,MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA==,9599,simonw,2021-03-22T04:34:20Z,2021-03-22T04:35:10Z,OWNER,"Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load...
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
(Which produced a 550MB image)

And when I ran Datasette I got that same error where the database listing page hangs!
```
docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite --load-extension=spatialite
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803751068,https://api.github.com/repos/simonw/datasette/issues/1249,803751068,MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA==,9599,simonw,2021-03-22T04:26:45Z,2021-03-22T04:26:45Z,OWNER,"Here's why:
```
datasette % docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash                                                      
root@3430352ff378:/# datasette
bash: /usr/local/bin/datasette: /usr/bin/python3: bad interpreter: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750617,https://api.github.com/repos/simonw/datasette/issues/1249,803750617,MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw==,9599,simonw,2021-03-22T04:25:14Z,2021-03-22T04:25:14Z,OWNER,"Got this error attempting to run Datasette (with or without SpatiaLite):
```
standard_init_linux.go:219: exec user process caused: no such file or directory
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750399,https://api.github.com/repos/simonw/datasette/issues/1249,803750399,MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ==,9599,simonw,2021-03-22T04:24:25Z,2021-03-22T04:24:25Z,OWNER,"I'll try using `ubuntu:20.10` for everything:
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10 as build

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM ubuntu:20.10

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803749831,https://api.github.com/repos/simonw/datasette/issues/1249,803749831,MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ==,9599,simonw,2021-03-22T04:22:35Z,2021-03-22T04:22:35Z,OWNER,"I tried copying just the `mod_spatialite.so` file:
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
But when I ran Datasette with `--load-extension=spatialite` I got this:
```
  File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 151, in in_thread
    self.ds._prepare_connection(conn, self.name)
  File ""/usr/local/lib/python3.9/site-packages/datasette/app.py"", line 502, in _prepare_connection
    conn.execute(f""SELECT load_extension('{extension}')"")
sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748469,https://api.github.com/repos/simonw/datasette/issues/1249,803748469,MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ==,9599,simonw,2021-03-22T04:17:51Z,2021-03-22T04:17:51Z,OWNER,"... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:

```
datasette % docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  File ""/usr/local/lib/python3.9/sqlite3/dbapi2.py"", line 27, in <module>
    from _sqlite3 import *
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/lib/x86_64-linux-gnu/libsqlite3.so.0)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748158,https://api.github.com/repos/simonw/datasette/issues/1249,803748158,MDEyOklzc3VlQ29tbWVudDgwMzc0ODE1OA==,9599,simonw,2021-03-22T04:16:57Z,2021-03-22T04:16:57Z,OWNER,"Which is great, because the image on Docker Hub right now is 383MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803747701,https://api.github.com/repos/simonw/datasette/issues/1249,803747701,MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ==,9599,simonw,2021-03-22T04:15:40Z,2021-03-22T04:15:40Z,OWNER,"Here's a trick: install SpatiaLite in `ubuntu:20.10` and then copy it into the final `python:3.9.2-slim` image.

```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
That produced a 265MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700940,https://api.github.com/repos/simonw/datasette/issues/1249,803700940,MDEyOklzc3VlQ29tbWVudDgwMzcwMDk0MA==,9599,simonw,2021-03-22T01:14:24Z,2021-03-22T01:14:24Z,OWNER,I tried that with just `python3-pip` (removing `libsqlite3-mod-spatialite`) and got 435MB.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700626,https://api.github.com/repos/simonw/datasette/issues/1249,803700626,MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg==,9599,simonw,2021-03-22T01:13:04Z,2021-03-22T01:13:04Z,OWNER,"Building a Dockerfile containing just `FROM ubuntu:20.10` gave me `79.5MB`.

Building this one:
```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && \
    apt install -y python3-pip libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/
```
Resulted in a 515MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698983,https://api.github.com/repos/simonw/datasette/issues/1249,803698983,MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw==,9599,simonw,2021-03-22T01:05:36Z,2021-03-22T01:06:23Z,OWNER,"It's pretty big though. I tried this version which avoids copying junk from my laptop in:

```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

RUN pip install datasette

EXPOSE 8001
CMD [""datasette""]
```
And got this:
```
datasette % docker images datasette-spatialite    
REPOSITORY             TAG       IMAGE ID       CREATED         SIZE
datasette-spatialite   latest    0796950653c2   2 seconds ago   528MB
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698168,https://api.github.com/repos/simonw/datasette/issues/1249,803698168,MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA==,9599,simonw,2021-03-22T01:02:02Z,2021-03-22T01:02:30Z,OWNER,"This is the shortest Dockerfile that appeared to give me a working SpatiaLite module:
```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

# Add local code to the image instead of fetching from pypi.
COPY . /datasette

RUN pip install /datasette

RUN rm -rf /datasette

EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697546,https://api.github.com/repos/simonw/datasette/issues/1249,803697546,MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng==,9599,simonw,2021-03-22T00:59:47Z,2021-03-22T00:59:47Z,OWNER,"To debug I'm running:

    docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash

This gets me a shell I can use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697211,https://api.github.com/repos/simonw/datasette/issues/1249,803697211,MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ==,9599,simonw,2021-03-22T00:58:01Z,2021-03-22T00:58:01Z,OWNER,"I'm messing around with the `Dockerfile` and after each change I'm running:

    docker build . -t datasette-spatialite

And then:

    docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694661,https://api.github.com/repos/simonw/datasette/issues/1249,803694661,MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ==,9599,simonw,2021-03-22T00:46:49Z,2021-03-22T00:46:49Z,OWNER,Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694436,https://api.github.com/repos/simonw/datasette/issues/1249,803694436,MDEyOklzc3VlQ29tbWVudDgwMzY5NDQzNg==,9599,simonw,2021-03-22T00:46:00Z,2021-03-22T00:46:00Z,OWNER,So I'm going to try `20.10` and see where that gets me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694359,https://api.github.com/repos/simonw/datasette/issues/1249,803694359,MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ==,9599,simonw,2021-03-22T00:45:47Z,2021-03-22T00:45:47Z,OWNER,https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using `python:3.9-slim-buster` or `ubuntu:20.04` - but 20.04 is focal which still has SpatiaLite `4.3.0a-6build1` - It's `20.10` that has 5.0: https://packages.ubuntu.com/groovy/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803693181,https://api.github.com/repos/simonw/datasette/issues/1249,803693181,MDEyOklzc3VlQ29tbWVudDgwMzY5MzE4MQ==,9599,simonw,2021-03-22T00:41:02Z,2021-03-22T00:41:02Z,OWNER,Debian sid has it too: https://packages.debian.org/sid/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803692673,https://api.github.com/repos/simonw/datasette/issues/1249,803692673,MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw==,9599,simonw,2021-03-22T00:38:42Z,2021-03-22T00:38:42Z,OWNER,Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803691236,https://api.github.com/repos/simonw/datasette/issues/1249,803691236,MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg==,9599,simonw,2021-03-22T00:32:03Z,2021-03-22T00:32:03Z,OWNER,"Here's something odd: when I run `datasette tuscany_housenumbers.sqlite --load-extension=spatialite` on macOS against SpatiaLite installed using Homebrew (which reports `""spatialite"": ""5.0.0""` on the `/-/versions` page) I don't get any weird errors at all, everything works fine.

But when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages.

This is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1259#issuecomment-803674728,https://api.github.com/repos/simonw/datasette/issues/1259,803674728,MDEyOklzc3VlQ29tbWVudDgwMzY3NDcyOA==,9599,simonw,2021-03-21T22:55:31Z,2021-03-21T22:55:31Z,OWNER,CTEs were added in 2014-02-03 SQLite 3.8.3 - so I think it's OK to depend on them for Datasette.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/647#issuecomment-803673225,https://api.github.com/repos/simonw/datasette/issues/647,803673225,MDEyOklzc3VlQ29tbWVudDgwMzY3MzIyNQ==,9599,simonw,2021-03-21T22:44:19Z,2021-03-21T22:44:19Z,OWNER,"Now that I'm looking at refactoring how views work in #878 it's clear that the gnarliest, most convoluted code I need to deal with relates to this old feature.

I'm going to remove it entirely. Any performance enhancement or provides can be achieved just as well by using regular URLs and a caching proxy.

I may provide a 404 handling plugin that attempts to rewrite old URLs that used this mechanism, but I won't do any more than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin,
https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803501756,https://api.github.com/repos/simonw/sqlite-utils/issues/249,803501756,MDEyOklzc3VlQ29tbWVudDgwMzUwMTc1Ng==,9599,simonw,2021-03-21T02:33:45Z,2021-03-21T02:33:45Z,OWNER,"Did you run `enable-fts` before you inserted the data?

If so you'll need to run `populate-fts` after the insert to populate the FTS index.

A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836963850,Full text search possibly broken?,
https://github.com/simonw/datasette/issues/878#issuecomment-803473015,https://api.github.com/repos/simonw/datasette/issues/878,803473015,MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ==,9599,simonw,2021-03-20T22:33:05Z,2021-03-20T22:33:05Z,OWNER,"Things this mechanism needs to be able to support:

- Returning a default JSON representation
- Defining ""extra"" JSON representations blocks, which can be requested using `?_extra=`
- Returning rendered HTML, based on the default JSON + one or more extras + a template
- Using Datasette output renderers to return e.g. CSV data
- Potentially also supporting streaming output renderers for streaming CSV/TSV/JSON-nl etc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803472595,https://api.github.com/repos/simonw/datasette/issues/878,803472595,MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ==,9599,simonw,2021-03-20T22:28:12Z,2021-03-20T22:28:12Z,OWNER,"Another idea I had: a view is a class that takes the `datasette` instance in its constructor, and defines a `__call__` method that accepts a request and returns a response. Except `await __call__` looks like it might be a bit messy, discussion in https://github.com/encode/starlette/issues/886","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803472278,https://api.github.com/repos/simonw/datasette/issues/878,803472278,MDEyOklzc3VlQ29tbWVudDgwMzQ3MjI3OA==,9599,simonw,2021-03-20T22:25:04Z,2021-03-20T22:25:04Z,OWNER,"I came up with a slightly wild idea for this that would involve pytest-style dependency injection.

Prototype here: https://gist.github.com/simonw/496b24fdad44f6f8b7237fe394a0ced7

Copying from my private notes:

> Use the lazy evaluated DI mechanism to break up table view into different pieces eg for faceting
> 
> Use that to solve JSON vs HTML views
> 
> Oh here's an idea: what if the various components of the table view were each defined  as async functions.... and then executed using asyncio.gather in order to run the SQL queries in parallel? Then try benchmarking with different numbers of threads?
> 
> The async_call_with_arguments function could do this automatically for any awaitable dependencies
> 
> This would give me massively parallel dependency injection
> 
> (I could build an entire framework around this and call it c64)
> 
> Idea: arguments called eg ""count"" are executed and the result passed to the function. If called count_fn then a reference to the not-yet-called function is passed instead 
> 
> I'm not going to completely combine the views mechanism and the render hooks. Instead, the core view will define a bunch of functions used to compose the page and the render hook will have conditional access to those functions - which will otherwise be asyncio.gather executed directly by the HTML page version
> 
> Using asyncio.gather to execute facets and suggest facets in parallel would be VERY interesting 
> 
> suggest facets should be VERY cachable - doesn't matter if it's wrong unlike actual facets themselves
> 
> What if all Datasette views were defined in terms of dependency injection - and those dependency functions could themselves depend on others just like pytest fixtures. Everything would become composable and async stuff could execute in parallel
> 
> FURTHER IDEA: use this for the ?_extra= mechanism as well.
> 
> Any view in Datasette can be defined as a collection of named keys. Each of those keys maps to a function or an async function that accepts as input other named keys, using DI to handle them.
> 
> The HTML view is a defined function. So are the other outputs.
> 
> Default original inputs include “request” and “datasette”.
> 
> So… maybe a view function is a class methods that use DI. One of those methods as an .html() method used for the default page.
> 
> Output formats are a bit more complicated because they are supposed to be defined separately in plugins. They are unified across query, row and table though.
> 
> I’m going to try breaking up the TableView to see what happens.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803471917,https://api.github.com/repos/simonw/datasette/issues/878,803471917,MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw==,9599,simonw,2021-03-20T22:21:33Z,2021-03-20T22:21:33Z,OWNER,"This has been blocking things for too long.

If this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/1258#issuecomment-803471702,https://api.github.com/repos/simonw/datasette/issues/1258,803471702,MDEyOklzc3VlQ29tbWVudDgwMzQ3MTcwMg==,9599,simonw,2021-03-20T22:19:39Z,2021-03-20T22:19:39Z,OWNER,"This is a good idea. I avoided this initially because it should be possible to run a canned query with a parameter set to the empty string, but that view could definitely be smart enough to differentiate between `?sql=...&param=` and `?sql=` with no `param` specified at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/782#issuecomment-803469623,https://api.github.com/repos/simonw/datasette/issues/782,803469623,MDEyOklzc3VlQ29tbWVudDgwMzQ2OTYyMw==,9599,simonw,2021-03-20T22:01:23Z,2021-03-20T22:01:23Z,OWNER,"I'm going to keep `?_shape=array` working on the assumption that many existing uses of the Datasette API are already using that option, so it would be nice not to break them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1261#issuecomment-803468314,https://api.github.com/repos/simonw/datasette/issues/1261,803468314,MDEyOklzc3VlQ29tbWVudDgwMzQ2ODMxNA==,9599,simonw,2021-03-20T21:48:48Z,2021-03-20T21:48:48Z,OWNER,That's fixed in this release of `datasette-publish-vercel`: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.9.2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/1261#issuecomment-803466868,https://api.github.com/repos/simonw/datasette/issues/1261,803466868,MDEyOklzc3VlQ29tbWVudDgwMzQ2Njg2OA==,9599,simonw,2021-03-20T21:36:06Z,2021-03-20T21:36:06Z,OWNER,"This isn't a Datasette bug - it's a Vercel bug: https://github.com/simonw/datasette-publish-vercel/issues/28

I'm looking at a fix for that now, so watch that issue for updates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/1266#issuecomment-803466730,https://api.github.com/repos/simonw/datasette/issues/1266,803466730,MDEyOklzc3VlQ29tbWVudDgwMzQ2NjczMA==,9599,simonw,2021-03-20T21:35:00Z,2021-03-20T21:35:00Z,OWNER,https://docs.datasette.io/en/latest/internals.html#returning-a-response-with-asgi-send-send,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836273891,Documentation for Response.asgi_send(send) method,
https://github.com/simonw/datasette/issues/1265#issuecomment-803130332,https://api.github.com/repos/simonw/datasette/issues/1265,803130332,MDEyOklzc3VlQ29tbWVudDgwMzEzMDMzMg==,9599,simonw,2021-03-19T21:03:09Z,2021-03-19T21:03:09Z,OWNER,This is now available in `datasette-auth-passwords` 0.4! https://github.com/simonw/datasette-auth-passwords/releases/tag/0.4,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",836123030,Support for HTTP Basic Authentication,
https://github.com/simonw/datasette/issues/1262#issuecomment-802099264,https://api.github.com/repos/simonw/datasette/issues/1262,802099264,MDEyOklzc3VlQ29tbWVudDgwMjA5OTI2NA==,9599,simonw,2021-03-18T16:43:09Z,2021-03-18T16:43:09Z,OWNER,"I often find myself wanting this too, when I'm exploring a new dataset.

i agree with Bob that this is a good candidate for a plugin. The plugin system isn't quite setup for this yet though - there isn't an obvious mechanism for adding extra sort orders or other interface elements that manipulate the query used by the table view in some way.

I'm going to promote this issue to status of a plugin hook feature request - I have a hunch that a plugin hook that enables `order by random()` could enable a lot of other useful plugin features too.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",834602299,Plugin hook that could support 'order by random()' for table view,
https://github.com/simonw/sqlite-utils/issues/246#issuecomment-799479175,https://api.github.com/repos/simonw/sqlite-utils/issues/246,799479175,MDEyOklzc3VlQ29tbWVudDc5OTQ3OTE3NQ==,9599,simonw,2021-03-15T14:47:31Z,2021-03-15T14:47:31Z,OWNER,"This is a smart feature. I have something that does this in Datasette, extracting it out to `sqlite-utils` makes a lot of sense.

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L818-L829","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831751367,Escaping FTS search strings,
https://github.com/simonw/datasette/issues/236#issuecomment-799066252,https://api.github.com/repos/simonw/datasette/issues/236,799066252,MDEyOklzc3VlQ29tbWVudDc5OTA2NjI1Mg==,9599,simonw,2021-03-15T03:34:52Z,2021-03-15T03:34:52Z,OWNER,"Yeah the Lambda Docker stuff is pretty odd - you still don't get to speak HTTP, you have to speak their custom event protocol instead.

https://github.com/glassechidna/serverlessish looks interesting here - it adds a proxy inside the container which allows your existing HTTP Docker image to run within Docker-on-Lambda. I've not tried it out yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/issues/1259#issuecomment-797827038,https://api.github.com/repos/simonw/datasette/issues/1259,797827038,MDEyOklzc3VlQ29tbWVudDc5NzgyNzAzOA==,9599,simonw,2021-03-13T00:15:40Z,2021-03-13T00:15:40Z,OWNER,"If all of the facets were being calculated in a single query, I'd be willing to bump the facet time limit up to something a lot higher, maybe even a full second. There's a chance that could work amazingly well with a materialized CTE.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/1259#issuecomment-797804869,https://api.github.com/repos/simonw/datasette/issues/1259,797804869,MDEyOklzc3VlQ29tbWVudDc5NzgwNDg2OQ==,9599,simonw,2021-03-12T23:05:05Z,2021-03-12T23:05:05Z,OWNER,"I wonder if I could optimize facet suggestion in the same way?

One challenge: the query time limit will apply to the full CTE query, not to the individual columns.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/1259#issuecomment-797801075,https://api.github.com/repos/simonw/datasette/issues/1259,797801075,MDEyOklzc3VlQ29tbWVudDc5NzgwMTA3NQ==,9599,simonw,2021-03-12T22:53:56Z,2021-03-12T22:55:16Z,OWNER,"OK, a better comparison:

https://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28*%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28*%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aowner+as+%28%0D%0Aselect%0D%0A++%27owner%27+as+col%2C+owner+as+value%2C+count%28*%29+as+c+from+data+group+by+owner%0D%0A++order+by+c+desc+limit+31%0D%0A%29%0D%0Aselect+*+from+primary_fuel+union+select+*+from+country_long%0D%0Aunion+select+*+from+owner+order+by+col%2C+c+desc calculates facets against three columns. It takes **78.5ms** (and 34.5ms when I refreshed it, presumably after warming some SQLite caches of some sort).

https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet=country_long&_facet=primary_fuel&_trace=1&_size=0 shows those facets with size=0 on the SQL query - and shows a SQL trace at the bottom of the page.

The country_long facet query takes 45.36ms, owner takes 38.45ms, primary_fuel takes 49.04ms - so a total of 132.85ms

That's against https://global-power-plants.datasettes.com/-/versions says SQLite 3.27.3 - so even on a SQLite version that doesn't materialize the CTEs there's a significant performance boost to doing all three facets in a single CTE query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/1259#issuecomment-797790017,https://api.github.com/repos/simonw/datasette/issues/1259,797790017,MDEyOklzc3VlQ29tbWVudDc5Nzc5MDAxNw==,9599,simonw,2021-03-12T22:22:12Z,2021-03-12T22:22:12Z,OWNER,"https://sqlite.org/lang_with.html

>  Prior to SQLite 3.35.0, all CTEs where treated as if the NOT MATERIALIZED phrase was present

It looks like this optimization is completely unavailable on SQLite prior to 3.35.0 (released 12th March 2021). But I could still rewrite the faceting to work in this way, using the exact same SQL - it would just be significantly faster on 3.35.0+ (assuming it's actually faster in practice - would need to benchmark).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/1193#issuecomment-797159434,https://api.github.com/repos/simonw/datasette/issues/1193,797159434,MDEyOklzc3VlQ29tbWVudDc5NzE1OTQzNA==,9599,simonw,2021-03-12T01:01:54Z,2021-03-12T01:01:54Z,OWNER,"DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python

```python
import duckdb
con = duckdb.connect(database=""/tmp/blah.db"", read_only=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787173276,Research plugin hook for alternative database backends,
https://github.com/simonw/datasette/issues/1250#issuecomment-797159221,https://api.github.com/repos/simonw/datasette/issues/1250,797159221,MDEyOklzc3VlQ29tbWVudDc5NzE1OTIyMQ==,9599,simonw,2021-03-12T01:01:17Z,2021-03-12T01:01:17Z,OWNER,This is a duplicate of #1193.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824067604,Research: Plugin hook for alternative database connections,
https://github.com/simonw/datasette/issues/670#issuecomment-797158641,https://api.github.com/repos/simonw/datasette/issues/670,797158641,MDEyOklzc3VlQ29tbWVudDc5NzE1ODY0MQ==,9599,simonw,2021-03-12T00:59:49Z,2021-03-12T00:59:49Z,OWNER,"> Challenge: what's the equivalent for PostgreSQL of opening a database in read only mode? Will I have to talk users through creating read only credentials?

It looks like the answer to this is yes - I'll need users to setup read-only credentials. Here's a TIL about that: https://til.simonwillison.net/postgresql/read-only-postgresql-user","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL,
https://github.com/simonw/datasette/pull/1211#issuecomment-796854370,https://api.github.com/repos/simonw/datasette/issues/1211,796854370,MDEyOklzc3VlQ29tbWVudDc5Njg1NDM3MA==,9599,simonw,2021-03-11T16:15:29Z,2021-03-11T16:15:29Z,OWNER,Thanks very much for this - it's really comprehensive. I need to bake some of these patterns into my coding habits better!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797649915,Use context manager instead of plain open,
https://github.com/simonw/datasette/issues/838#issuecomment-795918377,https://api.github.com/repos/simonw/datasette/issues/838,795918377,MDEyOklzc3VlQ29tbWVudDc5NTkxODM3Nw==,9599,simonw,2021-03-10T19:01:48Z,2021-03-10T19:01:48Z,OWNER,The biggest challenge here I think is to replicate the exact situation here this happens in a Python unit test. The fix should be easy once we have a test in place.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/838#issuecomment-795895436,https://api.github.com/repos/simonw/datasette/issues/838,795895436,MDEyOklzc3VlQ29tbWVudDc5NTg5NTQzNg==,9599,simonw,2021-03-10T18:44:46Z,2021-03-10T18:44:57Z,OWNER,Let's reopen this.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/pull/1254#issuecomment-795870524,https://api.github.com/repos/simonw/datasette/issues/1254,795870524,MDEyOklzc3VlQ29tbWVudDc5NTg3MDUyNA==,9599,simonw,2021-03-10T18:27:45Z,2021-03-10T18:27:45Z,OWNER,What other breaks did you spot?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/pull/1256#issuecomment-795869144,https://api.github.com/repos/simonw/datasette/issues/1256,795869144,MDEyOklzc3VlQ29tbWVudDc5NTg2OTE0NA==,9599,simonw,2021-03-10T18:26:46Z,2021-03-10T18:26:46Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",827341657,Minor type in IP adress,
https://github.com/simonw/datasette/pull/1254#issuecomment-794439632,https://api.github.com/repos/simonw/datasette/issues/1254,794439632,MDEyOklzc3VlQ29tbWVudDc5NDQzOTYzMg==,9599,simonw,2021-03-09T20:53:02Z,2021-03-09T20:53:02Z,OWNER,Thanks for catching that documentation update!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/pull/1254#issuecomment-794437715,https://api.github.com/repos/simonw/datasette/issues/1254,794437715,MDEyOklzc3VlQ29tbWVudDc5NDQzNzcxNQ==,9599,simonw,2021-03-09T20:51:19Z,2021-03-09T20:51:19Z,OWNER,Did you see my note on https://github.com/simonw/datasette/issues/1249#issuecomment-792384382 about a weird issue I was having with the `/dbname` page hanging the server? Have you seen anything like that in your work here?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/issues/1250#issuecomment-792386484,https://api.github.com/repos/simonw/datasette/issues/1250,792386484,MDEyOklzc3VlQ29tbWVudDc5MjM4NjQ4NA==,9599,simonw,2021-03-08T00:29:06Z,2021-03-08T00:29:06Z,OWNER,"DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python

```python
import duckdb
con = duckdb.connect(database=""/tmp/blah.db"", read_only=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824067604,Research: Plugin hook for alternative database connections,
https://github.com/simonw/datasette/issues/1248#issuecomment-792385274,https://api.github.com/repos/simonw/datasette/issues/1248,792385274,MDEyOklzc3VlQ29tbWVudDc5MjM4NTI3NA==,9599,simonw,2021-03-08T00:25:10Z,2021-03-08T00:25:10Z,OWNER,"It's not possible yet, unfortunately. This came up on the forums recently: https://github.com/simonw/datasette/discussions/968

I'm leaning further towards making the database connection layer itself work via a plugin hook, which would open up the possibility of supporting DuckDB and other databases as well. I've not committed to doing this yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",823035080,duckdb database (very low performance in SQLite),
https://github.com/simonw/datasette/issues/1249#issuecomment-792384854,https://api.github.com/repos/simonw/datasette/issues/1249,792384854,MDEyOklzc3VlQ29tbWVudDc5MjM4NDg1NA==,9599,simonw,2021-03-08T00:23:38Z,2021-03-08T00:23:38Z,OWNER,One reason to prioritize this issue: Homebrew upgraded to SpatiaLite 5.0 recently https://formulae.brew.sh/formula/spatialite-tools and as a result SpatiaLite database created on my laptop don't appear to be compatible with Datasette when published using `datasette publish`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-792384382,https://api.github.com/repos/simonw/datasette/issues/1249,792384382,MDEyOklzc3VlQ29tbWVudDc5MjM4NDM4Mg==,9599,simonw,2021-03-08T00:22:02Z,2021-03-08T00:22:02Z,OWNER,"I tried this patch against `Dockerfile`:
```diff
diff --git a/Dockerfile b/Dockerfile
index f4b1414..dd659e1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,25 +1,26 @@
-FROM python:3.7.10-slim-stretch as build
+FROM python:3.9.2-slim-buster as build
 
 # Setup build dependencies
 RUN apt update \
-&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
- && apt clean
+  && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
+  libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
+  && apt clean
 
-
-RUN wget ""https://www.sqlite.org/2020/sqlite-autoconf-3310100.tar.gz"" && tar xzf sqlite-autoconf-3310100.tar.gz \
-    && cd sqlite-autoconf-3310100 && ./configure --disable-static --enable-fts5 --enable-json1 CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
+RUN wget ""https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz"" && tar xzf sqlite-autoconf-3340100.tar.gz \
+    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
+    CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
     && make && make install
 
-RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-sources/freexl-1.0.5.tar.gz"" && tar zxf freexl-1.0.5.tar.gz \
-    && cd freexl-1.0.5 && ./configure && make && make install
+RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz"" && tar zxf freexl-1.0.6.tar.gz \
+    && cd freexl-1.0.6 && ./configure && make && make install
 
-RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-4.4.0-RC0.tar.gz"" && tar zxf libspatialite-4.4.0-RC0.tar.gz \
-    && cd libspatialite-4.4.0-RC0 && ./configure && make && make install
+RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz"" && tar zxf libspatialite-5.0.1.tar.gz \
+    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install
 
 RUN wget ""http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz"" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install
 
-RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/spatialite-tools-4.4.0-RC0.tar.gz"" && tar zxf spatialite-tools-4.4.0-RC0.tar.gz \
-    && cd spatialite-tools-4.4.0-RC0 && ./configure && make && make install
+RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz"" && tar zxf spatialite-tools-5.0.0.tar.gz \
+    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install
 
 
 # Add local code to the image instead of fetching from pypi.
@@ -27,7 +28,7 @@ COPY . /datasette
 
 RUN pip install /datasette
 
-FROM python:3.7.10-slim-stretch
+FROM python:3.9.2-slim-buster
 
 # Copy python dependencies and spatialite libraries
 COPY --from=build /usr/local/lib/ /usr/local/lib/
```

I had to use `--disable-rttopo` from the tip in https://github.com/OSGeo/gdal/pull/3443 and also needed to install `libminizip-dev`.

This works, sort of... I'm getting a weird issue where the `/dbname` page is hanging some of the time instead of loading correctly. Other than that it seems to work, but a hanging page is bad!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-792383956,https://api.github.com/repos/simonw/datasette/issues/1249,792383956,MDEyOklzc3VlQ29tbWVudDc5MjM4Mzk1Ng==,9599,simonw,2021-03-08T00:20:09Z,2021-03-08T00:20:09Z,OWNER,"Worth noting that the Docker image used by `datasette publish cloudrun` doesn't actually use that Datasette docker image - it does this:

https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L349-L353

Where the apt extras for SpatiaLite are: https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L344-L345

`libsqlite3-mod-spatialite` against that official `python:3.8` image doesn't appear to install SpatiaLite 5.0.


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/pull/1223#issuecomment-792233255,https://api.github.com/repos/simonw/datasette/issues/1223,792233255,MDEyOklzc3VlQ29tbWVudDc5MjIzMzI1NQ==,9599,simonw,2021-03-07T07:41:01Z,2021-03-07T07:41:01Z,OWNER,"This is fantastic, thanks so much for tracking this down.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",806918878,Add compile option to Dockerfile to fix failing test (fixes #696),
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790695126,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790695126,MDEyOklzc3VlQ29tbWVudDc5MDY5NTEyNg==,9599,simonw,2021-03-04T15:20:42Z,2021-03-04T15:20:42Z,MEMBER,"I'm not sure why but my most recent import, when displayed in Datasette, looks like this:

<img width=""574"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109985836-0ab00080-7cba-11eb-97d5-0631a0835b61.png"">

Sorting by `id` in the opposite order gives me the data I would expect - so it looks like a bunch of null/blank messages are being imported at some point and showing up first due to ID ordering.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790693674,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790693674,MDEyOklzc3VlQ29tbWVudDc5MDY5MzY3NA==,9599,simonw,2021-03-04T15:18:36Z,2021-03-04T15:18:36Z,MEMBER,"I imported my 10GB mbox with 750,000 emails in it, ran this tool (with a hacked fix for the blob column problem) - and now a search that returns 92 results takes 25.37ms! This is fantastic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790669767,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790669767,MDEyOklzc3VlQ29tbWVudDc5MDY2OTc2Nw==,9599,simonw,2021-03-04T14:46:06Z,2021-03-04T14:46:06Z,MEMBER,"Solution could be to pre-process that string by splitting on `(` and dropping everything afterwards, assuming that the `(...)` bit isn't necessary for correctly parsing the date.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790668263,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790668263,MDEyOklzc3VlQ29tbWVudDc5MDY2ODI2Mw==,9599,simonw,2021-03-04T14:43:58Z,2021-03-04T14:43:58Z,MEMBER,"I added this code to output a message ID on errors:
```diff
             print(""Errors: {}"".format(num_errors))
             print(traceback.format_exc())
+            print(""Message-Id: {}"".format(email.get(""Message-Id"", ""None"")))
             continue
```
Having found a message ID that had an error, I ran this command to see the context:

    rg --text --context 20 '44F289B0.000001.02100@SCHWARZE-DWFXMI' ~/gmail.mbox

This was for the following error:
```
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 102, in get_mbox
    message[""date""] = get_message_date(email.get(""Date""), email.get_from())
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 178, in get_message_date
    datetime_tuple = email.utils.parsedate_tz(mail_date)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 50, in parsedate_tz
    res = _parsedate_tz(data)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 69, in _parsedate_tz
    data = data.split()
AttributeError: 'Header' object has no attribute 'split'
```
Here's what I spotted in the `ripgrep` output:
```
177133570:Message-Id: <44F289B0.000001.02100@SCHWARZE-DWFXMI>
177133571-Date: Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop�ische Sommerzeit)
177133572-X-Mailer: IncrediMail (5002253)
```
So it could it be that `_parsedate_tz` is having trouble with that `Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop�ische Sommerzeit)` string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/6#issuecomment-790384087,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6,790384087,MDEyOklzc3VlQ29tbWVudDc5MDM4NDA4Nw==,9599,simonw,2021-03-04T07:22:51Z,2021-03-04T07:22:51Z,MEMBER,#3 also mentions the conflicting version with other tools.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",821841046,Upgrade to latest sqlite-utils,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790380839,MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ==,9599,simonw,2021-03-04T07:17:05Z,2021-03-04T07:17:05Z,MEMBER,"Looks like you're doing this:
```python
    elif message.get_content_type() == ""text/plain"":
        body = message.get_payload(decode=True)
```
So presumably that decodes to a unicode string?

I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790379629,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790379629,MDEyOklzc3VlQ29tbWVudDc5MDM3OTYyOQ==,9599,simonw,2021-03-04T07:14:41Z,2021-03-04T07:14:41Z,MEMBER,"Confirmed: removing the `len()` call does not speed things up, so it's reading through the entire file for some other purpose too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790378658,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790378658,MDEyOklzc3VlQ29tbWVudDc5MDM3ODY1OA==,9599,simonw,2021-03-04T07:12:48Z,2021-03-04T07:12:48Z,MEMBER,"It looks like the `body` is being loaded into a BLOB column - so in Datasette default it looks like this:

<img width=""1650"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109924808-b4b96980-7c75-11eb-8c9e-307f2ae32d5a.png"">

If I `datasette install datasette-render-binary` and then try again I get this:

<img width=""1487"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109924944-ea5e5280-7c75-11eb-9a32-404f3d68455f.png"">

It would be great if we could store the `body` as unicode text instead. May have to do something clever to decode it based on some kind of charset header?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790373024,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790373024,MDEyOklzc3VlQ29tbWVudDc5MDM3MzAyNA==,9599,simonw,2021-03-04T07:01:58Z,2021-03-04T07:04:06Z,MEMBER,"I got 9 warnings that look like this:
```
Errors: 1
Traceback (most recent call last):
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 103, in get_mbox
    message[""date""] = get_message_date(email.get(""Date""), email.get_from())
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 167, in get_message_date
    datetime_tuple = email.utils.parsedate_tz(mail_date)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 50, in parsedate_tz
    res = _parsedate_tz(data)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 69, in _parsedate_tz
    data = data.split()
AttributeError: 'Header' object has no attribute 'split'
```
It would be useful if those warnings told me the message ID (or similar) of the affected message so I could grep for it in the `mbox` and see what was going on.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790372621,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790372621,MDEyOklzc3VlQ29tbWVudDc5MDM3MjYyMQ==,9599,simonw,2021-03-04T07:01:18Z,2021-03-04T07:01:18Z,MEMBER,"I'm not sure if it would work, but there is an alternative pattern for showing a progress bar against a really large file that I've used in `healthkit-to-sqlite` - you set the progress bar size to the size of the file in bytes, then update a counter as you read the file.

https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/cli.py#L24-L57 and https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/utils.py#L4-L19 (the `progress_callback()` bit) is where that happens.

It can be a bit of a convoluted pattern, and I'm not at all sure it would work for `mbox` files since it looks like that library has other reasons it needs to do a file scan rather than streaming it through one chunk of bytes at a time. So I imagine this would not work here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790370485,MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ==,9599,simonw,2021-03-04T06:57:25Z,2021-03-04T06:57:48Z,MEMBER,"The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:

https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67

I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790369076,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790369076,MDEyOklzc3VlQ29tbWVudDc5MDM2OTA3Ng==,9599,simonw,2021-03-04T06:54:46Z,2021-03-04T06:54:46Z,MEMBER,"The Rich-powered progress bar is pretty:

![rich](https://user-images.githubusercontent.com/9599/109923307-71f69200-7c73-11eb-9ee2-8f0a240f3994.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790312268,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790312268,MDEyOklzc3VlQ29tbWVudDc5MDMxMjI2OA==,9599,simonw,2021-03-04T05:48:16Z,2021-03-04T05:48:16Z,MEMBER,"Wow, my mbox is a 10.35 GB download!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/pull/1243#issuecomment-790311215,https://api.github.com/repos/simonw/datasette/issues/1243,790311215,MDEyOklzc3VlQ29tbWVudDc5MDMxMTIxNQ==,9599,simonw,2021-03-04T05:45:57Z,2021-03-04T05:45:57Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815955014,fix small typo,
https://github.com/simonw/datasette/issues/1247#issuecomment-787616446,https://api.github.com/repos/simonw/datasette/issues/1247,787616446,MDEyOklzc3VlQ29tbWVudDc4NzYxNjQ0Ng==,9599,simonw,2021-03-01T03:50:37Z,2021-03-01T03:50:37Z,OWNER,"I like the `.add_memory_database()` option. I also like that it makes it more obvious that this is a capability of Datasette, since I'm excited to see more plugins, features and tests that take advantage of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",818430405,datasette.add_memory_database() method,
https://github.com/simonw/datasette/issues/1247#issuecomment-787616158,https://api.github.com/repos/simonw/datasette/issues/1247,787616158,MDEyOklzc3VlQ29tbWVudDc4NzYxNjE1OA==,9599,simonw,2021-03-01T03:49:27Z,2021-03-01T03:49:27Z,OWNER,"A couple of options:
```python
datasette.add_memory_database(""test_json_array"")
# or make that first argument to add_database() optional and support:
datasette.add_database(memory_name=""test_json_array"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",818430405,datasette.add_memory_database() method,
https://github.com/simonw/datasette/issues/1246#issuecomment-787611153,https://api.github.com/repos/simonw/datasette/issues/1246,787611153,MDEyOklzc3VlQ29tbWVudDc4NzYxMTE1Mw==,9599,simonw,2021-03-01T03:30:57Z,2021-03-01T03:30:57Z,OWNER,"I'm going to try a new pattern for testing this, enabled by #1151 - the test will create a new named in-memory database, write some records to it and then run some test facets against that. This will save me from having to add yet another fixtures table for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817597268,Suggest for ArrayFacet possibly confused by blank values,
https://github.com/simonw/datasette/issues/1005#issuecomment-787536267,https://api.github.com/repos/simonw/datasette/issues/1005,787536267,MDEyOklzc3VlQ29tbWVudDc4NzUzNjI2Nw==,9599,simonw,2021-02-28T22:30:37Z,2021-02-28T22:30:37Z,OWNER,It's out! https://github.com/encode/httpx/releases/tag/0.17.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718259202,Remove xfail tests when new httpx is released,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787532279,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787532279,MDEyOklzc3VlQ29tbWVudDc4NzUzMjI3OQ==,9599,simonw,2021-02-28T22:09:37Z,2021-02-28T22:09:37Z,OWNER,Microsoft's playwright Python library solves this problem by code generating both their sync AND their async libraries https://github.com/microsoft/playwright-python/tree/master/scripts,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787198202,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787198202,MDEyOklzc3VlQ29tbWVudDc4NzE5ODIwMg==,9599,simonw,2021-02-27T22:33:58Z,2021-02-27T22:33:58Z,OWNER,"Hah or use this trick, which genuinely rewrites the code at runtime using a class decorator! https://github.com/python-happybase/aiohappybase/blob/0990ef45cfdb720dc987afdb4957a0fac591cb99/aiohappybase/sync/_util.py#L19-L32","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787195536,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787195536,MDEyOklzc3VlQ29tbWVudDc4NzE5NTUzNg==,9599,simonw,2021-02-27T22:13:24Z,2021-02-27T22:13:24Z,OWNER,Some other interesting background reading: https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html - in particular see how SQLALchemy has a `await conn.run_sync(meta.drop_all)` mechanism for running methods that haven't themselves been provided in an async version,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787190562,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787190562,MDEyOklzc3VlQ29tbWVudDc4NzE5MDU2Mg==,9599,simonw,2021-02-27T22:04:00Z,2021-02-27T22:04:00Z,OWNER,"From the poster here: https://github.com/sethmlarson/pycon-async-sync-poster/blob/master/poster.pdf

<img width=""624"" alt=""pycon-async-sync-poster_poster_pdf_at_master_·_sethmlarson_pycon-async-sync-poster"" src=""https://user-images.githubusercontent.com/9599/109401634-9f0a1400-7904-11eb-8b3a-37df0678b8dc.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787186826,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787186826,MDEyOklzc3VlQ29tbWVudDc4NzE4NjgyNg==,9599,simonw,2021-02-27T22:01:54Z,2021-02-27T22:01:54Z,OWNER,`unasync` is an implementation of the exact pattern I was talking about above - it uses the `tokenize` module from the Python standard library to apply some clever rules to transform an async codebase into a sync one. https://unasync.readthedocs.io/en/latest/ - implementation here: https://github.com/python-trio/unasync/blob/v0.5.0/src/unasync/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787175126,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787175126,MDEyOklzc3VlQ29tbWVudDc4NzE3NTEyNg==,9599,simonw,2021-02-27T21:55:05Z,2021-02-27T21:55:05Z,OWNER,"""how to use some new tools to more easily maintain a codebase that supports both async and synchronous I/O and multiple async libraries"" - yeah that's exactly what I need, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787144523,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787144523,MDEyOklzc3VlQ29tbWVudDc4NzE0NDUyMw==,9599,simonw,2021-02-27T21:18:46Z,2021-02-27T21:18:46Z,OWNER,"Here's a really wild idea: I wonder if it would be possible to run a source transformation against either the sync or the async versions of the code to produce the equivalent for the other paradigm?

Could that even be as simple as a set of regular expressions against the `await ...` version that strips out or replaces the `await` and `async def` and `async for` statements?

If so... I could maintain just the async version, generate the sync version with a script and rely on robust unit testing to guarantee that this actually works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787142066,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787142066,MDEyOklzc3VlQ29tbWVudDc4NzE0MjA2Ng==,9599,simonw,2021-02-27T21:17:10Z,2021-02-27T21:17:10Z,OWNER,"I have a hunch this is actually going to be quite difficult, due to the internal complexity of some of the `sqlite-utils` API methods.

Consider `db[table].extract(...)` for example. It does a whole bunch of extra queries inside the method - each of those would need to be turned into an `await` call for the async version. Here's the method body today:

https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1060-L1152

Writing this method twice - looking similar but with `await ...` tucked in before every internal method it calls that needs to execute SQL - is going to be pretty messy.

One thing that would help a LOT is figuring out how to share the majority of the test code. If the exact same tests could run against both the sync and async versions with a bit of test trickery, maintaining parallel implementations would at least be a bit more feasible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787120136,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787120136,MDEyOklzc3VlQ29tbWVudDc4NzEyMDEzNg==,9599,simonw,2021-02-27T19:04:47Z,2021-02-27T19:04:47Z,OWNER,"Another option here would be to add https://github.com/omnilib/aiosqlite/blob/main/aiosqlite/core.py as a dependency - it's four years old now and actively marinated, and the code is pretty small so it looks like a solid, stable, reliable dependency.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787118691,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787118691,MDEyOklzc3VlQ29tbWVudDc4NzExODY5MQ==,9599,simonw,2021-02-27T18:53:23Z,2021-02-27T18:53:23Z,OWNER,"Datasette has its own implementation of a write queue for exactly this purpose - and there's no reason at all that should stay in Datasette rather than being extracted out and moved over here to `sqlite-utils`.

One small concern I have is around the API design. I'd want to keep supporting the existing synchronous API while also providing a similar API with await-based methods.

What are some good examples of libraries that do this? I like how https://www.python-httpx.org/ handles it, maybe that's a good example to imitate?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-786925280,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,786925280,MDEyOklzc3VlQ29tbWVudDc4NjkyNTI4MA==,9599,simonw,2021-02-26T22:23:10Z,2021-02-26T22:23:10Z,MEMBER,"Thanks!

I requested my Gmail export from takeout - once that arrives I'll test it against this and then merge the PR.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/issues/1238#issuecomment-786849095,https://api.github.com/repos/simonw/datasette/issues/1238,786849095,MDEyOklzc3VlQ29tbWVudDc4Njg0OTA5NQ==,9599,simonw,2021-02-26T19:29:38Z,2021-02-26T19:29:38Z,OWNER,"Here's the test I wrote:
```diff
git diff tests/test_custom_pages.py
diff --git a/tests/test_custom_pages.py b/tests/test_custom_pages.py
index 6a23192..5a71f56 100644
--- a/tests/test_custom_pages.py
+++ b/tests/test_custom_pages.py
@@ -2,11 +2,19 @@ import pathlib
 import pytest
 from .fixtures import make_app_client
 
+TEST_TEMPLATE_DIRS = str(pathlib.Path(__file__).parent / ""test_templates"")
+
 
 @pytest.fixture(scope=""session"")
 def custom_pages_client():
+    with make_app_client(template_dir=TEST_TEMPLATE_DIRS) as client:
+        yield client
+
+
+@pytest.fixture(scope=""session"")
+def custom_pages_client_with_base_url():
     with make_app_client(
-        template_dir=str(pathlib.Path(__file__).parent / ""test_templates"")
+        template_dir=TEST_TEMPLATE_DIRS, config={""base_url"": ""/prefix/""}
     ) as client:
         yield client
 
@@ -23,6 +31,12 @@ def test_request_is_available(custom_pages_client):
     assert ""path:/request"" == response.text
 
 
+def test_custom_pages_with_base_url(custom_pages_client_with_base_url):
+    response = custom_pages_client_with_base_url.get(""/prefix/request"")
+    assert 200 == response.status
+    assert ""path:/prefix/request"" == response.text
+
+
 def test_custom_pages_nested(custom_pages_client):
     response = custom_pages_client.get(""/nested/nest"")
     assert 200 == response.status
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-786848654,https://api.github.com/repos/simonw/datasette/issues/1238,786848654,MDEyOklzc3VlQ29tbWVudDc4Njg0ODY1NA==,9599,simonw,2021-02-26T19:28:48Z,2021-02-26T19:28:48Z,OWNER,"I added a debug line just before `for regex, wildcard_template` here:

https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1148-L1155

And it showed that for some reason `request.path` is `/prefix/prefix/request` here - the prefix got doubled somehow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-786841261,https://api.github.com/repos/simonw/datasette/issues/1238,786841261,MDEyOklzc3VlQ29tbWVudDc4Njg0MTI2MQ==,9599,simonw,2021-02-26T19:13:44Z,2021-02-26T19:13:44Z,OWNER,Sounds like a bug - thanks for reporting this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1246#issuecomment-786840734,https://api.github.com/repos/simonw/datasette/issues/1246,786840734,MDEyOklzc3VlQ29tbWVudDc4Njg0MDczNA==,9599,simonw,2021-02-26T19:12:39Z,2021-02-26T19:12:47Z,OWNER,"Could I take this part:
```python
             suggested_facet_sql = """""" 
                 select distinct json_type({column}) 
                 from ({sql}) 
             """""".format( 
                 column=escape_sqlite(column), sql=self.sql 
             ) 
```
And add `where {column} is not null and {column} != ''` perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817597268,Suggest for ArrayFacet possibly confused by blank values,
https://github.com/simonw/datasette/issues/1246#issuecomment-786840425,https://api.github.com/repos/simonw/datasette/issues/1246,786840425,MDEyOklzc3VlQ29tbWVudDc4Njg0MDQyNQ==,9599,simonw,2021-02-26T19:11:56Z,2021-02-26T19:11:56Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/facets.py#L271-L295,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817597268,Suggest for ArrayFacet possibly confused by blank values,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786830832,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786830832,MDEyOklzc3VlQ29tbWVudDc4NjgzMDgzMg==,9599,simonw,2021-02-26T18:52:40Z,2021-02-26T18:52:40Z,OWNER,"Could this handle lists of objects too? That would be pretty amazing - if the column has a `[{...}, {...}]` list in it could turn that into a many-to-many.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/datasette/issues/1240#issuecomment-786813506,https://api.github.com/repos/simonw/datasette/issues/1240,786813506,MDEyOklzc3VlQ29tbWVudDc4NjgxMzUwNg==,9599,simonw,2021-02-26T18:19:46Z,2021-02-26T18:19:46Z,OWNER,"Linking to rows from custom queries is a lot harder - because given an arbitrary string of SQL it's difficult to analyze it and figure out which (if any) of the returned columns represent a primary key.

It's possible to manually write a SQL query that returns a column that will be treated as a link to another page using this plugin, but it's not particularly straight-forward: https://datasette.io/plugins/datasette-json-html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814591962,Allow facetting on custom queries,
https://github.com/simonw/datasette/issues/1240#issuecomment-786812716,https://api.github.com/repos/simonw/datasette/issues/1240,786812716,MDEyOklzc3VlQ29tbWVudDc4NjgxMjcxNg==,9599,simonw,2021-02-26T18:18:18Z,2021-02-26T18:18:18Z,OWNER,"Agreed, this would be extremely useful. I'd love to be able to facet against custom queries. It's a fair bit of work to implement but it's not impossible. Closing this as a duplicate of #972.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",814591962,Allow facetting on custom queries,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786795132,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786795132,MDEyOklzc3VlQ29tbWVudDc4Njc5NTEzMg==,9599,simonw,2021-02-26T17:45:53Z,2021-02-26T17:45:53Z,OWNER,If there's no primary key in the JSON could use the `hash_id` mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786794435,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786794435,MDEyOklzc3VlQ29tbWVudDc4Njc5NDQzNQ==,9599,simonw,2021-02-26T17:44:38Z,2021-02-26T17:44:38Z,OWNER,This came up in office hours!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/datasette/issues/1244#issuecomment-786786645,https://api.github.com/repos/simonw/datasette/issues/1244,786786645,MDEyOklzc3VlQ29tbWVudDc4Njc4NjY0NQ==,9599,simonw,2021-02-26T17:30:38Z,2021-02-26T17:30:38Z,OWNER,"New paragraph at the top of https://docs.datasette.io/en/latest/writing_plugins.html

> Want to start by looking at an example? The [Datasette plugins directory](https://datasette.io/plugins) lists more than 50 open source plugins with code you can explore. The [plugin hooks](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hooks) page includes links to example plugins for each of the documented hooks.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817528452,Plugin tip: look at the examples linked from the hooks page,
https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786050562,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786050562,MDEyOklzc3VlQ29tbWVudDc4NjA1MDU2Mg==,9599,simonw,2021-02-25T16:57:56Z,2021-02-25T16:57:56Z,OWNER,"`sqlite-utils create-view` currently has a `--ignore` option, so adding that to `sqlite-utils drop-view` and `sqlite-utils drop-table` makes sense as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore",
https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786049686,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786049686,MDEyOklzc3VlQ29tbWVudDc4NjA0OTY4Ng==,9599,simonw,2021-02-25T16:56:42Z,2021-02-25T16:56:42Z,OWNER,"So:
```python
    db[""my_table""].drop(ignore=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore",
https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786049394,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786049394,MDEyOklzc3VlQ29tbWVudDc4NjA0OTM5NA==,9599,simonw,2021-02-25T16:56:14Z,2021-02-25T16:56:14Z,OWNER,"Other methods (`db.create_view()` for example) have `ignore=True` to mean ""don't throw an error if this causes a problem"", so I'm good with adding that to `.drop_view()`.

I don't like using it as the default partly because that would be a very minor breaking API change, but mainly because I don't want to hide mistakes people make - e.g. if you mistype the name of the table you are trying to drop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore",
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786037219,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786037219,MDEyOklzc3VlQ29tbWVudDc4NjAzNzIxOQ==,9599,simonw,2021-02-25T16:39:23Z,2021-02-25T16:39:23Z,OWNER,"Example from the docs:
```pycon
>>> db = sqlite_utils.Database(memory=True)
>>> db[""dogs""].insert({""name"": ""Cleo""})
>>> for pk, row in db[""dogs""].pks_and_rows_where():
...     print(pk, row)
1 {'rowid': 1, 'name': 'Cleo'}

>>> db[""dogs_with_pk""].insert({""id"": 5, ""name"": ""Cleo""}, pk=""id"")
>>> for pk, row in db[""dogs_with_pk""].pks_and_rows_where():
...     print(pk, row)
5 {'id': 5, 'name': 'Cleo'}

>>> db[""dogs_with_compound_pk""].insert(
...     {""species"": ""dog"", ""id"": 3, ""name"": ""Cleo""},
...     pk=(""species"", ""id"")
... )
>>> for pk, row in db[""dogs_with_compound_pk""].pks_and_rows_where():
...     print(pk, row)
('dog', 3) {'species': 'dog', 'id': 3, 'name': 'Cleo'}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786036355,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786036355,MDEyOklzc3VlQ29tbWVudDc4NjAzNjM1NQ==,9599,simonw,2021-02-25T16:38:07Z,2021-02-25T16:38:07Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#listing-rows-with-their-primary-keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786035142,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786035142,MDEyOklzc3VlQ29tbWVudDc4NjAzNTE0Mg==,9599,simonw,2021-02-25T16:36:17Z,2021-02-25T16:36:17Z,OWNER,WIP in a pull request.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786016380,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786016380,MDEyOklzc3VlQ29tbWVudDc4NjAxNjM4MA==,9599,simonw,2021-02-25T16:10:01Z,2021-02-25T16:10:01Z,OWNER,"I prototyped this and I like it:
```
In [1]: import sqlite_utils
In [2]: db = sqlite_utils.Database(""/Users/simon/Dropbox/Development/datasette/fixtures.db"")
In [3]: list(db[""compound_primary_key""].pks_and_rows_where())
Out[3]: [(('a', 'b'), {'pk1': 'a', 'pk2': 'b', 'content': 'c'})]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786007209,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786007209,MDEyOklzc3VlQ29tbWVudDc4NjAwNzIwOQ==,9599,simonw,2021-02-25T15:57:50Z,2021-02-25T15:57:50Z,OWNER,`table.pks_and_rows_where(...)` is explicit and I think less ambiguous than the other options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786006794,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786006794,MDEyOklzc3VlQ29tbWVudDc4NjAwNjc5NA==,9599,simonw,2021-02-25T15:57:17Z,2021-02-25T15:57:28Z,OWNER,"I quite like `pks_with_rows_where(...)` - but grammatically it suggests it will return the primary keys that exist where their rows match the criteria - ""pks with rows"" can be interpreted as ""pks for the rows that..."" as opposed to ""pks accompanied by rows""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786005078,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786005078,MDEyOklzc3VlQ29tbWVudDc4NjAwNTA3OA==,9599,simonw,2021-02-25T15:54:59Z,2021-02-25T15:56:16Z,OWNER,"Is `pk_rows_where()` a good name? It sounds like it returns ""primary key rows"" which isn't a thing. It actually returns rows along with their primary key.

Other options:

- `table.rows_with_pk_where(...)` - should this return `(row, pk)` rather than `(pk, row)`?
- `table.rows_where_pk(...)`
- `table.pk_and_rows_where(...)`
- `table.pk_with_rows_where(...)`
- `table.pks_with_rows_where(...)` - because rows is pluralized, so pks should be pluralized too?
- `table.pks_rows_where(...)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786001768,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786001768,MDEyOklzc3VlQ29tbWVudDc4NjAwMTc2OA==,9599,simonw,2021-02-25T15:50:28Z,2021-02-25T15:52:12Z,OWNER,"One option: `.rows_where()` could grow a `ensure_pk=True` option which checks to see if the table is a `rowid` table and, if it is, includes that in the `select`.

Or... how about you can call `.rows_where(..., pks=True)` and it will yield `(pk, rowdict)` tuple pairs instead of just returning the sequence of dictionaries?

I'm always a little bit nervous of methods that vary their return type based on their arguments. Maybe this would be a separate method instead?
```python
    for pk, row in table.pk_rows_where(...):
        # ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785992158,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785992158,MDEyOklzc3VlQ29tbWVudDc4NTk5MjE1OA==,9599,simonw,2021-02-25T15:37:04Z,2021-02-25T15:37:04Z,OWNER,"Here's the current implementation of `.extract()`: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L1049-L1074

Tricky detail here: I create the lookup table first, based on the types of the columns that are being extracted.

I need to do this because extraction currently uses unique tuples of values, so the table has to be created in advance.

But if I'm using these new expand functions to figure out what's going to be extracted, I don't know the names of the columns and their types in advance. I'm only going to find those out during the transformation.

This may turn out to be incompatible with how `.extract()` works at the moment. I may need a new method, `.extract_expand()` perhaps? It could be simpler - work only against a single column for example.

I can still use the existing `sqlite-utils extract` CLI command though, with a `--json` flag and a rule that you can't run it against multiple columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983837,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785983837,MDEyOklzc3VlQ29tbWVudDc4NTk4MzgzNw==,9599,simonw,2021-02-25T15:25:21Z,2021-02-25T15:28:57Z,OWNER,"Problem with calling this argument `transform=` is that the term ""transform"" already means something else in this library.

I could use `convert=` instead.

... but that doesn't instantly make me think of turning a value into multiple columns.

How about `expand=`? I've not used that term anywhere yet.

    db[""Reports""].extract([""Reported by""], expand={""Reported by"": json.loads})

I think that works. You're expanding a single value into several columns of information.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983070,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785983070,MDEyOklzc3VlQ29tbWVudDc4NTk4MzA3MA==,9599,simonw,2021-02-25T15:24:17Z,2021-02-25T15:24:17Z,OWNER,I'm going to go with last-wins - so if multiple transform functions return the same key the last one will over-write the others.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980813,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785980813,MDEyOklzc3VlQ29tbWVudDc4NTk4MDgxMw==,9599,simonw,2021-02-25T15:21:02Z,2021-02-25T15:23:47Z,OWNER,"Maybe the Python version takes an optional dictionary mapping column names to transformation functions? It could then merge all of those results together - and maybe throw an error if the same key is produced by more than one column.

```python
    db[""Reports""].extract([""Reported by""], transform={""Reported by"": json.loads})
```
Or it could have an option for different strategies if keys collide: first wins, last wins, throw exception, add a prefix to the new column name. That feels a bit too complex for an edge-case though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980083,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785980083,MDEyOklzc3VlQ29tbWVudDc4NTk4MDA4Mw==,9599,simonw,2021-02-25T15:20:02Z,2021-02-25T15:20:02Z,OWNER,It would be OK if the CLI version only allows you to specify a single column if you are using the `--json` option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979769,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785979769,MDEyOklzc3VlQ29tbWVudDc4NTk3OTc2OQ==,9599,simonw,2021-02-25T15:19:37Z,2021-02-25T15:19:37Z,OWNER,For the Python version I'd like to be able to provide a transformation callback function - which can be `json.loads` but could also be anything else which accepts the value of the current column and returns a Python dictionary of columns and their values to use in the new table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979192,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785979192,MDEyOklzc3VlQ29tbWVudDc4NTk3OTE5Mg==,9599,simonw,2021-02-25T15:18:46Z,2021-02-25T15:18:46Z,OWNER,"Likewise the `sqlite-utils extract` command takes one or more columns:
```
Usage: sqlite-utils extract [OPTIONS] PATH TABLE COLUMNS...

  Extract one or more columns into a separate table

Options:
  --table TEXT             Name of the other table to extract columns to
  --fk-column TEXT         Name of the foreign key column to add to the table
  --rename <TEXT TEXT>...  Rename this column in extracted table
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785978689,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785978689,MDEyOklzc3VlQ29tbWVudDc4NTk3ODY4OQ==,9599,simonw,2021-02-25T15:18:03Z,2021-02-25T15:18:03Z,OWNER,"The Python `.extract()` method currently starts like this:
```python
def extract(self, columns, table=None, fk_column=None, rename=None):
        rename = rename or {}
        if isinstance(columns, str):
            columns = [columns]
        if not set(columns).issubset(self.columns_dict.keys()):
            raise InvalidColumns(
                ""Invalid columns {} for table with columns {}"".format(
                    columns, list(self.columns_dict.keys())
                )
            )
        ...
```
Note that it takes a list of columns (and treats a string as a single item list). That's because it can be called with a list of columns and it will use them to populate another table of unique tuples of those column values.

So a new mechanism that can instead read JSON values from a single column needs to be compatible with that existing design.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/238#issuecomment-785972074,https://api.github.com/repos/simonw/sqlite-utils/issues/238,785972074,MDEyOklzc3VlQ29tbWVudDc4NTk3MjA3NA==,9599,simonw,2021-02-25T15:08:36Z,2021-02-25T15:08:36Z,OWNER,I bet the bug is in here: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L593-L602,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816523763,.add_foreign_key() corrupts database if column contains a space,
https://github.com/simonw/datasette/issues/1241#issuecomment-784567547,https://api.github.com/repos/simonw/datasette/issues/1241,784567547,MDEyOklzc3VlQ29tbWVudDc4NDU2NzU0Nw==,9599,simonw,2021-02-23T22:45:56Z,2021-02-23T22:46:12Z,OWNER,"I really like the way the Share feature on Stack Overflow works: https://stackoverflow.com/questions/18934149/how-can-i-use-postgresqls-text-column-type-in-django
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/1241#issuecomment-784334931,https://api.github.com/repos/simonw/datasette/issues/1241,784334931,MDEyOklzc3VlQ29tbWVudDc4NDMzNDkzMQ==,9599,simonw,2021-02-23T16:37:26Z,2021-02-23T16:37:26Z,OWNER,"A ""Share link"" button would only be needed on the table page and the arbitrary query page I think - and maybe on the row page, especially as that page starts to grow more features in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/1241#issuecomment-784333768,https://api.github.com/repos/simonw/datasette/issues/1241,784333768,MDEyOklzc3VlQ29tbWVudDc4NDMzMzc2OA==,9599,simonw,2021-02-23T16:35:51Z,2021-02-23T16:35:51Z,OWNER,"This can definitely be done with a plugin.

Adding to Datasette itself is an interesting idea. I think it's possible that many users these days no longer assume they can paste a URL from the browser address bar (if they ever understood that at all) because to many apps are SPAs with broken URLs.

The shareable URLs are actually a key feature of Datasette - so maybe they should be highlighted in the default UI?

I built a ""copy to clipboard"" feature for `datasette-copyable` and wrote up how that works here: https://til.simonwillison.net/javascript/copy-button","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/1239#issuecomment-783774084,https://api.github.com/repos/simonw/datasette/issues/1239,783774084,MDEyOklzc3VlQ29tbWVudDc4Mzc3NDA4NA==,9599,simonw,2021-02-23T00:18:56Z,2021-02-23T00:19:18Z,OWNER,"Bug is here: https://github.com/simonw/datasette/blob/42caabf7e9e6e4d69ef6dd7de16f2cd96bc79d5b/datasette/filters.py#L149-L165

Those `json_each` lines should be:

    select {t}.rowid from {t}, json_each([{t}].[{c}]) j","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813978858,JSON filter fails if column contains spaces,
https://github.com/simonw/datasette/issues/1237#issuecomment-783676548,https://api.github.com/repos/simonw/datasette/issues/1237,783676548,MDEyOklzc3VlQ29tbWVudDc4MzY3NjU0OA==,9599,simonw,2021-02-22T21:10:19Z,2021-02-22T21:10:25Z,OWNER,This is another change which is a little bit hard to figure out because I haven't solved #878 yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812704869,?_pretty=1 option for pretty-printing JSON output,
https://github.com/simonw/datasette/issues/1234#issuecomment-783674659,https://api.github.com/repos/simonw/datasette/issues/1234,783674659,MDEyOklzc3VlQ29tbWVudDc4MzY3NDY1OQ==,9599,simonw,2021-02-22T21:06:28Z,2021-02-22T21:06:28Z,OWNER,"I'm not going to work on this for a while, but if anyone has needs or ideas around that they can add them to this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811505638,Runtime support for ATTACHing multiple databases,
https://github.com/simonw/datasette/issues/1236#issuecomment-783674038,https://api.github.com/repos/simonw/datasette/issues/1236,783674038,MDEyOklzc3VlQ29tbWVudDc4MzY3NDAzOA==,9599,simonw,2021-02-22T21:05:21Z,2021-02-22T21:05:21Z,OWNER,It's good on mobile - iOS at least. Going to close this open new issues if anyone reports bugs.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/782#issuecomment-782789598,https://api.github.com/repos/simonw/datasette/issues/782,782789598,MDEyOklzc3VlQ29tbWVudDc4Mjc4OTU5OA==,9599,simonw,2021-02-21T03:30:02Z,2021-02-21T03:30:02Z,OWNER,Another benefit to default:object - I could include a key that shows a list of available extras. I could then use that to power an interactive API explorer.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782765665,https://api.github.com/repos/simonw/datasette/issues/782,782765665,MDEyOklzc3VlQ29tbWVudDc4Mjc2NTY2NQ==,9599,simonw,2021-02-20T23:34:41Z,2021-02-20T23:34:41Z,OWNER,"OK, I'm back to the ""top level object as the default"" side of things now - it's pretty much unanimous at this point, and it's certainly true that it's not a decision you'll even regret.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782748501,https://api.github.com/repos/simonw/datasette/issues/782,782748501,MDEyOklzc3VlQ29tbWVudDc4Mjc0ODUwMQ==,9599,simonw,2021-02-20T20:58:18Z,2021-02-20T20:58:18Z,OWNER,"Yet another option: support a `?_path=x` option which returns a nested path from the result. So you could do this:

`/github/commits.json?_path=rows` - to get back a top-level array pulled from the `""rows""` key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782748093,https://api.github.com/repos/simonw/datasette/issues/782,782748093,MDEyOklzc3VlQ29tbWVudDc4Mjc0ODA5Mw==,9599,simonw,2021-02-20T20:54:52Z,2021-02-20T20:54:52Z,OWNER,"> Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in.

Adding a `?_pretty=1` option that does that is a great idea, I'm filing a ticket for it: #1237","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747878,https://api.github.com/repos/simonw/datasette/issues/782,782747878,MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzg3OA==,9599,simonw,2021-02-20T20:53:11Z,2021-02-20T20:53:11Z,OWNER,"... though thinking about this further, I could re-implement the `select * from commits` (but only return a max of 10 results) feature using a nested `select * from (select * from commits) limit 10` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747743,https://api.github.com/repos/simonw/datasette/issues/782,782747743,MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzc0Mw==,9599,simonw,2021-02-20T20:52:10Z,2021-02-20T20:52:10Z,OWNER,"> Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword?

The problem there is that `?_size=x` isn't actually doing the same thing as the SQL `limit` keyword. Consider this query:

https://latest-with-plugins.datasette.io/github?sql=select+*+from+commits - `select * from commits`

Datasette returns 1,000 results, and shows a ""Custom SQL query returning more than 1,000 rows"" message at the top. That's the `size` kicking in - I only fetch the first 1,000 results from the cursor to avoid exhausting resources. In the JSON version of that at https://latest-with-plugins.datasette.io/github.json?sql=select+*+from+commits there's a `""truncated"": true` key to let you know what happened.

I find myself using `?_size=2` against Datasette occasionally if I know the rows being returned are really big and I don't want to load 10+MB of HTML.

This is only really a concern for arbitrary SQL queries though - for table pages such as https://latest-with-plugins.datasette.io/github/commits?_size=10 adding `?_size=10` actually puts a `limit 10` on the underlying SQL query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747164,https://api.github.com/repos/simonw/datasette/issues/782,782747164,MDEyOklzc3VlQ29tbWVudDc4Mjc0NzE2NA==,9599,simonw,2021-02-20T20:47:16Z,2021-02-20T20:47:16Z,OWNER,(I started a thread on Twitter about this: https://twitter.com/simonw/status/1363220355318358016),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782746633,https://api.github.com/repos/simonw/datasette/issues/782,782746633,MDEyOklzc3VlQ29tbWVudDc4Mjc0NjYzMw==,9599,simonw,2021-02-20T20:43:07Z,2021-02-20T20:43:07Z,OWNER,"Another option: `.json` always returns an object with a list of keys that gets increased through adding `?_extra=` parameters.

`.jsona` always returns a JSON array of objects

I had something similar to this in Datasette a few years ago - a `.jsono` extension, which still redirects to the `shape=array` version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782742233,https://api.github.com/repos/simonw/datasette/issues/782,782742233,MDEyOklzc3VlQ29tbWVudDc4Mjc0MjIzMw==,9599,simonw,2021-02-20T20:09:16Z,2021-02-20T20:09:16Z,OWNER,I just noticed that https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0&_trace=1 executes 35 SQL queries at the moment! A great reminder that a big improvement from this change will be a reduction in queries through not calculating things like suggested facets unless they are explicitly requested.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782741719,https://api.github.com/repos/simonw/datasette/issues/782,782741719,MDEyOklzc3VlQ29tbWVudDc4Mjc0MTcxOQ==,9599,simonw,2021-02-20T20:05:04Z,2021-02-20T20:05:04Z,OWNER,"> The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest.

This is something I've not thought very hard about. If there's an error, I need to return a top-level object, not a top-level array, so I can provide details of the error.

But this means that client code will have to handle this difference - it will have to know that the returned data can be array-shaped if nothing went wrong, and object-shaped if there's an error.

The HTTP status code helps here - calling client code can know that a 200 status code means there will be an array, but an error status code means an object.

If developers really hate that the shape could be different, they can always use `?_extra=next` to ensure that the top level item is an object whether or not an error occurred. So I think this is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782741107,https://api.github.com/repos/simonw/datasette/issues/782,782741107,MDEyOklzc3VlQ29tbWVudDc4Mjc0MTEwNw==,9599,simonw,2021-02-20T20:00:22Z,2021-02-20T20:00:22Z,OWNER,"A really exciting opportunity this opens up is for parallel execution - the `facets()` and `suggested_facets()` and `total()` async functions could be called in parallel, which could speed things up if I'm confident the SQLite thread pool can execute on multiple CPU cores (it should be able to because the Python `sqlite3` module releases the GIL while it's executing C code).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740985,https://api.github.com/repos/simonw/datasette/issues/782,782740985,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDk4NQ==,9599,simonw,2021-02-20T19:59:21Z,2021-02-20T19:59:21Z,OWNER,"This design should be influenced by how it's implemented.

One implementation that could be nice is that each of the keys that can be requested - `next_url`, `total` etc - maps to an `async def` function which can do the work. So that expensive `count(*)` will only be executed by the `async def total` function if it is requested.

This raises more questions: Both `next` and `next_url` work off the same underlying data, so if they are both requested can we re-use the work that `next` does somehow? Maybe by letting these functions depend on each other (so `next_url()` knows to first call `next()`, but only if it hasn't been called already.

I think I need to flesh out the full default collection of `?_extra=` parameters in order to design how they will work under the hood.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740604,https://api.github.com/repos/simonw/datasette/issues/782,782740604,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDYwNA==,9599,simonw,2021-02-20T19:56:21Z,2021-02-20T19:56:33Z,OWNER,"I think I want to support `?_extra=next_url,total` in addition to `?_extra=next_url&_extra=total` - partly because it's less characters to type, and also because I know there exist URL handling library that don't know how to handle the same parameter multiple times (though they're going to break against Datasette already, so it's not a big deal).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740488,https://api.github.com/repos/simonw/datasette/issues/782,782740488,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDQ4OA==,9599,simonw,2021-02-20T19:55:23Z,2021-02-20T19:55:23Z,OWNER,"Am I saying you won't get back a key in the response unless you explicitly request it, either by name or by specifying a bundle of extras (e.g. `all` or `paginated`)?

The `""truncated"": true` key that tells you that your arbitrary query returned more than X results but was truncated is pretty important, do I really want people to have to opt-in to that one?

Also: having bundles like `all` or `paginated` live in the same namespace as single keys like `next_url` or `total` is a little odd - you can't tell by looking at them if they'll add a key called `all` or if they'll add a bunch of other stuff.

Maybe bundles could be prefixed with something, perhaps an underscore? `?_extra=_all` and `?_extra=_paginated` for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782739926,https://api.github.com/repos/simonw/datasette/issues/782,782739926,MDEyOklzc3VlQ29tbWVudDc4MjczOTkyNg==,9599,simonw,2021-02-20T19:51:30Z,2021-02-20T19:52:19Z,OWNER,"Demos:

- https://latest-with-plugins.datasette.io/github/commits.json-preview
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url&_extra=total
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782709425,https://api.github.com/repos/simonw/datasette/issues/782,782709425,MDEyOklzc3VlQ29tbWVudDc4MjcwOTQyNQ==,9599,simonw,2021-02-20T16:24:54Z,2021-02-20T16:24:54Z,OWNER,Having shortcuts means I could support `?_extra=all` for returning ALL possible keys.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782709270,https://api.github.com/repos/simonw/datasette/issues/782,782709270,MDEyOklzc3VlQ29tbWVudDc4MjcwOTI3MA==,9599,simonw,2021-02-20T16:23:51Z,2021-02-20T16:24:11Z,OWNER,"Also how would you opt out of returning the `""rows""` key? I sometimes want to do this - if I want to get back just the count or just the facets for example.

Some options:

* `/fixtures/roadside_attractions.json?_extra=total&_extra=-rows`
* `/fixtures/roadside_attractions.json?_extra=total&_skip=rows`
* `/fixtures/roadside_attractions.json?_extra=total&_size=0`

I quite like that last one with `?_size=0`. I think it would still return `""rows"": []` but that's OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782708938,https://api.github.com/repos/simonw/datasette/issues/782,782708938,MDEyOklzc3VlQ29tbWVudDc4MjcwODkzOA==,9599,simonw,2021-02-20T16:22:14Z,2021-02-20T16:22:14Z,OWNER,"I'm leaning back in the direction of a flat JSON array of objects as the default - this:

`/fixtures/roadside_attractions.json`

Would return:

```json
[
    {
      ""pk"": 1,
      ""name"": ""The Mystery Spot"",
      ""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
      ""latitude"": 37.0167,
      ""longitude"": -122.0024
    },
    {
      ""pk"": 2,
      ""name"": ""Winchester Mystery House"",
      ""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
      ""latitude"": 37.3184,
      ""longitude"": -121.9511
    },
    {
      ""pk"": 3,
      ""name"": ""Burlingame Museum of PEZ Memorabilia"",
      ""address"": ""214 California Drive, Burlingame, CA 94010"",
      ""latitude"": 37.5793,
      ""longitude"": -122.3442
    },
    {
      ""pk"": 4,
      ""name"": ""Bigfoot Discovery Museum"",
      ""address"": ""5497 Highway 9, Felton, CA 95018"",
      ""latitude"": 37.0414,
      ""longitude"": -122.0725
    }
]
```
To get the version that includes pagination information you would use the `?_extra=` parameter. For example:

`/fixtures/roadside_attractions.json?_extra=total&_extra=next_url`

```json
{
  ""rows"": [
    {
      ""pk"": 1,
      ""name"": ""The Mystery Spot"",
      ""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
      ""latitude"": 37.0167,
      ""longitude"": -122.0024
    },
    {
      ""pk"": 2,
      ""name"": ""Winchester Mystery House"",
      ""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
      ""latitude"": 37.3184,
      ""longitude"": -121.9511
    },
    {
      ""pk"": 3,
      ""name"": ""Burlingame Museum of PEZ Memorabilia"",
      ""address"": ""214 California Drive, Burlingame, CA 94010"",
      ""latitude"": 37.5793,
      ""longitude"": -122.3442
    },
    {
      ""pk"": 4,
      ""name"": ""Bigfoot Discovery Museum"",
      ""address"": ""5497 Highway 9, Felton, CA 95018"",
      ""latitude"": 37.0414,
      ""longitude"": -122.0725
    }
  ],
  ""total"": 4,
  ""next_url"": null
}
```
ANY usage of the `?_extra=` parameter would turn the list into an object with a `""rows""` key.

Opting in to the `total` is nice because it's actually expensive to run a count, so only doing a count if the user requests it feels good.

But... having to add `?_extra=total&_extra=next_url` for the common case of wanting both the total count and the URL to get the next page of results is a bit verbose. So maybe support aliases, like `?_extra=paginated` which is a shortcut for `?_extra=total&_extra=next_url`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1236#issuecomment-782464306,https://api.github.com/repos/simonw/datasette/issues/1236,782464306,MDEyOklzc3VlQ29tbWVudDc4MjQ2NDMwNg==,9599,simonw,2021-02-19T23:57:32Z,2021-02-19T23:57:32Z,OWNER,Need to test this on mobile.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782464215,https://api.github.com/repos/simonw/datasette/issues/1236,782464215,MDEyOklzc3VlQ29tbWVudDc4MjQ2NDIxNQ==,9599,simonw,2021-02-19T23:57:13Z,2021-02-19T23:57:13Z,OWNER,Now live on https://latest.datasette.io/_memory,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782462049,https://api.github.com/repos/simonw/datasette/issues/1236,782462049,MDEyOklzc3VlQ29tbWVudDc4MjQ2MjA0OQ==,9599,simonw,2021-02-19T23:51:12Z,2021-02-19T23:51:12Z,OWNER,"![resize-demo](https://user-images.githubusercontent.com/9599/108573758-4914eb00-72ca-11eb-989c-e642eee68021.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782459550,https://api.github.com/repos/simonw/datasette/issues/1236,782459550,MDEyOklzc3VlQ29tbWVudDc4MjQ1OTU1MA==,9599,simonw,2021-02-19T23:45:30Z,2021-02-19T23:45:30Z,OWNER,"Encoded using https://meyerweb.com/eric/tools/dencoder/

`%3Csvg%20aria-labelledby%3D%22cm-drag-to-resize%22%20role%3D%22img%22%20fill%3D%22%23ccc%22%20stroke%3D%22%23ccc%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2016%2016%22%20width%3D%2216%22%20height%3D%2216%22%3E%0A%20%20%3Ctitle%20id%3D%22cm-drag-to-resize%22%3EDrag%20to%20resize%3C%2Ftitle%3E%0A%20%20%3Cpath%20fill-rule%3D%22evenodd%22%20d%3D%22M1%202.75A.75.75%200%20011.75%202h12.5a.75.75%200%20110%201.5H1.75A.75.75%200%20011%202.75zm0%205A.75.75%200%20011.75%207h12.5a.75.75%200%20110%201.5H1.75A.75.75%200%20011%207.75zM1.75%2012a.75.75%200%20100%201.5h12.5a.75.75%200%20100-1.5H1.75z%22%3E%3C%2Fpath%3E%0A%3C%2Fsvg%3E`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782459405,https://api.github.com/repos/simonw/datasette/issues/1236,782459405,MDEyOklzc3VlQ29tbWVudDc4MjQ1OTQwNQ==,9599,simonw,2021-02-19T23:45:02Z,2021-02-19T23:45:02Z,OWNER,"I'm going to use a variant of the Datasette menu icon. Here it is in `#ccc` with an ARIA label:

```svg
<svg aria-labelledby=""cm-drag-to-resize"" role=""img"" fill=""#ccc"" stroke=""#ccc"" xmlns=""http://www.w3.org/2000/svg"" viewBox=""0 0 16 16"" width=""16"" height=""16"">
  <title id=""cm-drag-to-resize"">Drag to resize</title>
  <path fill-rule=""evenodd"" d=""M1 2.75A.75.75 0 011.75 2h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 2.75zm0 5A.75.75 0 011.75 7h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 7.75zM1.75 12a.75.75 0 100 1.5h12.5a.75.75 0 100-1.5H1.75z""></path>
</svg>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782458983,https://api.github.com/repos/simonw/datasette/issues/1236,782458983,MDEyOklzc3VlQ29tbWVudDc4MjQ1ODk4Mw==,9599,simonw,2021-02-19T23:43:34Z,2021-02-19T23:43:34Z,OWNER,"I only want it to resize up and down, not left to right - so I'm not keen on the default resize handle:

<img width=""722"" alt=""cm-resize_demo"" src=""https://user-images.githubusercontent.com/9599/108573363-364de680-72c9-11eb-8741-5112463ebfaa.png"">

https://rawgit.com/Sphinxxxx/cm-resize/master/demo/index.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782458744,https://api.github.com/repos/simonw/datasette/issues/1236,782458744,MDEyOklzc3VlQ29tbWVudDc4MjQ1ODc0NA==,9599,simonw,2021-02-19T23:42:42Z,2021-02-19T23:42:42Z,OWNER,I can use https://github.com/Sphinxxxx/cm-resize for this,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/619#issuecomment-782246111,https://api.github.com/repos/simonw/datasette/issues/619,782246111,MDEyOklzc3VlQ29tbWVudDc4MjI0NjExMQ==,9599,simonw,2021-02-19T18:11:22Z,2021-02-19T18:11:22Z,OWNER,"Big usability improvement, see also #1236","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/sqlite-utils/issues/236#issuecomment-781825726,https://api.github.com/repos/simonw/sqlite-utils/issues/236,781825726,MDEyOklzc3VlQ29tbWVudDc4MTgyNTcyNg==,9599,simonw,2021-02-19T05:10:41Z,2021-02-19T05:10:41Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#attaching-additional-databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811680502,--attach command line option for attaching extra databases,
https://github.com/simonw/sqlite-utils/issues/113#issuecomment-781825187,https://api.github.com/repos/simonw/sqlite-utils/issues/113,781825187,MDEyOklzc3VlQ29tbWVudDc4MTgyNTE4Nw==,9599,simonw,2021-02-19T05:09:12Z,2021-02-19T05:09:12Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#attaching-additional-databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621286870,Syntactic sugar for ATTACH DATABASE,
https://github.com/simonw/datasette/issues/283#issuecomment-781764561,https://api.github.com/repos/simonw/datasette/issues/283,781764561,MDEyOklzc3VlQ29tbWVudDc4MTc2NDU2MQ==,9599,simonw,2021-02-19T02:10:21Z,2021-02-19T02:10:21Z,OWNER,This feature is now released! https://docs.datasette.io/en/stable/changelog.html#v0-55,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/1235#issuecomment-781736855,https://api.github.com/repos/simonw/datasette/issues/1235,781736855,MDEyOklzc3VlQ29tbWVudDc4MTczNjg1NQ==,9599,simonw,2021-02-19T00:52:47Z,2021-02-19T01:47:53Z,OWNER,"I bumped the two lines in the `Dockerfile` to `FROM python:3.7.10-slim-stretch as build` and ran this to build it:

    docker build -f Dockerfile -t datasetteproject/datasette:python-3-7-10 .

Then I ran it with:

    docker run -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette:python-3-7-10 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

http://0.0.0.0:8001/-/versions confirmed that it was now running Python 3.7.10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811589344,Upgrade Python version used by official Datasette Docker image,
https://github.com/simonw/datasette/issues/1235#issuecomment-781735887,https://api.github.com/repos/simonw/datasette/issues/1235,781735887,MDEyOklzc3VlQ29tbWVudDc4MTczNTg4Nw==,9599,simonw,2021-02-19T00:50:21Z,2021-02-19T00:50:55Z,OWNER,"I'll bump to `3.7.10` for the moment - the fix for 3.8 isn't out until March 1st according to https://news.ycombinator.com/item?id=26186434

https://www.python.org/downloads/release/python-3710/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811589344,Upgrade Python version used by official Datasette Docker image,
https://github.com/simonw/datasette/issues/283#issuecomment-781670827,https://api.github.com/repos/simonw/datasette/issues/283,781670827,MDEyOklzc3VlQ29tbWVudDc4MTY3MDgyNw==,9599,simonw,2021-02-18T22:16:46Z,2021-02-18T22:16:46Z,OWNER,"Demo is now live here: https://latest.datasette.io/_memory

The documentation is at https://docs.datasette.io/en/latest/sql_queries.html#cross-database-queries - it links to this example query: https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/283#issuecomment-781665560,https://api.github.com/repos/simonw/datasette/issues/283,781665560,MDEyOklzc3VlQ29tbWVudDc4MTY2NTU2MA==,9599,simonw,2021-02-18T22:06:14Z,2021-02-18T22:06:14Z,OWNER,"The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run `datasette one.db two.db three.db --crossdb` and then use the `/_memory` page to run joins across tables from multiple databases.

It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see [Datasette Library](https://github.com/simonw/datasette/issues/417)) this won't be particularly useful for you.

So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting `/_memory?attach=db1&attach=db2` to get a special connection.

Also worth noting: plugins that implement the [prepare_connection()](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette) hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/pull/1232#issuecomment-781651283,https://api.github.com/repos/simonw/datasette/issues/1232,781651283,MDEyOklzc3VlQ29tbWVudDc4MTY1MTI4Mw==,9599,simonw,2021-02-18T21:37:55Z,2021-02-18T21:37:55Z,OWNER,"UI listing the attached tables:

<img width=""888"" alt=""_memory"" src=""https://user-images.githubusercontent.com/9599/108424809-8015ce80-71ee-11eb-8752-03c459f89320.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781641728,https://api.github.com/repos/simonw/datasette/issues/1232,781641728,MDEyOklzc3VlQ29tbWVudDc4MTY0MTcyOA==,9599,simonw,2021-02-18T21:19:34Z,2021-02-18T21:19:34Z,OWNER,"I tested the demo deployment like this:
```
datasette publish cloudrun fixtures.db extra_database.db \                                      
            -m fixtures.json \
            --plugins-dir=plugins \
            --branch=crossdb \
            --extra-options=""--setting template_debug 1 --crossdb"" \
            --install=pysqlite3-binary \
            --service=datasette-latest-crossdb
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781637292,https://api.github.com/repos/simonw/datasette/issues/1232,781637292,MDEyOklzc3VlQ29tbWVudDc4MTYzNzI5Mg==,9599,simonw,2021-02-18T21:11:31Z,2021-02-18T21:11:31Z,OWNER,Due to bug #1233 I'm going to publish the additional database as `extra_database.db` rather than `extra database.db` as it is used in the tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/issues/1233#issuecomment-781636590,https://api.github.com/repos/simonw/datasette/issues/1233,781636590,MDEyOklzc3VlQ29tbWVudDc4MTYzNjU5MA==,9599,simonw,2021-02-18T21:10:08Z,2021-02-18T21:10:08Z,OWNER,I think the bug is here: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/__init__.py#L349-L373,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811458446,"""datasette publish cloudrun"" cannot publish files with spaces in their name",
https://github.com/simonw/datasette/pull/1232#issuecomment-781634819,https://api.github.com/repos/simonw/datasette/issues/1232,781634819,MDEyOklzc3VlQ29tbWVudDc4MTYzNDgxOQ==,9599,simonw,2021-02-18T21:06:43Z,2021-02-18T21:06:43Z,OWNER,I'll document this option on https://docs.datasette.io/en/stable/sql_queries.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781629841,https://api.github.com/repos/simonw/datasette/issues/1232,781629841,MDEyOklzc3VlQ29tbWVudDc4MTYyOTg0MQ==,9599,simonw,2021-02-18T20:57:23Z,2021-02-18T20:57:23Z,OWNER,"The new warning looks like this:

<img width=""514"" alt=""datasette_—_pipenv_shell_▸_Python_—_182×66"" src=""https://user-images.githubusercontent.com/9599/108420562-d41db480-71e8-11eb-87e3-1cddd65627c2.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781598585,https://api.github.com/repos/simonw/datasette/issues/1232,781598585,MDEyOklzc3VlQ29tbWVudDc4MTU5ODU4NQ==,9599,simonw,2021-02-18T19:57:30Z,2021-02-18T19:57:30Z,OWNER,It would also be neat if https://latest.datasette.io/ had multiple databases attached in order to demonstrate this feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781594632,https://api.github.com/repos/simonw/datasette/issues/1232,781594632,MDEyOklzc3VlQ29tbWVudDc4MTU5NDYzMg==,9599,simonw,2021-02-18T19:50:21Z,2021-02-18T19:50:21Z,OWNER,"It would be neat if the `/_memory` page showed a list of attached databases, to indicate that the `--crossdb` option is working and give people links to click to start running queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/issues/283#issuecomment-781593169,https://api.github.com/repos/simonw/datasette/issues/283,781593169,MDEyOklzc3VlQ29tbWVudDc4MTU5MzE2OQ==,9599,simonw,2021-02-18T19:47:34Z,2021-02-18T19:47:34Z,OWNER,"I have a working version now, moving development to a pull request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/283#issuecomment-781591015,https://api.github.com/repos/simonw/datasette/issues/283,781591015,MDEyOklzc3VlQ29tbWVudDc4MTU5MTAxNQ==,9599,simonw,2021-02-18T19:44:02Z,2021-02-18T19:44:02Z,OWNER,For the moment I'm going to hard-code a `SQLITE_LIMIT_ATTACHED=10` constant and only attach the first 10 databases to the `_memory` connection.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/283#issuecomment-781574786,https://api.github.com/repos/simonw/datasette/issues/283,781574786,MDEyOklzc3VlQ29tbWVudDc4MTU3NDc4Ng==,9599,simonw,2021-02-18T19:15:37Z,2021-02-18T19:15:37Z,OWNER,`select * from pragma_database_list();` is useful - shows all attached databases for the current connection.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/283#issuecomment-781573676,https://api.github.com/repos/simonw/datasette/issues/283,781573676,MDEyOklzc3VlQ29tbWVudDc4MTU3MzY3Ng==,9599,simonw,2021-02-18T19:13:30Z,2021-02-18T19:13:30Z,OWNER,"It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/1231#issuecomment-781560989,https://api.github.com/repos/simonw/datasette/issues/1231,781560989,MDEyOklzc3VlQ29tbWVudDc4MTU2MDk4OQ==,9599,simonw,2021-02-18T18:50:53Z,2021-02-18T18:50:53Z,OWNER,Ideally I'd figure out a way to replicate this error in a concurrent unit test.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-781560865,https://api.github.com/repos/simonw/datasette/issues/1231,781560865,MDEyOklzc3VlQ29tbWVudDc4MTU2MDg2NQ==,9599,simonw,2021-02-18T18:50:38Z,2021-02-18T18:50:38Z,OWNER,"I started trying to use locks to resolve this but I've not figured out the right way to do that yet - here's my first experiment:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 9e15a16..1681c9d 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -217,6 +217,7 @@ class Datasette:
         self.inspect_data = inspect_data
         self.immutables = set(immutables or [])
         self.databases = collections.OrderedDict()
+        self._refresh_schemas_lock = threading.Lock()
         if memory or not self.files:
             self.add_database(Database(self, is_memory=True), name=""_memory"")
         # memory_name is a random string so that each Datasette instance gets its own
@@ -324,6 +325,13 @@ class Datasette:
         self.client = DatasetteClient(self)
 
     async def refresh_schemas(self):
+        return
+        if self._refresh_schemas_lock.locked():
+            return
+        with self._refresh_schemas_lock:
+            await self._refresh_schemas()
+
+    async def _refresh_schemas(self):
         internal_db = self.databases[""_internal""]
         if not self.internal_db_created:
             await init_internal_db(internal_db)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1226#issuecomment-781546512,https://api.github.com/repos/simonw/datasette/issues/1226,781546512,MDEyOklzc3VlQ29tbWVudDc4MTU0NjUxMg==,9599,simonw,2021-02-18T18:26:19Z,2021-02-18T18:26:19Z,OWNER,This broke CI: https://github.com/simonw/datasette/runs/1929355965?check_suite_focus=true,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,--port option should validate port is between 0 and 65535,
https://github.com/simonw/datasette/issues/1226#issuecomment-781530157,https://api.github.com/repos/simonw/datasette/issues/1226,781530157,MDEyOklzc3VlQ29tbWVudDc4MTUzMDE1Nw==,9599,simonw,2021-02-18T18:00:15Z,2021-02-18T18:00:15Z,OWNER,"I can use `click.IntRange(min=None, max=None)` for this. https://click.palletsprojects.com/en/7.x/options/#ranges - inclusive on both edges.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,--port option should validate port is between 0 and 65535,
https://github.com/simonw/datasette/issues/283#issuecomment-781077127,https://api.github.com/repos/simonw/datasette/issues/283,781077127,MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw==,9599,simonw,2021-02-18T05:56:30Z,2021-02-18T05:57:34Z,OWNER,I'm going to to try prototyping the `--crossdb` option that causes `/_memory` to connect to all databases as a starting point and see how well that works.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/1226#issuecomment-779467451,https://api.github.com/repos/simonw/datasette/issues/1226,779467451,MDEyOklzc3VlQ29tbWVudDc3OTQ2NzQ1MQ==,9599,simonw,2021-02-15T22:02:46Z,2021-02-15T22:02:46Z,OWNER,"I'm OK with the current error message shown if you try to use too low a port:
```
datasette fivethirtyeight.db -p 800 
INFO:     Started server process [45511]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
ERROR:    [Errno 13] error while attempting to bind on address ('127.0.0.1', 800): permission denied
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,--port option should validate port is between 0 and 65535,
https://github.com/simonw/datasette/issues/1226#issuecomment-779467160,https://api.github.com/repos/simonw/datasette/issues/1226,779467160,MDEyOklzc3VlQ29tbWVudDc3OTQ2NzE2MA==,9599,simonw,2021-02-15T22:01:53Z,2021-02-15T22:01:53Z,OWNER,"This check needs to happen in two places:

https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L222-L227

https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L328-L333","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,--port option should validate port is between 0 and 65535,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779416619,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779416619,MDEyOklzc3VlQ29tbWVudDc3OTQxNjYxOQ==,9599,simonw,2021-02-15T19:40:57Z,2021-02-15T21:27:55Z,OWNER,"Tried this experiment (not proper binary search, it only searches downwards):
```python
import sqlite3

db = sqlite3.connect("":memory:"")

def tryit(n):
    sql = ""select 1 where 1 in ({})"".format("", "".join(""?"" for i in range(n)))
    db.execute(sql, [0 for i in range(n)])


def find_limit(min=0, max=5_000_000):
    value = max
    while True:
        print('Trying', value)
        try:
            tryit(value)
            return value
        except:
            value = value // 2
```
Running `find_limit()` with those default parameters takes about 1.47s on my laptop:
```
In [9]: %timeit find_limit()
Trying 5000000
Trying 2500000...
1.47 s ± 28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Interestingly the value it suggested was 156250 - suggesting that the macOS `sqlite3` binary with a 500,000 limit isn't the same as whatever my Python is using here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779448912,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779448912,MDEyOklzc3VlQ29tbWVudDc3OTQ0ODkxMg==,9599,simonw,2021-02-15T21:09:50Z,2021-02-15T21:09:50Z,OWNER,"I fiddled around and replaced that line with `batch_size = SQLITE_MAX_VARS // num_columns` - which evaluated to `10416` for this particular file. That got me this:

       40.71s user 1.81s system 98% cpu 43.081 total

43s is definitely better than 56s, but it's still not as big as the  ~26.5s to ~3.5s improvement described by @simonwiles at the top of this issue. I wonder what I'm missing here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779446652,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779446652,MDEyOklzc3VlQ29tbWVudDc3OTQ0NjY1Mg==,9599,simonw,2021-02-15T21:04:19Z,2021-02-15T21:04:19Z,OWNER,"... but it looks like `batch_size` is hard-coded to 100, rather than `None` - which means it's not being calculated using that value:

https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L704

And

https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L1877","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779445423,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779445423,MDEyOklzc3VlQ29tbWVudDc3OTQ0NTQyMw==,9599,simonw,2021-02-15T21:00:44Z,2021-02-15T21:01:09Z,OWNER,"I tried changing the hard-coded value from 999 to 156_250 and running `sqlite-utils insert` against a 500MB CSV file, with these results:
```
(sqlite-utils) sqlite-utils % time sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers
  [###################################-]   99%  00:00:00sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv
44.74s user 7.61s system 92% cpu 56.601 total
# Increased the setting here
(sqlite-utils) sqlite-utils % time sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers
  [###################################-]   99%  00:00:00sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv
39.40s user 5.15s system 96% cpu 46.320 total
```
Not as big a difference as I was expecting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779417723,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779417723,MDEyOklzc3VlQ29tbWVudDc3OTQxNzcyMw==,9599,simonw,2021-02-15T19:44:02Z,2021-02-15T19:47:00Z,OWNER,"`%timeit find_limit(max=1_000_000)` took 378ms on my laptop

`%timeit find_limit(max=500_000)` took 197ms

`%timeit find_limit(max=200_000)` reported 53ms per loop

`%timeit find_limit(max=100_000)` reported 26.8ms per loop.

All of these are still slow enough that I'm not comfortable running this search for every time the library is imported. Allowing users to opt-in to this as a performance enhancement might be better.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779409770,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779409770,MDEyOklzc3VlQ29tbWVudDc3OTQwOTc3MA==,9599,simonw,2021-02-15T19:23:11Z,2021-02-15T19:23:11Z,OWNER,"On my Mac right now I'm seeing a limit of 500,000:
```
% sqlite3 -cmd "".limits variable_number""
     variable_number 500000
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778854808,https://api.github.com/repos/simonw/sqlite-utils/issues/227,778854808,MDEyOklzc3VlQ29tbWVudDc3ODg1NDgwOA==,9599,simonw,2021-02-14T22:46:54Z,2021-02-14T22:46:54Z,OWNER,Fix is released in 3.5.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,Error reading csv files with large column data,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778851721,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778851721,MDEyOklzc3VlQ29tbWVudDc3ODg1MTcyMQ==,9599,simonw,2021-02-14T22:23:46Z,2021-02-14T22:23:46Z,OWNER,I called this `--no-headers` for consistency with the existing output option: https://github.com/simonw/sqlite-utils/blob/427dace184c7da57f4a04df07b1e84cdae3261e8/sqlite_utils/cli.py#L61-L64,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778849394,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778849394,MDEyOklzc3VlQ29tbWVudDc3ODg0OTM5NA==,9599,simonw,2021-02-14T22:06:53Z,2021-02-14T22:06:53Z,OWNER,"For the moment I think just adding `--no-header` - which causes column names ""unknown1,unknown2,..."" to be used - should be enough.

Users can import with that option, then use `sqlite-utils transform --rename` to rename them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778844016,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778844016,MDEyOklzc3VlQ29tbWVudDc3ODg0NDAxNg==,9599,simonw,2021-02-14T21:22:45Z,2021-02-14T21:22:45Z,OWNER,"I'm going to use this pattern from https://stackoverflow.com/a/15063941
```python
import sys
import csv
maxInt = sys.maxsize

while True:
    # decrease the maxInt value by factor 10 
    # as long as the OverflowError occurs.

    try:
        csv.field_size_limit(maxInt)
        break
    except OverflowError:
        maxInt = int(maxInt/10)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,Hitting `_csv.Error: field larger than field limit (131072)`,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843503,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778843503,MDEyOklzc3VlQ29tbWVudDc3ODg0MzUwMw==,9599,simonw,2021-02-14T21:18:51Z,2021-02-14T21:18:51Z,OWNER,"I want to set this to the maximum allowed limit, which seems to be surprisingly hard! That StackOverflow thread is full of ideas for that, many of them involving `ctypes`. I'm a bit loathe to add a dependency on `ctypes` though - even though it's in the Python standard library I worry that it might not be available on some architectures.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,Hitting `_csv.Error: field larger than field limit (131072)`,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843362,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778843362,MDEyOklzc3VlQ29tbWVudDc3ODg0MzM2Mg==,9599,simonw,2021-02-14T21:17:53Z,2021-02-14T21:17:53Z,OWNER,Same issue as #227.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,Hitting `_csv.Error: field larger than field limit (131072)`,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811746,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778811746,MDEyOklzc3VlQ29tbWVudDc3ODgxMTc0Ng==,9599,simonw,2021-02-14T17:39:30Z,2021-02-14T21:16:54Z,OWNER,"I'm going to detach this from the #131 column types idea.

The three things I need to handle here are:

- The CSV file doesn't have a header row at all, so I need to specify what the column names should be
- The CSV file DOES have a header row but I want to ignore it and use alternative column names
- The CSV doesn't have a header row at all and I want to automatically use `unknown1,unknown2...` so I can start exploring it as quickly as possible.

Here's a potential design that covers the first two:

`--replace-header=""foo,bar,baz""` - ignore whatever is in the first row and pretend it was this instead
`--add-header=""foo,bar,baz""` - add a first row with these details, to use as the header

It doesn't cover the ""give me unknown column names"" case though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778843086,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778843086,MDEyOklzc3VlQ29tbWVudDc3ODg0MzA4Ng==,9599,simonw,2021-02-14T21:15:43Z,2021-02-14T21:15:43Z,OWNER,"I'm not convinced the `.has_header()` rules are useful for the kind of CSV files I work with: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L383

```python
    def has_header(self, sample):
        # Creates a dictionary of types of data in each column. If any
        # column is of a single type (say, integers), *except* for the first
        # row, then the first row is presumed to be labels. If the type
        # can't be determined, it is assumed to be a string in which case
        # the length of the string is the determining factor: if all of the
        # rows except for the first are the same length, it's a header.
        # Finally, a 'vote' is taken at the end for each column, adding or
        # subtracting from the likelihood of the first row being a header.
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778842982,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778842982,MDEyOklzc3VlQ29tbWVudDc3ODg0Mjk4Mg==,9599,simonw,2021-02-14T21:15:11Z,2021-02-14T21:15:11Z,OWNER,"Implementation tip: I have code that reads the first row and uses it as headers here: https://github.com/simonw/sqlite-utils/blob/8f042ae1fd323995d966a94e8e6df85cc843b938/sqlite_utils/cli.py#L689-L691

So If I want to use `unknown1,unknown2...` I can do that by reading the first row, counting the number of columns, generating headers based on that range and then continuing to build that generator (maybe with `itertools.chain()` to replay the record we already read).

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778841704,https://api.github.com/repos/simonw/sqlite-utils/issues/227,778841704,MDEyOklzc3VlQ29tbWVudDc3ODg0MTcwNA==,9599,simonw,2021-02-14T21:05:20Z,2021-02-14T21:05:20Z,OWNER,This has also been reported in #229.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,Error reading csv files with large column data,
https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778841547,https://api.github.com/repos/simonw/sqlite-utils/issues/225,778841547,MDEyOklzc3VlQ29tbWVudDc3ODg0MTU0Nw==,9599,simonw,2021-02-14T21:04:13Z,2021-02-14T21:04:13Z,OWNER,I added a test and fixed this in #234 - thanks for the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797159961,fix for problem in Table.insert_all on search for columns per chunk of rows,
https://github.com/simonw/sqlite-utils/issues/234#issuecomment-778841278,https://api.github.com/repos/simonw/sqlite-utils/issues/234,778841278,MDEyOklzc3VlQ29tbWVudDc3ODg0MTI3OA==,9599,simonw,2021-02-14T21:02:11Z,2021-02-14T21:02:11Z,OWNER,"I managed to replicate this in a test:
```python
def test_insert_all_with_extra_columns_in_later_chunks(fresh_db):
    chunk = [
        {""record"": ""Record 1""},
        {""record"": ""Record 2""},
        {""record"": ""Record 3""},
        {""record"": ""Record 4"", ""extra"": 1},
    ]
    fresh_db[""t""].insert_all(chunk, batch_size=2, alter=True)
    assert list(fresh_db[""t""].rows) == [
        {""record"": ""Record 1"", ""extra"": None},
        {""record"": ""Record 2"", ""extra"": None},
        {""record"": ""Record 3"", ""extra"": None},
        {""record"": ""Record 4"", ""extra"": 1},
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808046597,.insert_all() fails if subsequent chunks contain additional columns,
https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778834504,https://api.github.com/repos/simonw/sqlite-utils/issues/225,778834504,MDEyOklzc3VlQ29tbWVudDc3ODgzNDUwNA==,9599,simonw,2021-02-14T20:09:30Z,2021-02-14T20:09:30Z,OWNER,Thanks for this. I'm going to try and get the test suite to run in Windows on GitHub Actions.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797159961,fix for problem in Table.insert_all on search for columns per chunk of rows,
https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778829456,https://api.github.com/repos/simonw/sqlite-utils/issues/231,778829456,MDEyOklzc3VlQ29tbWVudDc3ODgyOTQ1Ng==,9599,simonw,2021-02-14T19:37:52Z,2021-02-14T19:37:52Z,OWNER,"I'm going to add `limit` and `offset` to the following methods:

- `rows_where()`
- `search_sql()`
- `search()`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808028757,"limit=X, offset=Y parameters for more Python methods",
https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778828758,https://api.github.com/repos/simonw/sqlite-utils/issues/231,778828758,MDEyOklzc3VlQ29tbWVudDc3ODgyODc1OA==,9599,simonw,2021-02-14T19:33:14Z,2021-02-14T19:33:14Z,OWNER,The `limit=` parameter is currently only available on the `.search()` method - it would make sense to add this to other methods as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808028757,"limit=X, offset=Y parameters for more Python methods",
https://github.com/simonw/sqlite-utils/pull/224#issuecomment-778828495,https://api.github.com/repos/simonw/sqlite-utils/issues/224,778828495,MDEyOklzc3VlQ29tbWVudDc3ODgyODQ5NQ==,9599,simonw,2021-02-14T19:31:06Z,2021-02-14T19:31:06Z,OWNER,I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792297010,Add fts offset docs.,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778827570,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778827570,MDEyOklzc3VlQ29tbWVudDc3ODgyNzU3MA==,9599,simonw,2021-02-14T19:24:20Z,2021-02-14T19:24:20Z,OWNER,Here's the implementation in Python: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L204-L225,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778824361,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778824361,MDEyOklzc3VlQ29tbWVudDc3ODgyNDM2MQ==,9599,simonw,2021-02-14T18:59:22Z,2021-02-14T18:59:22Z,OWNER,"I think I've got it. I can use `io.BufferedReader()` to get an object I can run `.peek(2048)` on, then wrap THAT in `io.TextIOWrapper`:

```python
    encoding = encoding or ""utf-8""
    buffered = io.BufferedReader(json_file, buffer_size=4096)
    decoded = io.TextIOWrapper(buffered, encoding=encoding, line_buffering=True)
    if pk and len(pk) == 1:
        pk = pk[0]
    if csv or tsv:
        if sniff:
            # Read first 2048 bytes and use that to detect
            first_bytes = buffered.peek(2048)
            print('first_bytes', first_bytes)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778821403,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778821403,MDEyOklzc3VlQ29tbWVudDc3ODgyMTQwMw==,9599,simonw,2021-02-14T18:38:16Z,2021-02-14T18:38:16Z,OWNER,"There are two code paths here that matter:

- For a regular file, can read the first 2048 bytes, then `.seek(0)` before continuing. That's easy.
- `stdin` is harder. I need to read and buffer the first 2048 bytes, then pass an object to `csv.reader()` which will replay that chunk and then play the rest of stdin.

I'm a bit stuck on the second one. Ideally I could use something like `itertools.chain()` but I can't find an alternative for file-like objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778818639,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778818639,MDEyOklzc3VlQ29tbWVudDc3ODgxODYzOQ==,9599,simonw,2021-02-14T18:22:38Z,2021-02-14T18:22:38Z,OWNER,Maybe I shouldn't be using `StreamReader` at all - https://www.python.org/dev/peps/pep-0400/ suggests that it should be deprecated in favour of `io.TextIOWrapper`. I'm using `StreamReader` due to this line: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L667-L668,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778817494,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778817494,MDEyOklzc3VlQ29tbWVudDc3ODgxNzQ5NA==,9599,simonw,2021-02-14T18:16:06Z,2021-02-14T18:16:06Z,OWNER,"Types involved:
```
(Pdb) type(json_file.raw)
<class '_io.FileIO'>
(Pdb) type(json_file)
<class 'encodings.utf_8.StreamReader'>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778816333,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778816333,MDEyOklzc3VlQ29tbWVudDc3ODgxNjMzMw==,9599,simonw,2021-02-14T18:08:44Z,2021-02-14T18:08:44Z,OWNER,"No, you can't `.seek(0)` on stdin:
```
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 678, in insert_upsert_implementation
    json_file.raw.seek(0)
OSError: [Errno 29] Illegal seek
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778815740,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778815740,MDEyOklzc3VlQ29tbWVudDc3ODgxNTc0MA==,9599,simonw,2021-02-14T18:05:03Z,2021-02-14T18:05:03Z,OWNER,"The challenge here is how to read the first 2048 bytes and then reset the incoming file.

The Python docs example looks like this:

```python
with open('example.csv', newline='') as csvfile:
    dialect = csv.Sniffer().sniff(csvfile.read(1024))
    csvfile.seek(0)
    reader = csv.reader(csvfile, dialect)
```
Here's the relevant code in `sqlite-utils`: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L671-L679

The challenge is going to be having the `--sniff` option work with the progress bar. Here's how `file_progress()` works: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/utils.py#L106-L113

If `file.raw` is `stdin` can I do the equivalent of `csvfile.seek(0)` on it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778812684,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778812684,MDEyOklzc3VlQ29tbWVudDc3ODgxMjY4NA==,9599,simonw,2021-02-14T17:45:16Z,2021-02-14T17:45:16Z,OWNER,"Running this could take any CSV (or TSV) file and automatically detect the delimiter. If no header row is detected it could add `unknown1,unknown2` headers:

    sqlite-utils insert db.db data file.csv --sniff

(Using `--sniff` would imply `--csv`)

This could be called `--sniffer` instead but I like `--sniff` better.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778812050,MDEyOklzc3VlQ29tbWVudDc3ODgxMjA1MA==,9599,simonw,2021-02-14T17:41:30Z,2021-02-14T17:41:30Z,OWNER,"I just spotted that `csv.Sniffer` in the Python standard library has a `.has_header(sample)` method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811934,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778811934,MDEyOklzc3VlQ29tbWVudDc3ODgxMTkzNA==,9599,simonw,2021-02-14T17:40:48Z,2021-02-14T17:40:48Z,OWNER,"Another pattern that might be useful is to generate a header that is just ""unknown1,unknown2,unknown3"" for each of the columns in the rest of the file. This makes it easy to e.g. facet-explore within Datasette to figure out the correct names, then use `sqlite-utils transform --rename` to rename the columns.

I needed to do that for the https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778511347,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778511347,MDEyOklzc3VlQ29tbWVudDc3ODUxMTM0Nw==,9599,simonw,2021-02-12T23:27:50Z,2021-02-12T23:27:50Z,OWNER,"For the moment, a workaround can be to `cat` an additional row onto the start of the file.

    echo ""name,url,description"" | cat - missing_headings.csv | sqlite-utils insert blah.db table - --csv","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778510528,https://api.github.com/repos/simonw/sqlite-utils/issues/131,778510528,MDEyOklzc3VlQ29tbWVudDc3ODUxMDUyOA==,9599,simonw,2021-02-12T23:25:06Z,2021-02-12T23:25:06Z,OWNER,"If `-c` isn't available, maybe `-t` or `--type` would work for specifying column types:
```
sqlite-utils insert db.db images images.tsv \
  --tsv \
  --type id int \
  --type score float
```
or
```
sqlite-utils insert db.db images images.tsv \
  --tsv \
  -t id int \
  -t score float
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778508887,https://api.github.com/repos/simonw/sqlite-utils/issues/131,778508887,MDEyOklzc3VlQ29tbWVudDc3ODUwODg4Nw==,9599,simonw,2021-02-12T23:20:11Z,2021-02-12T23:20:11Z,OWNER,"Annoyingly `-c` is currently a shortcut for `--csv` - so I'd have to do a major version bump to use that.

https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L601-L603

Particularly annoying because I attempted to remove the `-c` shortcut in https://github.com/simonw/sqlite-utils/commit/2c00567aac6d9c79087cfff0d054f64922b1473d#diff-76294b3d4afeb27e74e738daa01c26dd4dc9ccb6f4477451483a2ece1095902eL48 but forgot to remove it from the input options (I removed it from the output options).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778349672,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778349672,MDEyOklzc3VlQ29tbWVudDc3ODM0OTY3Mg==,9599,simonw,2021-02-12T18:00:43Z,2021-02-12T18:00:43Z,OWNER,"I could combine this with #131 to allow types to be specified in addition to column names.

Probably need an option that means ""ignore the existing heading row and use this one instead"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778349142,https://api.github.com/repos/simonw/sqlite-utils/issues/227,778349142,MDEyOklzc3VlQ29tbWVudDc3ODM0OTE0Mg==,9599,simonw,2021-02-12T17:59:35Z,2021-02-12T17:59:35Z,OWNER,It looks like I can at least bump this size limit up to the maximum allowed by Python - I'll take a look at that. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,Error reading csv files with large column data,
https://github.com/simonw/datasette/issues/1221#issuecomment-777901052,https://api.github.com/repos/simonw/datasette/issues/1221,777901052,MDEyOklzc3VlQ29tbWVudDc3NzkwMTA1Mg==,9599,simonw,2021-02-12T01:09:54Z,2021-02-12T01:09:54Z,OWNER,"I also tested this manually. I generated certificate files like so:

    cd /tmp
    python -m trustme

This created `/tmp/server.pem`, `/tmp/client.pem` and `/tmp/server.key`

Then I started Datasette like this:

    datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem

And exercise it using `curl` like so:

    /tmp % curl --cacert /tmp/client.pem 'https://localhost:8001/_memory.json'
    {""database"": ""_memory"", ""path"": ""/_memory"", ""size"": 0, ""tables"": [], ""hidden_count"": 0, ""views"": [], ""queries"": [],
    ""private"": false, ""allow_execute_sql"": true, ""query_ms"": 0.8843200000114848}

Note that without the `--cacert` option I get an error:

```
/tmp % curl 'https://localhost:8001/_memory.json' 
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806849424,Support SSL/TLS directly,
https://github.com/simonw/datasette/issues/1221#issuecomment-777887190,https://api.github.com/repos/simonw/datasette/issues/1221,777887190,MDEyOklzc3VlQ29tbWVudDc3Nzg4NzE5MA==,9599,simonw,2021-02-12T00:29:18Z,2021-02-12T00:29:18Z,OWNER,I can use this recipe to start a `datasette` server in a sub-process during the pytest run and exercise it with real HTTP requests: https://til.simonwillison.net/pytest/subprocess-server,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806849424,Support SSL/TLS directly,
https://github.com/simonw/datasette/issues/1221#issuecomment-777883452,https://api.github.com/repos/simonw/datasette/issues/1221,777883452,MDEyOklzc3VlQ29tbWVudDc3Nzg4MzQ1Mg==,9599,simonw,2021-02-12T00:19:30Z,2021-02-12T00:19:40Z,OWNER,"Uvicorn supports these options: https://www.uvicorn.org/#command-line-options
```
  --ssl-keyfile TEXT              SSL key file
  --ssl-certfile TEXT             SSL certificate file
  --ssl-keyfile-password TEXT     SSL keyfile password
  --ssl-version INTEGER           SSL version to use (see stdlib ssl module's)
                                  [default: 2]

  --ssl-cert-reqs INTEGER         Whether client certificate is required (see
                                  stdlib ssl module's)  [default: 0]

  --ssl-ca-certs TEXT             CA certificates file
  --ssl-ciphers TEXT              Ciphers to use (see stdlib ssl module's)
                                  [default: TLSv1]
```
For the moment I'm going to support just `--ssl-keyfile` and `--ssl-certfile` as arguments to `datasette serve`. I'll add other options if people ask for them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806849424,Support SSL/TLS directly,
https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10,777839351,MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ==,9599,simonw,2021-02-11T22:37:55Z,2021-02-11T22:37:55Z,MEMBER,"I've merged these changes by hand now, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770712149,BugFix for encoding and not update info.,
https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7,777827396,MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng==,9599,simonw,2021-02-11T22:13:14Z,2021-02-11T22:13:14Z,MEMBER,My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743297582,evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace',
https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9,777821383,MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw==,9599,simonw,2021-02-11T22:01:28Z,2021-02-11T22:01:28Z,MEMBER,"Aha! I think I've figured out what's going on here.

The CData blocks containing the notes look like this:

`<![CDATA[<!DOCTYPE en-note SYSTEM ""http://xml.evernote.com/pub/enml2.dtd""><en-note><div>This note includes two images.</div><div><br /></div>...`

The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:

```
<!--=========== External character mnemonic entities ===================-->

<!ENTITY % HTMLlat1 PUBLIC
   ""-//W3C//ENTITIES Latin 1 for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent"">
%HTMLlat1;

<!ENTITY % HTMLsymbol PUBLIC
   ""-//W3C//ENTITIES Symbols for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent"">
%HTMLsymbol;

<!ENTITY % HTMLspecial PUBLIC
   ""-//W3C//ENTITIES Special for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent"">
%HTMLspecial;
```
So I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this:

```python
{'Aacute': 'Á',
 'aacute': 'á',
 'Aacute;': 'Á',
 'aacute;': 'á',
 'Abreve;': 'Ă',
 'abreve;': 'ă',
 'ac;': '∾',
 'acd;': '∿',
# ...
}
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",748372469,ParseError: undefined entity &scaron;,
https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777798330,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11,777798330,MDEyOklzc3VlQ29tbWVudDc3Nzc5ODMzMA==,9599,simonw,2021-02-11T21:18:58Z,2021-02-11T21:18:58Z,MEMBER,Thanks for the fix!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792851444,XML parse error,
https://github.com/simonw/datasette/issues/1200#issuecomment-777178728,https://api.github.com/repos/simonw/datasette/issues/1200,777178728,MDEyOklzc3VlQ29tbWVudDc3NzE3ODcyOA==,9599,simonw,2021-02-11T03:13:59Z,2021-02-11T03:13:59Z,OWNER,"I came up with the need for this while playing with this tool: https://calands.datasettes.com/calands?sql=select%0D%0A++AsGeoJSON(geometry)%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+'%25mini%25'+and%0D%0A++Intersects(GeomFromGeoJSON(%3Afreedraw)%2C+geometry)+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+(%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+'CPAD_2020a_SuperUnits'%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON(%3Afreedraw)%0D%0A++)&freedraw={""type""%3A""MultiPolygon""%2C""coordinates""%3A[[[[-122.42202758789064%2C37.82280243352759]%2C[-122.39868164062501%2C37.823887203271454]%2C[-122.38220214843751%2C37.81846319511331]%2C[-122.35061645507814%2C37.77071473849611]%2C[-122.34924316406251%2C37.74465712069939]%2C[-122.37258911132814%2C37.703380457832374]%2C[-122.39044189453125%2C37.690340943717715]%2C[-122.41241455078126%2C37.680559803205135]%2C[-122.44262695312501%2C37.67295135774715]%2C[-122.47283935546876%2C37.67295135774715]%2C[-122.52502441406251%2C37.68382032669382]%2C[-122.53463745117189%2C37.6892542140253]%2C[-122.54699707031251%2C37.690340943717715]%2C[-122.55798339843751%2C37.72945260537781]%2C[-122.54287719726564%2C37.77831314799672]%2C[-122.49893188476564%2C37.81303878836991]%2C[-122.46185302734376%2C37.82822612280363]%2C[-122.42889404296876%2C37.82822612280363]%2C[-122.42202758789064%2C37.82280243352759]]]]} - before I fixed https://github.com/simonw/datasette-leaflet-geojson/issues/16 it was loading a LOT of maps, which felt bad. I wanted to be able to link people to that page with a hard limit on the number of rows displayed on that page.

It's mainly to guard against unexpected behaviour from limit-less queries though. It's not a very high priority feature!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792890765,?_size=10 option for the arbitrary query page would be useful,
https://github.com/simonw/datasette/issues/1219#issuecomment-775442039,https://api.github.com/repos/simonw/datasette/issues/1219,775442039,MDEyOklzc3VlQ29tbWVudDc3NTQ0MjAzOQ==,9599,simonw,2021-02-08T20:39:52Z,2021-02-08T22:13:00Z,OWNER,"This comment helped me find a pattern for running Scalene against the Datasette test suite: https://github.com/emeryberger/scalene/issues/70#issuecomment-755245858

```
pip install scalene
```
Then I created a file called `run_tests.py` with the following contents:
```python
if __name__ == ""__main__"":
    import sys, pytest
    pytest.main(sys.argv)
```
Then I ran this:
```
scalene --profile-all run_tests.py -sv -x .
```
But... it quit with a segmentation fault!
```
(datasette) datasette % scalene --profile-all run_tests.py -sv -x .
======================================================================== test session starts ========================================================================
platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- python
cachedir: .pytest_cache
rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
plugins: asyncio-0.14.0, timeout-1.4.2
collecting ... Fatal Python error: Segmentation fault

Current thread 0x0000000110c1edc0 (most recent call first):
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 553 in detect_json1
  File ""/Users/simon/Dropbox/Development/datasette/datasette/filters.py"", line 168 in Filters
  File ""/Users/simon/Dropbox/Development/datasette/datasette/filters.py"", line 94 in <module>
  File ""<frozen importlib._bootstrap>"", line 219 in _call_with_frames_removed
  File ""<frozen importlib._bootstrap_external>"", line 783 in exec_module
  File ""<frozen importlib._bootstrap>"", line 671 in _load_unlocked
  File ""<frozen importlib._bootstrap>"", line 975 in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 991 in _find_and_load
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 27 in <module>
  File ""<frozen importlib._bootstrap>"", line 219 in _call_with_frames_removed
  File ""<frozen importlib._bootstrap_external>"", line 783 in exec_module
  File ""<frozen importlib._bootstrap>"", line 671 in _load_unlocked
  File ""<frozen importlib._bootstrap>"", line 975 in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 991 in _find_and_load
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 42 in <module>
  File ""<frozen importlib._bootstrap>"", line 219 in _call_with_frames_removed
  File ""<frozen importlib._bootstrap_external>"", line 783 in exec_module
  File ""<frozen importlib._bootstrap>"", line 671 in _load_unlocked
  File ""<frozen importlib._bootstrap>"", line 975 in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 991 in _find_and_load
  File ""/Users/simon/Dropbox/Development/datasette/tests/test_api.py"", line 1 in <module>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/assertion/rewrite.py"", line 170 in exec_module
  File ""<frozen importlib._bootstrap>"", line 671 in _load_unlocked
  File ""<frozen importlib._bootstrap>"", line 975 in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 991 in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 1014 in _gcd_import
  File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py"", line 127 in import_module
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/pathlib.py"", line 520 in import_path
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 552 in _importtestmodule
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 484 in _getobj
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 288 in obj
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 500 in _inject_setup_module_fixture
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 487 in collect
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py"", line 324 in <lambda>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py"", line 294 in from_call
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py"", line 324 in pytest_make_collect_report
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py"", line 187 in _multicall
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 84 in <lambda>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 93 in _hookexec
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py"", line 286 in __call__
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py"", line 441 in collect_one_node
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 768 in genitems
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 771 in genitems
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 568 in _perform_collect
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 516 in perform_collect
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 306 in pytest_collection
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py"", line 187 in _multicall
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 84 in <lambda>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 93 in _hookexec
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py"", line 286 in __call__
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 295 in _main
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 240 in wrap_session
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 289 in pytest_cmdline_main
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py"", line 187 in _multicall
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 84 in <lambda>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 93 in _hookexec
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py"", line 286 in __call__
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/config/__init__.py"", line 157 in main
  File ""run_tests.py"", line 3 in <module>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/scalene_profiler.py"", line 1525 in main
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/__main__.py"", line 7 in main
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/__main__.py"", line 14 in <module>
  File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py"", line 87 in _run_code
  File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py"", line 194 in _run_module_as_main
Scalene error: received signal SIGSEGV
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803929694,Try profiling Datasette using scalene,
https://github.com/simonw/datasette/issues/1219#issuecomment-775497449,https://api.github.com/repos/simonw/datasette/issues/1219,775497449,MDEyOklzc3VlQ29tbWVudDc3NTQ5NzQ0OQ==,9599,simonw,2021-02-08T22:11:34Z,2021-02-08T22:11:34Z,OWNER,"https://github.com/emeryberger/scalene/issues/110 reports a ""received signal SIGSEGV"" error that was fixed by upgrading to the latest Scalene version, but I'm running that already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803929694,Try profiling Datasette using scalene,
https://github.com/simonw/sqlite-utils/issues/223#issuecomment-774373829,https://api.github.com/repos/simonw/sqlite-utils/issues/223,774373829,MDEyOklzc3VlQ29tbWVudDc3NDM3MzgyOQ==,9599,simonw,2021-02-06T01:39:47Z,2021-02-06T01:39:47Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-csv-tsv-delimiter,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788527932,--delimiter option for CSV import,
https://github.com/simonw/datasette/issues/1216#issuecomment-772796111,https://api.github.com/repos/simonw/datasette/issues/1216,772796111,MDEyOklzc3VlQ29tbWVudDc3Mjc5NjExMQ==,9599,simonw,2021-02-03T20:20:48Z,2021-02-03T20:20:48Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/1600d2a3ec3ada1f6fb5b1eb73bdaeccb5f80530/datasette/app.py#L620-L632,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",800669347,"/-/databases should reflect connection order, not alphabetical order",
https://github.com/simonw/datasette/issues/1214#issuecomment-772001787,https://api.github.com/repos/simonw/datasette/issues/1214,772001787,MDEyOklzc3VlQ29tbWVudDc3MjAwMTc4Nw==,9599,simonw,2021-02-02T21:28:53Z,2021-02-02T21:28:53Z,OWNER,"Fix is now live on https://latest.datasette.io/fixtures/searchable?_search=terry - clearing ""terry"" and re-submitting the form now works as expected.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799693777,Re-submitting filter form duplicates _x querystring arguments,
https://github.com/simonw/datasette/issues/1214#issuecomment-771992628,https://api.github.com/repos/simonw/datasette/issues/1214,771992628,MDEyOklzc3VlQ29tbWVudDc3MTk5MjYyOA==,9599,simonw,2021-02-02T21:15:18Z,2021-02-02T21:15:18Z,OWNER,"The cause of this bug is form fields which begin with `_` but ARE displayed as form inputs on the page - hence should not be duplicated in an `<input type=""hidden"">` element.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799693777,Re-submitting filter form duplicates _x querystring arguments,
https://github.com/simonw/datasette/issues/1214#issuecomment-771992025,https://api.github.com/repos/simonw/datasette/issues/1214,771992025,MDEyOklzc3VlQ29tbWVudDc3MTk5MjAyNQ==,9599,simonw,2021-02-02T21:14:16Z,2021-02-02T21:14:16Z,OWNER,"As a result, navigating to https://github-to-sqlite.dogsheep.net/github/labels?_search=help and clearing out the `_search` field then submitting the form does NOT clear the search term.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799693777,Re-submitting filter form duplicates _x querystring arguments,
https://github.com/simonw/datasette/issues/1212#issuecomment-771976561,https://api.github.com/repos/simonw/datasette/issues/1212,771976561,MDEyOklzc3VlQ29tbWVudDc3MTk3NjU2MQ==,9599,simonw,2021-02-02T20:53:27Z,2021-02-02T20:53:27Z,OWNER,"It would be great if we could get `python-xdist` to run too - I tried it in the past and gave up when I ran into those race conditions, but I've not done any further digging to see if there's a way to fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797651831,Tests are very slow. ,
https://github.com/simonw/datasette/issues/1212#issuecomment-771975941,https://api.github.com/repos/simonw/datasette/issues/1212,771975941,MDEyOklzc3VlQ29tbWVudDc3MTk3NTk0MQ==,9599,simonw,2021-02-02T20:52:36Z,2021-02-02T20:52:36Z,OWNER,"37 minutes, wow! They're a little slow for me (4-5 minutes perhaps) but not nearly that bad.

Thanks for running that profile. I think you're right: figuring out how to use more session scopes would definitely help.

The `:memory:` idea is interesting too. The new `memory_name=` feature added in #1151 (released in Datasette 0.54) could help a lot here, since it allows Datasette instances to share the same in-memory database across multiple HTTP requests and connections.

Note that `memory_name=` also persists within test runs themselves, independently of any `scope=` options on the fixtures. That might actually help us here!

I'd be delighted if you explored this issue further, especially the option of using `memory_name=` for the fixtures databases used by the tests.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797651831,Tests are very slow. ,
https://github.com/simonw/datasette/issues/1213#issuecomment-771968675,https://api.github.com/repos/simonw/datasette/issues/1213,771968675,MDEyOklzc3VlQ29tbWVudDc3MTk2ODY3NQ==,9599,simonw,2021-02-02T20:41:55Z,2021-02-02T20:41:55Z,OWNER,"So maybe I could a special response header which ASGI middleware can pick up that means ""Don't attempt to gzip this, just stream it through"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799663959,gzip support for HTML (and JSON) responses,
https://github.com/simonw/datasette/issues/1213#issuecomment-771968177,https://api.github.com/repos/simonw/datasette/issues/1213,771968177,MDEyOklzc3VlQ29tbWVudDc3MTk2ODE3Nw==,9599,simonw,2021-02-02T20:41:13Z,2021-02-02T20:41:13Z,OWNER,"Starlette accumulates the full response body in a `body` variable and then does this:
```python
        elif message_type == ""http.response.body"":
            # Remaining body in streaming GZip response.
            body = message.get(""body"", b"""")
            more_body = message.get(""more_body"", False)

            self.gzip_file.write(body)
            if not more_body:
                self.gzip_file.close()

            message[""body""] = self.gzip_buffer.getvalue()
            self.gzip_buffer.seek(0)
            self.gzip_buffer.truncate()

            await self.send(message)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799663959,gzip support for HTML (and JSON) responses,
https://github.com/simonw/datasette/issues/1213#issuecomment-771965281,https://api.github.com/repos/simonw/datasette/issues/1213,771965281,MDEyOklzc3VlQ29tbWVudDc3MTk2NTI4MQ==,9599,simonw,2021-02-02T20:37:08Z,2021-02-02T20:39:24Z,OWNER,Starlette's gzip middleware implementation is here: https://github.com/encode/starlette/blob/0.14.2/starlette/middleware/gzip.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799663959,gzip support for HTML (and JSON) responses,
https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60,770071568,MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA==,9599,simonw,2021-01-29T21:56:15Z,2021-01-29T21:56:15Z,MEMBER,"I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow:

https://github.com/dogsheep/github-to-sqlite/blob/62dfd3bc4014b108200001ef4bc746feb6f33b45/.github/workflows/deploy-demo.yml#L52-L82

`twitter-to-sqlite` actually has a mechanism for doing this kind of thing, documented at https://github.com/dogsheep/twitter-to-sqlite#providing-input-from-a-sql-query-with---sql-and---attach

It lets you do things like:

```
$ twitter-to-sqlite users-lookup my.db --sql=""select follower_id from following"" --ids
```
Maybe I should add something similar to `github-to-sqlite`? Feels like it could be really useful.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797097140,Use Data from SQLite in other commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56,769957751,MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ==,9599,simonw,2021-01-29T17:59:40Z,2021-01-29T17:59:40Z,MEMBER,"This is interesting - how did you create that initial table? Was this using the `twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip` command, or something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796736607,Not all quoted statuses get fetched?,
https://github.com/simonw/datasette/issues/1207#issuecomment-769534187,https://api.github.com/repos/simonw/datasette/issues/1207,769534187,MDEyOklzc3VlQ29tbWVudDc2OTUzNDE4Nw==,9599,simonw,2021-01-29T02:37:19Z,2021-01-29T02:37:19Z,OWNER,https://docs.datasette.io/en/latest/testing_plugins.html#using-pdb-for-errors-thrown-inside-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793881756,"Document the Datasette(..., pdb=True) testing pattern",
https://github.com/simonw/datasette/issues/1209#issuecomment-769455370,https://api.github.com/repos/simonw/datasette/issues/1209,769455370,MDEyOklzc3VlQ29tbWVudDc2OTQ1NTM3MA==,9599,simonw,2021-01-28T23:00:21Z,2021-01-28T23:00:21Z,OWNER,"Good catch on the workaround here. The root problem is that `datasette-template-sql` looks for the first available databsae if you don't provide it with a `database=` argument, and in Datasette 0.54 the first available database changed to being the new `_internal` database.

Is this a bug? I think it is - because the documented behaviour on https://docs.datasette.io/en/stable/internals.html#get-database-name is this:

> `name` - string, optional
>
> The name to be used for this database - this will be used in the URL path, e.g. `/dbname`. If not specified Datasette will pick one based on the filename or memory name.

Since the new behaviour differs from what was in the documentation I'm going to treat this as a bug and fix it.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",795367402,v0.54 500 error from sql query in custom template; code worked in v0.53; found a workaround,
https://github.com/simonw/datasette/issues/1205#issuecomment-769453074,https://api.github.com/repos/simonw/datasette/issues/1205,769453074,MDEyOklzc3VlQ29tbWVudDc2OTQ1MzA3NA==,9599,simonw,2021-01-28T22:54:49Z,2021-01-28T22:55:02Z,OWNER,"
I also checked that the following works:

    echo '{""foo"": ""bar""}' | sqlite-utils insert _memory.db demo -
    datasette _memory.db --memory

Sure enough, it results in the following Datasette homepage - thanks to #509

<img width=""274"" alt=""Datasette___memory___memory_2"" src=""https://user-images.githubusercontent.com/9599/106208790-c8564980-6178-11eb-8b8b-053a9f1d0193.png"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793027837,Rename /:memory: to /_memory,
https://github.com/simonw/datasette/issues/1205#issuecomment-769452084,https://api.github.com/repos/simonw/datasette/issues/1205,769452084,MDEyOklzc3VlQ29tbWVudDc2OTQ1MjA4NA==,9599,simonw,2021-01-28T22:52:23Z,2021-01-28T22:52:23Z,OWNER,Here are the redirect tests: https://github.com/simonw/datasette/blob/1600d2a3ec3ada1f6fb5b1eb73bdaeccb5f80530/tests/test_api.py#L635-L648,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793027837,Rename /:memory: to /_memory,
https://github.com/simonw/datasette/issues/1205#issuecomment-769442165,https://api.github.com/repos/simonw/datasette/issues/1205,769442165,MDEyOklzc3VlQ29tbWVudDc2OTQ0MjE2NQ==,9599,simonw,2021-01-28T22:30:16Z,2021-01-28T22:30:27Z,OWNER,"I'm going to do this, with redirects from `/:memory:*`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793027837,Rename /:memory: to /_memory,
https://github.com/simonw/datasette/issues/1210#issuecomment-769274591,https://api.github.com/repos/simonw/datasette/issues/1210,769274591,MDEyOklzc3VlQ29tbWVudDc2OTI3NDU5MQ==,9599,simonw,2021-01-28T18:10:02Z,2021-01-28T18:10:02Z,OWNER,That definitely sounds like a bug! Can you provide a copy of your `metadata.JSON` and the command-line you are using to launch Datasette?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796234313,Immutable Database w/ Canned Queries,
https://github.com/simonw/datasette/issues/1208#issuecomment-767823684,https://api.github.com/repos/simonw/datasette/issues/1208,767823684,MDEyOklzc3VlQ29tbWVudDc2NzgyMzY4NA==,9599,simonw,2021-01-26T20:58:51Z,2021-01-26T20:58:51Z,OWNER,"This is a good catch - I've been lazy about this, but you're right that it's an issue that needs cleaning up.  Would be very happy to apply a PR, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",794554881,A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper,
https://github.com/simonw/datasette/issues/1151#issuecomment-767762551,https://api.github.com/repos/simonw/datasette/issues/1151,767762551,MDEyOklzc3VlQ29tbWVudDc2Nzc2MjU1MQ==,9599,simonw,2021-01-26T19:07:44Z,2021-01-26T19:07:44Z,OWNER,Mentioned in https://simonwillison.net/2021/Jan/25/datasette/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/991#issuecomment-767761155,https://api.github.com/repos/simonw/datasette/issues/991,767761155,MDEyOklzc3VlQ29tbWVudDc2Nzc2MTE1NQ==,9599,simonw,2021-01-26T19:05:21Z,2021-01-26T19:06:36Z,OWNER,"Idea: implement this using the existing table view, with a custom template called `table-internal-bb0ec0-tables.html` - that's the custom template listed in the HTML comments at the bottom of https://latest.datasette.io/_internal/tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/1201#issuecomment-766991680,https://api.github.com/repos/simonw/datasette/issues/1201,766991680,MDEyOklzc3VlQ29tbWVudDc2Njk5MTY4MA==,9599,simonw,2021-01-25T17:42:21Z,2021-01-25T17:42:21Z,OWNER,https://docs.datasette.io/en/stable/changelog.html#v0-54,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/pull/1206#issuecomment-766588371,https://api.github.com/repos/simonw/datasette/issues/1206,766588371,MDEyOklzc3VlQ29tbWVudDc2NjU4ODM3MQ==,9599,simonw,2021-01-25T06:49:06Z,2021-01-25T06:49:06Z,OWNER,"Last thing to do: write up the annotated version of these release notes, assign it a URL on my blog and link to it from the release notes here so I can publish them simultaneously.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793086333,Release 0.54,
https://github.com/simonw/datasette/pull/1206#issuecomment-766588020,https://api.github.com/repos/simonw/datasette/issues/1206,766588020,MDEyOklzc3VlQ29tbWVudDc2NjU4ODAyMA==,9599,simonw,2021-01-25T06:48:20Z,2021-01-25T06:48:20Z,OWNER,"Issues to reference in the commit message: #509, #1091, #1150, #1151, #1166, #1167, #1178, #1181, #1182, #1184, #1185, #1186, #1187, #1194, #1198","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793086333,Release 0.54,
https://github.com/simonw/datasette/issues/1201#issuecomment-766586151,https://api.github.com/repos/simonw/datasette/issues/1201,766586151,MDEyOklzc3VlQ29tbWVudDc2NjU4NjE1MQ==,9599,simonw,2021-01-25T06:44:43Z,2021-01-25T06:44:43Z,OWNER,"OK, release notes are ready to merge from that branch. I'll ship the release in the morning, to give me time to write the accompanying annotated release notes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/issues/1201#issuecomment-766545604,https://api.github.com/repos/simonw/datasette/issues/1201,766545604,MDEyOklzc3VlQ29tbWVudDc2NjU0NTYwNA==,9599,simonw,2021-01-25T05:14:31Z,2021-01-25T05:14:31Z,OWNER,"The two big ticket items are `<script type=""module"">` support and the new `_internal` mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/issues/1201#issuecomment-766545442,https://api.github.com/repos/simonw/datasette/issues/1201,766545442,MDEyOklzc3VlQ29tbWVudDc2NjU0NTQ0Mg==,9599,simonw,2021-01-25T05:13:59Z,2021-01-25T05:13:59Z,OWNER,"The big stuff:

- Database(memory_name=) for shared in-memory databases, closes #1151 
- The `_internal` database -  #1150
-  script type=module support, closes #1186 , #1187
- Improved design for the `.add_database()` method 8919f99c2f7f245aca7f94bd53d5ac9d04aa42b5 - which means databases with the same stem can now be opened, #509
- Adopted Prettier #1166 

Smaller:

- force_https_urls on for publish cloudrun, refs #1178 
- Fixed bug in example nginx config, refs #1091 
-  Shrunk ecosystem docs in favour of datasette.io, closes #1182 
-  request.full_path property, closes #1184 
-  Better PRAGMA error message, closes #1185 
-  publish heroku now uses python-3.8.7 
-  Plugin testing documentation on using pytest-httpx Closes #1198
-  Contributing docs for Black and Prettier, closes #1167 
-  All ?_ parameters now copied to hidden form fields, closes #1194 
- Fixed bug loading database called 'test-database (1).sqlite' - Closes #1181.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/issues/1201#issuecomment-766543387,https://api.github.com/repos/simonw/datasette/issues/1201,766543387,MDEyOklzc3VlQ29tbWVudDc2NjU0MzM4Nw==,9599,simonw,2021-01-25T05:07:40Z,2021-01-25T05:13:29Z,OWNER,Changes: https://github.com/simonw/datasette/compare/0.53...a5ede3cdd455e2bb1a1fb2f4e1b5a9855caf5179,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/issues/983#issuecomment-766536076,https://api.github.com/repos/simonw/datasette/issues/983,766536076,MDEyOklzc3VlQ29tbWVudDc2NjUzNjA3Ng==,9599,simonw,2021-01-25T04:43:53Z,2021-01-25T04:43:53Z,OWNER,"... actually not going to include this in 0.54, I need to write a couple of plugins myself using it before I even make it available in preview.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1195#issuecomment-766535046,https://api.github.com/repos/simonw/datasette/issues/1195,766535046,MDEyOklzc3VlQ29tbWVudDc2NjUzNTA0Ng==,9599,simonw,2021-01-25T04:40:08Z,2021-01-25T04:40:08Z,OWNER,"Also: should the view name really be the same for both of these pages?

- https://latest.datasette.io/fixtures?sql=select+*+from+facetable
- https://latest.datasette.io/fixtures/neighborhood_search

Where one of them is a canned query and the other is an arbitrary query?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",789336592,"view_name = ""query"" for the query page",
https://github.com/simonw/datasette/issues/1195#issuecomment-766534748,https://api.github.com/repos/simonw/datasette/issues/1195,766534748,MDEyOklzc3VlQ29tbWVudDc2NjUzNDc0OA==,9599,simonw,2021-01-25T04:38:56Z,2021-01-25T04:38:56Z,OWNER,"Here's a diff showing how far I got before I abandoned this particular effort:

```diff
diff --git a/datasette/views/base.py b/datasette/views/base.py
index a21b929..04e4aa9 100644
--- a/datasette/views/base.py
+++ b/datasette/views/base.py
@@ -120,7 +120,7 @@ class BaseView:
         handler = getattr(self, request.method.lower(), None)
         return await handler(request, *args, **kwargs)
 
-    async def render(self, templates, request, context=None):
+    async def render(self, templates, request, context=None, view_name=None):
         context = context or {}
         template = self.ds.jinja_env.select_template(templates)
         template_context = {
@@ -135,7 +135,7 @@ class BaseView:
         }
         return Response.html(
             await self.ds.render_template(
-                template, template_context, request=request, view_name=self.name
+                template, template_context, request=request, view_name=view_name
             )
         )
 
@@ -155,7 +155,7 @@ class BaseView:
 
 
 class DataView(BaseView):
-    name = """"
+    view_name = ""no-view-name""
     re_named_parameter = re.compile("":([a-zA-Z0-9_]+)"")
 
     async def options(self, request, *args, **kwargs):
@@ -414,6 +414,10 @@ class DataView(BaseView):
             args[""table""] = urllib.parse.unquote_plus(args[""table""])
         return _format, args
 
+    async def get_view_name(self, request, database, hash, **kwargs):
+        # Defaults to self.view_name, but can be over-ridden by subclasses
+        return self.view_name
+
     async def view_get(self, request, database, hash, correct_hash_provided, **kwargs):
         _format, kwargs = await self.get_format(request, database, kwargs)
 
@@ -424,6 +428,8 @@ class DataView(BaseView):
             # HTML views default to expanding all foreign key labels
             kwargs[""default_labels""] = True
 
+        view_name = await self.get_view_name(request, database, hash, **kwargs)
+
         extra_template_data = {}
         start = time.perf_counter()
         status_code = 200
@@ -489,7 +495,7 @@ class DataView(BaseView):
                 database=database,
                 table=data.get(""table""),
                 request=request,
-                view_name=self.name,
+                view_name=view_name,
                 # These will be deprecated in Datasette 1.0:
                 args=request.args,
                 data=data,
@@ -533,7 +539,7 @@ class DataView(BaseView):
                     database=database,
                     table=data.get(""table""),
                     request=request,
-                    view_name=self.name,
+                    view_name=view_name,
                 )
                 it_can_render = await await_me_maybe(it_can_render)
                 if it_can_render:
@@ -565,7 +571,7 @@ class DataView(BaseView):
             }
             if ""metadata"" not in context:
                 context[""metadata""] = self.ds.metadata
-            r = await self.render(templates, request=request, context=context)
+            r = await self.render(templates, request=request, context=context, view_name=view_name)
             r.status = status_code
 
         ttl = request.args.get(""_ttl"", None)
diff --git a/datasette/views/database.py b/datasette/views/database.py
index f6fd579..e425213 100644
--- a/datasette/views/database.py
+++ b/datasette/views/database.py
@@ -23,7 +23,11 @@ from .base import DatasetteError, DataView
 
 
 class DatabaseView(DataView):
-    name = ""database""
+    async def get_view_name(self, request, db_name, table_and_format):
+        if request.args.get(""sql""):
+            return ""query""
+        else:
+            return ""database""
 
     async def data(self, request, database, hash, default_labels=False, _size=None):
         await self.check_permissions(
@@ -145,7 +149,7 @@ class DatabaseView(DataView):
 
 
 class DatabaseDownload(DataView):
-    name = ""database_download""
+    view_name = ""database_download""
 
     async def view_get(self, request, database, hash, correct_hash_present, **kwargs):
         await self.check_permissions(
diff --git a/datasette/views/index.py b/datasette/views/index.py
index b6b8cbe..d750e3d 100644
--- a/datasette/views/index.py
+++ b/datasette/views/index.py
@@ -16,7 +16,7 @@ COUNT_DB_SIZE_LIMIT = 100 * 1024 * 1024
 
 
 class IndexView(BaseView):
-    name = ""index""
+    view_name = ""index""
 
     async def get(self, request, as_format):
         await self.check_permission(request, ""view-instance"")
diff --git a/datasette/views/special.py b/datasette/views/special.py
index 9750dd0..dbd1e00 100644
--- a/datasette/views/special.py
+++ b/datasette/views/special.py
@@ -6,7 +6,7 @@ import secrets
 
 
 class JsonDataView(BaseView):
-    name = ""json_data""
+    view_name = ""json_data""
 
     def __init__(self, datasette, filename, data_callback, needs_request=False):
         self.ds = datasette
@@ -42,7 +42,7 @@ class JsonDataView(BaseView):
 
 
 class PatternPortfolioView(BaseView):
-    name = ""patterns""
+    view_name = ""patterns""
 
     async def get(self, request):
         await self.check_permission(request, ""view-instance"")
@@ -50,7 +50,7 @@ class PatternPortfolioView(BaseView):
 
 
 class AuthTokenView(BaseView):
-    name = ""auth_token""
+    view_name = ""auth_token""
 
     async def get(self, request):
         token = request.args.get(""token"") or """"
@@ -68,7 +68,7 @@ class AuthTokenView(BaseView):
 
 
 class LogoutView(BaseView):
-    name = ""logout""
+    view_name = ""logout""
 
     async def get(self, request):
         if not request.actor:
@@ -87,7 +87,7 @@ class LogoutView(BaseView):
 
 
 class PermissionsDebugView(BaseView):
-    name = ""permissions_debug""
+    view_name = ""permissions_debug""
 
     async def get(self, request):
         await self.check_permission(request, ""view-instance"")
@@ -102,7 +102,7 @@ class PermissionsDebugView(BaseView):
 
 
 class AllowDebugView(BaseView):
-    name = ""allow_debug""
+    view_name = ""allow_debug""
 
     async def get(self, request):
         errors = []
@@ -136,7 +136,7 @@ class AllowDebugView(BaseView):
 
 
 class MessagesDebugView(BaseView):
-    name = ""messages_debug""
+    view_name = ""messages_debug""
 
     async def get(self, request):
         await self.check_permission(request, ""view-instance"")
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 0a3504b..45d298a 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -257,7 +257,16 @@ class RowTableShared(DataView):
 
 
 class TableView(RowTableShared):
-    name = ""table""
+    view_name = ""table""
+
+    async def get_view_name(self, request, db_name, table_and_format):
+        canned_query = await self.ds.get_canned_query(
+            db_name, table_and_format, request.actor
+        )
+        if canned_query:
+            return ""query""
+        else:
+            return ""table""
 
     async def post(self, request, db_name, table_and_format):
         # Handle POST to a canned query
@@ -923,7 +932,7 @@ async def _sql_params_pks(db, table, pk_values):
 
 
 class RowView(RowTableShared):
-    name = ""row""
+    view_name = ""row""
 
     async def data(self, request, database, hash, table, pk_path, default_labels=False):
         await self.check_permissions(
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index 715c7c1..7ce2b1b 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -252,7 +252,7 @@ def test_plugin_config_file(app_client):
             },
         ),
         (
-            ""/fixtures/"",
+            ""/fixtures"",
             {
                 ""template"": ""database.html"",
                 ""database"": ""fixtures"",
@@ -285,6 +285,38 @@ def test_plugin_config_file(app_client):
                 ],
             },
         ),
+        (
+            ""/fixtures?sql=select+1+as+one"",
+            {
+                ""template"": ""query.html"",
+                ""database"": ""fixtures"",
+                ""table"": None,
+                ""config"": {""depth"": ""database""},
+                ""view_name"": ""query"",
+                ""request_path"": ""/fixtures"",
+                ""added"": 15,
+                ""columns"": [
+                    ""one"",
+                ],
+            },
+        ),
+        (
+            ""/fixtures/neighborhood_search"",
+            {
+                ""template"": ""query.html"",
+                ""database"": ""fixtures"",
+                ""table"": None,
+                ""config"": {""depth"": ""database""},
+                ""view_name"": ""query"",
+                ""request_path"": ""/fixtures/neighborhood_search"",
+                ""added"": 15,
+                ""columns"": [
+                    ""neighborhood"",
+                    ""name"",
+                    ""state"",
+                ],
+            },
+        ),
     ],
 )
 def test_hook_extra_body_script(app_client, path, expected_extra_body_script):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",789336592,"view_name = ""query"" for the query page",
https://github.com/simonw/datasette/issues/1195#issuecomment-766534634,https://api.github.com/repos/simonw/datasette/issues/1195,766534634,MDEyOklzc3VlQ29tbWVudDc2NjUzNDYzNA==,9599,simonw,2021-01-25T04:38:30Z,2021-01-25T04:38:30Z,OWNER,"This has proved surprisingly difficult to implement, due to the weird way the QueryView is actually called. The class itself isn't used like other view classes - instead, the `.data()` methods of both `DatabaseView` and `TableView` dispatch out to `QueryView.data()` when they need to:

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L259-L270

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L290-L294

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/database.py#L39-L44

It turns out this is a bad pattern because it makes changes like this one WAY harder than they should be.

I think I should clean this up as part of #878.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",789336592,"view_name = ""query"" for the query page",
https://github.com/simonw/datasette/pull/1204#issuecomment-766525337,https://api.github.com/repos/simonw/datasette/issues/1204,766525337,MDEyOklzc3VlQ29tbWVudDc2NjUyNTMzNw==,9599,simonw,2021-01-25T04:04:28Z,2021-01-25T04:04:28Z,OWNER,"Writing the tests will be a bit tricky since we need to confirm that the `include_table_top(datasette, database, actor, table)` arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes,
https://github.com/simonw/datasette/issues/987#issuecomment-766524141,https://api.github.com/repos/simonw/datasette/issues/987,766524141,MDEyOklzc3VlQ29tbWVudDc2NjUyNDE0MQ==,9599,simonw,2021-01-25T03:59:55Z,2021-01-25T03:59:55Z,OWNER,"This is joined with #1191 now, which I've bumped from 0.54.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/1191#issuecomment-766524016,https://api.github.com/repos/simonw/datasette/issues/1191,766524016,MDEyOklzc3VlQ29tbWVudDc2NjUyNDAxNg==,9599,simonw,2021-01-25T03:59:17Z,2021-01-25T03:59:17Z,OWNER,More work can happen in the PR: #1204,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1191#issuecomment-766523866,https://api.github.com/repos/simonw/datasette/issues/1191,766523866,MDEyOklzc3VlQ29tbWVudDc2NjUyMzg2Ng==,9599,simonw,2021-01-25T03:58:34Z,2021-01-25T03:58:34Z,OWNER,"I've got a good prototype working now, but I'm dropping this from the Datasette 0.54 milestone because it requires a bunch of additional work to make sure it is really well tested and documented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1194#issuecomment-766491911,https://api.github.com/repos/simonw/datasette/issues/1194,766491911,MDEyOklzc3VlQ29tbWVudDc2NjQ5MTkxMQ==,9599,simonw,2021-01-25T02:02:15Z,2021-01-25T02:02:15Z,OWNER,I'm going to copy across anything starting with an underscore.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788447787,?_size= argument is not persisted by hidden form fields in the table filters,
https://github.com/simonw/datasette/issues/1167#issuecomment-766491566,https://api.github.com/repos/simonw/datasette/issues/1167,766491566,MDEyOklzc3VlQ29tbWVudDc2NjQ5MTU2Ng==,9599,simonw,2021-01-25T02:01:19Z,2021-01-25T02:01:19Z,OWNER,New documentation section here (I documented Black as well): https://docs.datasette.io/en/latest/contributing.html#code-formatting,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777145954,Add Prettier to contributing documentation,
https://github.com/simonw/datasette/issues/1167#issuecomment-766487520,https://api.github.com/repos/simonw/datasette/issues/1167,766487520,MDEyOklzc3VlQ29tbWVudDc2NjQ4NzUyMA==,9599,simonw,2021-01-25T01:44:43Z,2021-01-25T01:44:43Z,OWNER,"Thanks @benpickles, I just merged that in. I'll use it in the documentation.

    # To check code is conformant
    npm run prettier -- --check
    # To fix it if it isn't
    npm run fix
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777145954,Add Prettier to contributing documentation,
https://github.com/simonw/datasette/issues/983#issuecomment-766484915,https://api.github.com/repos/simonw/datasette/issues/983,766484915,MDEyOklzc3VlQ29tbWVudDc2NjQ4NDkxNQ==,9599,simonw,2021-01-25T01:33:29Z,2021-01-25T01:33:29Z,OWNER,I'm going to ship a version of this in Datasette 0.54 with a warning that the interface should be considered unstable (see #1202) so that we can start trying this out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1179#issuecomment-766484435,https://api.github.com/repos/simonw/datasette/issues/1179,766484435,MDEyOklzc3VlQ29tbWVudDc2NjQ4NDQzNQ==,9599,simonw,2021-01-25T01:31:36Z,2021-01-25T01:31:36Z,OWNER,Relevant existing tests: https://github.com/simonw/datasette/blob/461670a0b87efa953141b449a9a261919864ceb3/tests/test_utils.py#L365-L398,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1179#issuecomment-766484257,https://api.github.com/repos/simonw/datasette/issues/1179,766484257,MDEyOklzc3VlQ29tbWVudDc2NjQ4NDI1Nw==,9599,simonw,2021-01-25T01:30:57Z,2021-01-25T01:30:57Z,OWNER,"The challenge here is figuring out what the original path, without the `.format`, actually was - while taking into account that Datasette has a special case for tables that themselves end in a `.something`.

The `path_with_format()` function nearly does what we need here:

https://github.com/simonw/datasette/blob/b6a7b58fa01af0cd5a5e94bd17d686d283a46819/datasette/utils/__init__.py#L710-L729

It can be called with `replace_format=""csv""` to REMOVE the `.csv` format and replace it with something else.

Problem is, we want to use it to get rid of the format entirely.

We could update `path_with_format()` to accept `format=''` to mean ""remove the format entirely"", but it's a bit messy. It may be better to reconsider the design of `path_with_format()` and related utility functions entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1191#issuecomment-766466030,https://api.github.com/repos/simonw/datasette/issues/1191,766466030,MDEyOklzc3VlQ29tbWVudDc2NjQ2NjAzMA==,9599,simonw,2021-01-25T00:11:04Z,2021-01-25T00:11:04Z,OWNER,"I can combine this with #987 - each of these areas of the page can be wrapped in a `<div>` with a class that matches the name of the plugin hook, that way JavaScript plugins can append their content in the same place as Python plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1154#issuecomment-766465719,https://api.github.com/repos/simonw/datasette/issues/1154,766465719,MDEyOklzc3VlQ29tbWVudDc2NjQ2NTcxOQ==,9599,simonw,2021-01-25T00:09:22Z,2021-01-25T00:09:22Z,OWNER,"https://docs.datasette.io/en/latest/internals.html#the-internal-database

<img width=""738"" alt=""Internals_for_plugins_—_Datasette_documentation"" src=""https://user-images.githubusercontent.com/9599/105648087-76eb4900-5e5e-11eb-9c73-c00ba37b7b10.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771208009,Documentation for new _internal database and tables,
https://github.com/simonw/datasette/issues/1202#issuecomment-766464136,https://api.github.com/repos/simonw/datasette/issues/1202,766464136,MDEyOklzc3VlQ29tbWVudDc2NjQ2NDEzNg==,9599,simonw,2021-01-25T00:01:02Z,2021-01-25T00:01:02Z,OWNER,I'm going to use the existing `.. warning::` pattern for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792931244,Documentation convention for marking unstable APIs.,
https://github.com/simonw/datasette/issues/1090#issuecomment-766463496,https://api.github.com/repos/simonw/datasette/issues/1090,766463496,MDEyOklzc3VlQ29tbWVudDc2NjQ2MzQ5Ng==,9599,simonw,2021-01-24T23:57:00Z,2021-01-24T23:57:00Z,OWNER,Related: I built [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw) which turns any canned query field called `freedraw` or `something_freedraw` into an interactive map that you can draw on to create a GeoJSON MultiPolygon.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1202#issuecomment-766462475,https://api.github.com/repos/simonw/datasette/issues/1202,766462475,MDEyOklzc3VlQ29tbWVudDc2NjQ2MjQ3NQ==,9599,simonw,2021-01-24T23:49:28Z,2021-01-24T23:50:33Z,OWNER,"Can use an ""admonition"" similar to this:
```sphinx
.. warning::
    Restricting access to tables and views in this way will NOT prevent users from querying them using arbitrary SQL queries, `like this <https://latest.datasette.io/fixtures?sql=select+*+from+facetable>`__ for example.
```
As seen on https://docs.datasette.io/en/stable/authentication.html#controlling-access-to-specific-tables-and-views

Documentation: https://docutils.sourceforge.io/docs/ref/rst/directives.html#specific-admonitions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792931244,Documentation convention for marking unstable APIs.,
https://github.com/simonw/datasette/issues/1154#issuecomment-766462197,https://api.github.com/repos/simonw/datasette/issues/1154,766462197,MDEyOklzc3VlQ29tbWVudDc2NjQ2MjE5Nw==,9599,simonw,2021-01-24T23:47:06Z,2021-01-24T23:47:06Z,OWNER,"I'm going to document this but mark it as unstable, using a new documentation convention for marking unstable APIs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771208009,Documentation for new _internal database and tables,
https://github.com/simonw/datasette/issues/1179#issuecomment-766434629,https://api.github.com/repos/simonw/datasette/issues/1179,766434629,MDEyOklzc3VlQ29tbWVudDc2NjQzNDYyOQ==,9599,simonw,2021-01-24T21:23:47Z,2021-01-24T21:23:47Z,OWNER,I'm just going to do `path` and `full_path` (which includes the querystring)`. The `datasette.absolute_url()` method can be used by plugins that need the full URL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1190#issuecomment-766433153,https://api.github.com/repos/simonw/datasette/issues/1190,766433153,MDEyOklzc3VlQ29tbWVudDc2NjQzMzE1Mw==,9599,simonw,2021-01-24T21:13:25Z,2021-01-24T21:13:25Z,OWNER,"This ties in to a bunch of other ideas that are in flight at the moment.

If you're publishing databases by uploading them, how do you attach metadata? Ideally by baking it into the database file itself, using the mechanism from #1169.

How could this interact with the `datasette insert` concept from #1163? Could you pass a CSV file to the `upload` command and have that converted and uploaded for you, or would you create the database file locally using `datasette insert` and then upload it as a separate `datasette upload` step?

Lots to think about here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098146,`datasette publish upload` mechanism for uploading databases to an existing Datasette instance,
https://github.com/simonw/datasette/issues/1190#issuecomment-766430644,https://api.github.com/repos/simonw/datasette/issues/1190,766430644,MDEyOklzc3VlQ29tbWVudDc2NjQzMDY0NA==,9599,simonw,2021-01-24T20:57:03Z,2021-01-24T20:57:03Z,OWNER,"I really like this idea. It feels like an opportunity for a plugin that adds two things: an API endpoint to Datasette for accepting uploaded databases, and a `datasette publish upload` subcommand which can upload files to that endpoint (with some kind of authentication mechanism).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098146,`datasette publish upload` mechanism for uploading databases to an existing Datasette instance,
https://github.com/simonw/datasette/issues/1197#issuecomment-766430111,https://api.github.com/repos/simonw/datasette/issues/1197,766430111,MDEyOklzc3VlQ29tbWVudDc2NjQzMDExMQ==,9599,simonw,2021-01-24T20:53:40Z,2021-01-24T20:53:40Z,OWNER,"https://devcenter.heroku.com/articles/slug-compiler#slug-size says that the maximum allowed size is 500MB - my hunch is that the Datasette application itself weighs in at only a dozen or so MB but I haven't measured it. So I would imagine anything up to around 450MB should work OK on Heroku.

Cloud Run works for up to about 2GB in my experience.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",791381623,DB size limit for publishing with Heroku,
https://github.com/simonw/datasette/issues/1198#issuecomment-766428183,https://api.github.com/repos/simonw/datasette/issues/1198,766428183,MDEyOklzc3VlQ29tbWVudDc2NjQyODE4Mw==,9599,simonw,2021-01-24T20:40:37Z,2021-01-24T20:40:37Z,OWNER,https://docs.datasette.io/en/latest/testing_plugins.html#testing-outbound-http-calls-with-pytest-httpx,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792625812,Plugin testing documentation on using pytest-httpx,
https://github.com/simonw/datasette/issues/1199#issuecomment-766181628,https://api.github.com/repos/simonw/datasette/issues/1199,766181628,MDEyOklzc3VlQ29tbWVudDc2NjE4MTYyOA==,9599,simonw,2021-01-23T21:25:18Z,2021-01-23T21:25:18Z,OWNER,"Comment thread here: https://news.ycombinator.com/item?id=25881911 - cperciva says:

>  There's an even better reason for databases to not write to memory mapped pages: Pages get synched out to disk at the kernel's leisure. This can be ok for a cache but it's definitely not what you want for a database!

But... Datasette is often used in read-only mode, so that disadvantage often doesn't apply.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792652391,Experiment with PRAGMA mmap_size=N,
https://github.com/simonw/datasette/issues/1191#issuecomment-765757433,https://api.github.com/repos/simonw/datasette/issues/1191,765757433,MDEyOklzc3VlQ29tbWVudDc2NTc1NzQzMw==,9599,simonw,2021-01-22T23:43:43Z,2021-01-22T23:43:43Z,OWNER,"Another potential use for this: plugins that provide authentication (like `datasette-auth-passwords` and `datasette-auth-github`) could use it to add a chunk of HTML to the ""permission denied"" page that links to their mechanism of authenticating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1195#issuecomment-763108730,https://api.github.com/repos/simonw/datasette/issues/1195,763108730,MDEyOklzc3VlQ29tbWVudDc2MzEwODczMA==,9599,simonw,2021-01-19T20:22:37Z,2021-01-19T20:22:37Z,OWNER,I can use this test: https://github.com/simonw/datasette/blob/c38c42948cbfddd587729413fd6082ba352eaece/tests/test_plugins.py#L238-L294,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",789336592,"view_name = ""query"" for the query page",
https://github.com/simonw/sqlite-utils/issues/223#issuecomment-762540514,https://api.github.com/repos/simonw/sqlite-utils/issues/223,762540514,MDEyOklzc3VlQ29tbWVudDc2MjU0MDUxNA==,9599,simonw,2021-01-19T01:14:58Z,2021-01-19T01:15:54Z,OWNER,"Example from https://docs.python.org/3/library/csv.html#csv.reader
```pycon
>>> import csv
>>> with open('eggs.csv', newline='') as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
```
I'm going to add `--quotechar` as well as `--delimiter`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788527932,--delimiter option for CSV import,
https://github.com/simonw/datasette/issues/1194#issuecomment-762390568,https://api.github.com/repos/simonw/datasette/issues/1194,762390568,MDEyOklzc3VlQ29tbWVudDc2MjM5MDU2OA==,9599,simonw,2021-01-18T17:43:03Z,2021-01-18T17:43:03Z,OWNER,Should I just blanket copy over any query string argument that starts with an underscore? Any reason _not_ to do that?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788447787,?_size= argument is not persisted by hidden form fields in the table filters,
https://github.com/simonw/datasette/issues/1194#issuecomment-762390401,https://api.github.com/repos/simonw/datasette/issues/1194,762390401,MDEyOklzc3VlQ29tbWVudDc2MjM5MDQwMQ==,9599,simonw,2021-01-18T17:42:38Z,2021-01-18T17:42:38Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/views/table.py#L815-L827

It looks like there are other arguments that may not be persisted too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788447787,?_size= argument is not persisted by hidden form fields in the table filters,
https://github.com/simonw/datasette/issues/1036#issuecomment-762387875,https://api.github.com/repos/simonw/datasette/issues/1036,762387875,MDEyOklzc3VlQ29tbWVudDc2MjM4Nzg3NQ==,9599,simonw,2021-01-18T17:36:36Z,2021-01-18T17:36:36Z,OWNER,"As you can see, I'm pretty paranoid about serving content with `Content-Type` HTTP headers because I'm so worried about execution vulnerabilities. I'm much more comfortable exploring that kind of thing in plugins, since that way people can opt-in to riskier features.

You found `datasette-media` which is my most comprehensive exploration of that idea so far - but there's definitely lots of room for more plugins along those lines.

Maybe even an output plugin? `.jpg` as an export format which returns the `BLOB` column for a row as a JPEG image with the correct `content-type` header (but first verifies that the binary content does indeed look like a real JPEG) could be interesting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-761967094,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11,761967094,MDEyOklzc3VlQ29tbWVudDc2MTk2NzA5NA==,9599,simonw,2021-01-18T04:11:13Z,2021-01-18T04:11:13Z,MEMBER,"I just got a similar error:

```
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/swarm_to_sqlite/utils.py"", line 79, in save_checkin
    checkins_table.m2m(""users"", user, m2m_table=""with"", pk=""id"")
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 2048, in m2m
    id = other_table.insert(record, pk=pk, replace=True).last_pk
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1781, in insert
    return self.insert_all(
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1899, in insert_all
    self.insert_chunk(
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1709, in insert_chunk
    result = self.db.execute(query, params)
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 226, in execute
    return self.conn.execute(sql, parameters)
pysqlite3.dbapi2.OperationalError: table users has no column named countryCode

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743400216,Error thrown: sqlite3.OperationalError: table users has no column named lastName,
https://github.com/simonw/datasette/issues/1191#issuecomment-761705076,https://api.github.com/repos/simonw/datasette/issues/1191,761705076,MDEyOklzc3VlQ29tbWVudDc2MTcwNTA3Ng==,9599,simonw,2021-01-17T00:35:13Z,2021-01-17T00:37:51Z,OWNER,"I'm going to try using Jinja macros to implement this: https://jinja.palletsprojects.com/en/2.11.x/templates/#macros

Maybe using one of these tricks to auto-load the macro? http://codyaray.com/2015/05/auto-load-jinja2-macros","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1181#issuecomment-761703555,https://api.github.com/repos/simonw/datasette/issues/1181,761703555,MDEyOklzc3VlQ29tbWVudDc2MTcwMzU1NQ==,9599,simonw,2021-01-17T00:24:20Z,2021-01-17T00:24:40Z,OWNER,"Here's the incomplete sketch of a test - to go at the bottom of `test_cli.py`.

```python
@pytest.mark.parametrize(
    ""filename"", [""test-database (1).sqlite"", ""database (1).sqlite""]
)
def test_weird_database_names(ensure_eventloop, tmpdir, filename):
    # https://github.com/simonw/datasette/issues/1181
    runner = CliRunner()
    db_path = str(tmpdir / filename)
    sqlite3.connect(db_path).execute(""vacuum"")
    result1 = runner.invoke(cli, [db_path, ""--get"", ""/""])
    assert result1.exit_code == 0, result1.output
    homepage_html = result1.output
    assert False
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",781262510,"Certain database names results in 404: ""Database not found: None""",
https://github.com/simonw/datasette/issues/1191#issuecomment-761703368,https://api.github.com/repos/simonw/datasette/issues/1191,761703368,MDEyOklzc3VlQ29tbWVudDc2MTcwMzM2OA==,9599,simonw,2021-01-17T00:22:46Z,2021-01-17T00:22:46Z,OWNER,I'm going to prototype this in a branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1191#issuecomment-761703232,https://api.github.com/repos/simonw/datasette/issues/1191,761703232,MDEyOklzc3VlQ29tbWVudDc2MTcwMzIzMg==,9599,simonw,2021-01-17T00:21:31Z,2021-01-17T00:21:54Z,OWNER,"I think this ends up being a whole collection of new plugin hooks, something like:

- `include_table_top`
- `include_table_bottom`
- `include_row_top`
- `include_row_bottom`
- `include_database_top`
- `include_database_bottom`
- `include_query_bottom`
- `include_query_bottom`
- `include_index_bottom`
- `include_index_bottom`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1191#issuecomment-761703022,https://api.github.com/repos/simonw/datasette/issues/1191,761703022,MDEyOklzc3VlQ29tbWVudDc2MTcwMzAyMg==,9599,simonw,2021-01-17T00:20:00Z,2021-01-17T00:20:00Z,OWNER,"Plugins that want to provide extra context to the template can already do so using the `extra_template_vars()` plugin hook.

So this hook could work by returning a list of template filenames to be included. Those templates can be bundled with the plugin. Since they are included they will have access to the template context and to any `extra_template_vars()` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/657#issuecomment-761179229,https://api.github.com/repos/simonw/datasette/issues/657,761179229,MDEyOklzc3VlQ29tbWVudDc2MTE3OTIyOQ==,9599,simonw,2021-01-15T20:24:35Z,2021-01-15T20:24:35Z,OWNER,"I'm not sure how I missed this issue but it's almost a year later and I'm finally taking a look at your Parquet work.

This is yet more evidence that allowing plugins to provide their own custom `Database` objects would be a good idea.

I started exploring what Datasette would like on PostgreSQL in #670 - my concern was that I would need to add a large amount of database abstraction code which would dramatically increase the complexity of the core project, but my thinking now is that it might be tractable - Datasette doesn't actually construct SQL in complex ways anywhere outside of the `TableView` class so abstracting away just that bit should be feasible.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",548591089,Allow creation of virtual tables at startup,
https://github.com/simonw/datasette/issues/1191#issuecomment-761103910,https://api.github.com/repos/simonw/datasette/issues/1191,761103910,MDEyOklzc3VlQ29tbWVudDc2MTEwMzkxMA==,9599,simonw,2021-01-15T18:19:29Z,2021-01-15T18:19:29Z,OWNER,This relates to #987 (documented HTML hooks for JavaScript plugins) but is not quite the same thing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/657#issuecomment-761101878,https://api.github.com/repos/simonw/datasette/issues/657,761101878,MDEyOklzc3VlQ29tbWVudDc2MTEwMTg3OA==,9599,simonw,2021-01-15T18:16:01Z,2021-01-15T18:16:01Z,OWNER,"I think the `startup()` plugin hook at https://docs.datasette.io/en/stable/plugin_hooks.html#startup-datasette should be able to fit this. You can write a plugin which uses that hook to execute `CREATE VIRTUAL TABLE` against  one or more databases when Datasette first starts running.

Would that work here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",548591089,Allow creation of virtual tables at startup,
https://github.com/simonw/datasette/issues/1187#issuecomment-759875239,https://api.github.com/repos/simonw/datasette/issues/1187,759875239,MDEyOklzc3VlQ29tbWVudDc1OTg3NTIzOQ==,9599,simonw,2021-01-14T02:02:24Z,2021-01-14T02:02:31Z,OWNER,"This plugin hook currently returns a string of JavaScript. It could optionally return this instead of a string:
```json
{
  ""script"": ""string of JavaScript goes here"",
  ""module"": true
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",785588942,"extra_body_script() support for script type=""module""",
https://github.com/simonw/datasette/issues/1186#issuecomment-759874332,https://api.github.com/repos/simonw/datasette/issues/1186,759874332,MDEyOklzc3VlQ29tbWVudDc1OTg3NDMzMg==,9599,simonw,2021-01-14T01:59:35Z,2021-01-14T01:59:35Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript and https://docs.datasette.io/en/latest/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",785573793,"script type=""module"" support",
https://github.com/simonw/sqlite-utils/issues/220#issuecomment-759098964,https://api.github.com/repos/simonw/sqlite-utils/issues/220,759098964,MDEyOklzc3VlQ29tbWVudDc1OTA5ODk2NA==,9599,simonw,2021-01-12T23:19:55Z,2021-01-12T23:19:55Z,OWNER,"I don't think it makes sense to call `.enable_fts()` on a view does it? When I'm working with views and FTS I tend to write my queries against a FTS table for one of the tables that is used by the view - https://docs.datasette.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view describes how I do that in Datasette for example.

Can you expand on your use-case for FTS and views? I'm ready to be convinced otherwise, but I don't see how it would work right now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",783778672,Better error message for *_fts methods against views,
https://github.com/simonw/datasette/issues/1185#issuecomment-759069342,https://api.github.com/repos/simonw/datasette/issues/1185,759069342,MDEyOklzc3VlQ29tbWVudDc1OTA2OTM0Mg==,9599,simonw,2021-01-12T22:13:18Z,2021-01-12T22:13:18Z,OWNER,I'm going to change the error message to list the allowed pragmas.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",784628163,"""Statement may not contain PRAGMA"" error is not strictly true",
https://github.com/simonw/datasette/issues/1185#issuecomment-759067427,https://api.github.com/repos/simonw/datasette/issues/1185,759067427,MDEyOklzc3VlQ29tbWVudDc1OTA2NzQyNw==,9599,simonw,2021-01-12T22:09:21Z,2021-01-12T22:09:21Z,OWNER,"That allow-list was added in #761 but is not currently documented. It's here in the code:

https://github.com/simonw/datasette/blob/8e8fc5cee5c78da8334495c6d6257d5612c40792/datasette/utils/__init__.py#L173-L186","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",784628163,"""Statement may not contain PRAGMA"" error is not strictly true",
https://github.com/simonw/datasette/issues/1185#issuecomment-759066777,https://api.github.com/repos/simonw/datasette/issues/1185,759066777,MDEyOklzc3VlQ29tbWVudDc1OTA2Njc3Nw==,9599,simonw,2021-01-12T22:07:58Z,2021-01-12T22:07:58Z,OWNER,"https://docs.datasette.io/en/stable/sql_queries.html?highlight=pragma#named-parameters documentation is out-of-date as well:

> Datasette disallows custom SQL containing the string PRAGMA, as SQLite pragma statements can be used to change database settings at runtime. If you need to include the string ""pragma"" in a query you can do so safely using a named parameter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",784628163,"""Statement may not contain PRAGMA"" error is not strictly true",
https://github.com/simonw/datasette/issues/1183#issuecomment-758356640,https://api.github.com/repos/simonw/datasette/issues/1183,758356640,MDEyOklzc3VlQ29tbWVudDc1ODM1NjY0MA==,9599,simonw,2021-01-12T02:42:08Z,2021-01-12T02:42:08Z,OWNER,"Should Datasette have subcommands for this? `datasette enable-counts data.db` and `datasette disable-counts data.db` and `datasette reset-counts data.db` commands?

Maybe. The `sqlite-utils` CLI tool could be used here instead, but that won't be easily available if Datasette was installed as a standalone binary or using `brew install datasette` or `pipx install datasette`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782708469,"Take advantage of sqlite-utils cached table counts, if available",
https://github.com/simonw/datasette/issues/1183#issuecomment-758356097,https://api.github.com/repos/simonw/datasette/issues/1183,758356097,MDEyOklzc3VlQ29tbWVudDc1ODM1NjA5Nw==,9599,simonw,2021-01-12T02:40:30Z,2021-01-12T02:40:30Z,OWNER,"So how would this work?

I think I'm going to automatically use these values if the `_counts` table exists, unless a `ignore_counts_table` boolean setting has been set. I won't bother looking to see if the triggers have been created.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782708469,"Take advantage of sqlite-utils cached table counts, if available",
https://github.com/simonw/datasette/issues/1091#issuecomment-758283074,https://api.github.com/repos/simonw/datasette/issues/1091,758283074,MDEyOklzc3VlQ29tbWVudDc1ODI4MzA3NA==,9599,simonw,2021-01-11T23:12:46Z,2021-01-11T23:12:46Z,OWNER,Fantastic!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1182#issuecomment-757375858,https://api.github.com/repos/simonw/datasette/issues/1182,757375858,MDEyOklzc3VlQ29tbWVudDc1NzM3NTg1OA==,9599,simonw,2021-01-09T22:18:47Z,2021-01-09T22:18:47Z,OWNER,https://docs.datasette.io/en/latest/ecosystem.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782692159,"Retire ""Ecosystem"" page in favour of datasette.io/plugins and /tools",
https://github.com/simonw/datasette/issues/1182#issuecomment-757373741,https://api.github.com/repos/simonw/datasette/issues/1182,757373741,MDEyOklzc3VlQ29tbWVudDc1NzM3Mzc0MQ==,9599,simonw,2021-01-09T22:01:41Z,2021-01-09T22:01:41Z,OWNER,It can talk about Dogsheep too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782692159,"Retire ""Ecosystem"" page in favour of datasette.io/plugins and /tools",
https://github.com/simonw/datasette/issues/1182#issuecomment-757373082,https://api.github.com/repos/simonw/datasette/issues/1182,757373082,MDEyOklzc3VlQ29tbWVudDc1NzM3MzA4Mg==,9599,simonw,2021-01-09T21:55:33Z,2021-01-09T21:55:33Z,OWNER,I'll leave the page there but change it into more of a blurb about the existence of the plugins and tools directories.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782692159,"Retire ""Ecosystem"" page in favour of datasette.io/plugins and /tools",
https://github.com/simonw/datasette/issues/1181#issuecomment-756487966,https://api.github.com/repos/simonw/datasette/issues/1181,756487966,MDEyOklzc3VlQ29tbWVudDc1NjQ4Nzk2Ng==,9599,simonw,2021-01-08T01:25:42Z,2021-01-08T01:25:42Z,OWNER,I'm going to add a unit test that tries a variety of weird database names.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",781262510,"Certain database names results in 404: ""Database not found: None""",
https://github.com/simonw/datasette/issues/1181#issuecomment-756482163,https://api.github.com/repos/simonw/datasette/issues/1181,756482163,MDEyOklzc3VlQ29tbWVudDc1NjQ4MjE2Mw==,9599,simonw,2021-01-08T01:06:23Z,2021-01-08T01:06:54Z,OWNER,"Yes, that logic is definitely at fault. It looks like it applies `urllib.parse.unquote_plus()` AFTER it's tried to do the `-` hash splitting thing, which is why it's failing here:

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184-L198","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",781262510,"Certain database names results in 404: ""Database not found: None""",
https://github.com/simonw/datasette/issues/1091#issuecomment-756453945,https://api.github.com/repos/simonw/datasette/issues/1091,756453945,MDEyOklzc3VlQ29tbWVudDc1NjQ1Mzk0NQ==,9599,simonw,2021-01-07T23:42:50Z,2021-01-07T23:42:50Z,OWNER,"@henry501 it looks like you spotted a bug in the documentation - I just addressed that, the fix is now live here: https://docs.datasette.io/en/latest/deploying.html#running-datasette-behind-a-proxy","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-756453010,https://api.github.com/repos/simonw/datasette/issues/1091,756453010,MDEyOklzc3VlQ29tbWVudDc1NjQ1MzAxMA==,9599,simonw,2021-01-07T23:39:58Z,2021-01-07T23:40:25Z,OWNER,"@tballison I think that's the solution! It looks like you need to use this in your config:

`ProxyPass /datasette http://127.0.0.1:8001/datasette`

Instead of this:

`ProxyPass /datasette http://127.0.0.1:8001/`

Give that a go and let me know if it fixes it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1171#issuecomment-756335394,https://api.github.com/repos/simonw/datasette/issues/1171,756335394,MDEyOklzc3VlQ29tbWVudDc1NjMzNTM5NA==,9599,simonw,2021-01-07T19:35:59Z,2021-01-07T19:35:59Z,OWNER,"I requested a D-U-N-S number as a first step in getting a developer certificate: https://developer.apple.com/support/D-U-N-S/
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1180#issuecomment-756312213,https://api.github.com/repos/simonw/datasette/issues/1180,756312213,MDEyOklzc3VlQ29tbWVudDc1NjMxMjIxMw==,9599,simonw,2021-01-07T18:56:24Z,2021-01-07T18:56:24Z,OWNER,The `async_call_with_supported_arguments` version should be able to await any of the lazy arguments that are awaitable - can use `await_me_maybe` for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780767542,Lazily evaluated arguments for call_with_supported_arguments,
https://github.com/simonw/datasette/issues/1180#issuecomment-755500475,https://api.github.com/repos/simonw/datasette/issues/1180,755500475,MDEyOklzc3VlQ29tbWVudDc1NTUwMDQ3NQ==,9599,simonw,2021-01-06T18:43:41Z,2021-01-06T18:43:41Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/__init__.py#L919-L940,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780767542,Lazily evaluated arguments for call_with_supported_arguments,
https://github.com/simonw/datasette/issues/1179#issuecomment-755495387,https://api.github.com/repos/simonw/datasette/issues/1179,755495387,MDEyOklzc3VlQ29tbWVudDc1NTQ5NTM4Nw==,9599,simonw,2021-01-06T18:39:23Z,2021-01-06T18:39:23Z,OWNER,"In that case maybe there are three new arguments: `path`, `full_path` and `url`.

I'll also add `request.full_path` for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1179#issuecomment-755492945,https://api.github.com/repos/simonw/datasette/issues/1179,755492945,MDEyOklzc3VlQ29tbWVudDc1NTQ5Mjk0NQ==,9599,simonw,2021-01-06T18:37:39Z,2021-01-06T18:37:39Z,OWNER,I think I'll call this `full_path` for consistency with Django.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1179#issuecomment-755489974,https://api.github.com/repos/simonw/datasette/issues/1179,755489974,MDEyOklzc3VlQ29tbWVudDc1NTQ4OTk3NA==,9599,simonw,2021-01-06T18:35:24Z,2021-01-06T18:35:24Z,OWNER,Django calls this ` HttpRequest.get_full_path()` - for the path plus the querystring.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1179#issuecomment-755486103,https://api.github.com/repos/simonw/datasette/issues/1179,755486103,MDEyOklzc3VlQ29tbWVudDc1NTQ4NjEwMw==,9599,simonw,2021-01-06T18:32:41Z,2021-01-06T18:34:11Z,OWNER,"This parameter will return the URL path, with querystring arguments, to the HTML version of the page - e.g. `/github/issue_comments` or `/github/issue_comments?_sort_desc=created_at`

Open questions:

- What should it be called? `path` could be misleading since it also includes the querystring.
- Should I provide a `url` or `full_url` version which includes `https://blah.com/...`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/782#issuecomment-755484384,https://api.github.com/repos/simonw/datasette/issues/782,755484384,MDEyOklzc3VlQ29tbWVudDc1NTQ4NDM4NA==,9599,simonw,2021-01-06T18:31:14Z,2021-01-06T18:31:57Z,OWNER,"In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3:
```python
import pandas
df = pandas.read_json(
  ""https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array""
)
```
And:
```javascript
d3 = require(""d3@5"")
rows = d3.json(
    ""https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array""
)
```
Once again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both `pandas.read_json()` and `d3.json()` - but it leaves nowhere to put the extra information like pagination and suchlike!

Even given this I still think the correct default is an object with `""rows""`, `""total""` and `""next_url""` keys. I should commit to that and implement it - this thought exercise has been running for far too long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1178#issuecomment-755476820,https://api.github.com/repos/simonw/datasette/issues/1178,755476820,MDEyOklzc3VlQ29tbWVudDc1NTQ3NjgyMA==,9599,simonw,2021-01-06T18:24:47Z,2021-01-06T18:24:47Z,OWNER,"Issue fixed - https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on displays the correct schemes now.

I can't think of a reason anyone on Cloud Run would ever NOT want the `force_https_urls` option, but just in case I've made it so if you pass `--extra-options --setting force_https_urls off` to `publish cloudrun` your setting will be respected.

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/publish/cloudrun.py#L105-L110","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755468795,https://api.github.com/repos/simonw/datasette/issues/1178,755468795,MDEyOklzc3VlQ29tbWVudDc1NTQ2ODc5NQ==,9599,simonw,2021-01-06T18:14:35Z,2021-01-06T18:14:35Z,OWNER,Deploying that change now to test it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755163886,https://api.github.com/repos/simonw/datasette/issues/1178,755163886,MDEyOklzc3VlQ29tbWVudDc1NTE2Mzg4Ng==,9599,simonw,2021-01-06T08:37:51Z,2021-01-06T08:37:51Z,OWNER,"Easiest fix would be for `publish cloudrun` to set `force_https_urls`:

`datasette publish now` used to do this: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/publish/now.py#L59-L63","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1179#issuecomment-755161574,https://api.github.com/repos/simonw/datasette/issues/1179,755161574,MDEyOklzc3VlQ29tbWVudDc1NTE2MTU3NA==,9599,simonw,2021-01-06T08:32:31Z,2021-01-06T08:32:31Z,OWNER,An optional `path` argument to https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette which shows the path WITHOUT the `.Notebook` extension would be useful here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1178#issuecomment-755160187,https://api.github.com/repos/simonw/datasette/issues/1178,755160187,MDEyOklzc3VlQ29tbWVudDc1NTE2MDE4Nw==,9599,simonw,2021-01-06T08:29:35Z,2021-01-06T08:29:35Z,OWNER,"https://latest-with-plugins.datasette.io/-/asgi-scope

```
{'asgi': {'spec_version': '2.1', 'version': '3.0'},
 'client': ('169.254.8.129', 54971),
 'headers': [(b'host', b'latest-with-plugins.datasette.io'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko'
              b'/20100101 Firefox/84.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'dnt', b'1'),
             (b'cookie',
              b'_ga_LL6M7BK6D4=GS1.1.1609886546.49.1.1609886923.0; _ga=GA1.1'
              b'.894633707.1607575712'),
             (b'upgrade-insecure-requests', b'1'),
             (b'x-client-data', b'CgSL6ZsV'),
             (b'x-cloud-trace-context',
              b'e776af843c657d2a3da28a73b726e6fe/14187666787557102189;o=1'),
             (b'x-forwarded-for', b'148.64.98.14'),
             (b'x-forwarded-proto', b'https'),
             (b'forwarded', b'for=""148.64.98.14"";proto=https'),
             (b'accept-encoding', b'gzip, deflate, br'),
             (b'content-length', b'0')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope',
 'root_path': '',
 'scheme': 'http',
 'server': ('169.254.8.130', 8080),
 'type': 'http'}
```
Note the `'scheme': 'http'` but also the `(b'x-forwarded-proto', b'https')`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1176#issuecomment-755159583,https://api.github.com/repos/simonw/datasette/issues/1176,755159583,MDEyOklzc3VlQ29tbWVudDc1NTE1OTU4Mw==,9599,simonw,2021-01-06T08:28:20Z,2021-01-06T08:28:20Z,OWNER,I used `from datasette.utils import path_with_format` in https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py just now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1178#issuecomment-755158310,https://api.github.com/repos/simonw/datasette/issues/1178,755158310,MDEyOklzc3VlQ29tbWVudDc1NTE1ODMxMA==,9599,simonw,2021-01-06T08:25:31Z,2021-01-06T08:25:31Z,OWNER,Moving this to the Datasette repo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755157732,https://api.github.com/repos/simonw/datasette/issues/1178,755157732,MDEyOklzc3VlQ29tbWVudDc1NTE1NzczMg==,9599,simonw,2021-01-06T08:24:12Z,2021-01-06T08:24:12Z,OWNER,https://latest-with-plugins.datasette.io/fixtures/sortable.json has the bug too - the `next_url` is `http://` when it should be `https://`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755157281,https://api.github.com/repos/simonw/datasette/issues/1178,755157281,MDEyOklzc3VlQ29tbWVudDc1NTE1NzI4MQ==,9599,simonw,2021-01-06T08:23:14Z,2021-01-06T08:23:14Z,OWNER,"https://latest-with-plugins.datasette.io/-/settings says `""force_https_urls"": false`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755157066,https://api.github.com/repos/simonw/datasette/issues/1178,755157066,MDEyOklzc3VlQ29tbWVudDc1NTE1NzA2Ng==,9599,simonw,2021-01-06T08:22:47Z,2021-01-06T08:22:47Z,OWNER,"Weird... https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/app.py#L609-L613

```python
    def absolute_url(self, request, path):
        url = urllib.parse.urljoin(request.url, path)
        if url.startswith(""http://"") and self.setting(""force_https_urls""):
            url = ""https://"" + url[len(""http://"") :]
        return url
```
That looks like it should work. Needs more digging.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755156606,https://api.github.com/repos/simonw/datasette/issues/1178,755156606,MDEyOklzc3VlQ29tbWVudDc1NTE1NjYwNg==,9599,simonw,2021-01-06T08:21:49Z,2021-01-06T08:21:49Z,OWNER,"https://github.com/simonw/datasette-export-notebook/blob/aec398eab4f34791d240d7bc47b6eec575b357be/datasette_export_notebook/__init__.py#L18-L23

Maybe this is a bug in `datasette.absolute_url`? Perhaps it doesn't take the scheme into account.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1101#issuecomment-755134771,https://api.github.com/repos/simonw/datasette/issues/1101,755134771,MDEyOklzc3VlQ29tbWVudDc1NTEzNDc3MQ==,9599,simonw,2021-01-06T07:28:01Z,2021-01-06T07:28:01Z,OWNER,"With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the `stream_rows()` method could output `[` first, then each row followed by a comma, then `]` after the very last row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1101#issuecomment-755133937,https://api.github.com/repos/simonw/datasette/issues/1101,755133937,MDEyOklzc3VlQ29tbWVudDc1NTEzMzkzNw==,9599,simonw,2021-01-06T07:25:48Z,2021-01-06T07:26:43Z,OWNER,"Idea: instead of returning a dictionary, `register_output_renderer` could return an object. The object could have the following properties:

- `.extension` - the extension to use
- `.can_render(...)` - says if it can render this
- `.can_stream(...)` - says if streaming is supported
- `async .stream_rows(rows_iterator, send)` - method that loops through all rows and uses `send` to send them to the response in the correct format

I can then deprecate the existing `dict` return type for 1.0.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1101#issuecomment-755128038,https://api.github.com/repos/simonw/datasette/issues/1101,755128038,MDEyOklzc3VlQ29tbWVudDc1NTEyODAzOA==,9599,simonw,2021-01-06T07:10:22Z,2021-01-06T07:10:22Z,OWNER,"Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:

    pandas.read_json(""https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on"", lines=True)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1171#issuecomment-754958998,https://api.github.com/repos/simonw/datasette/issues/1171,754958998,MDEyOklzc3VlQ29tbWVudDc1NDk1ODk5OA==,9599,simonw,2021-01-05T23:16:33Z,2021-01-05T23:16:33Z,OWNER,"That's really useful, thanks @rcoup ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/782#issuecomment-754958610,https://api.github.com/repos/simonw/datasette/issues/782,754958610,MDEyOklzc3VlQ29tbWVudDc1NDk1ODYxMA==,9599,simonw,2021-01-05T23:15:24Z,2021-01-05T23:15:24Z,OWNER,https://latest-with-plugins.datasette.io/fixtures/roadside_attraction_characteristics/1.json-preview returns a 500 error at the moment - a KeyError on 'filtered_table_rows_count'.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/576#issuecomment-754957658,https://api.github.com/repos/simonw/datasette/issues/576,754957658,MDEyOklzc3VlQ29tbWVudDc1NDk1NzY1OA==,9599,simonw,2021-01-05T23:12:50Z,2021-01-05T23:12:50Z,OWNER,See https://docs.datasette.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-754957563,https://api.github.com/repos/simonw/datasette/issues/576,754957563,MDEyOklzc3VlQ29tbWVudDc1NDk1NzU2Mw==,9599,simonw,2021-01-05T23:12:37Z,2021-01-05T23:12:37Z,OWNER,"I'm happy with how this has evolved, so I'm closing the issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/1176#issuecomment-754957378,https://api.github.com/repos/simonw/datasette/issues/1176,754957378,MDEyOklzc3VlQ29tbWVudDc1NDk1NzM3OA==,9599,simonw,2021-01-05T23:12:03Z,2021-01-05T23:12:03Z,OWNER,This needs to be done for Datasette 1.0. At the very least I need to ensure it's clear that `datasette.utils` is not part of the public API unless explicitly marked as such.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1176#issuecomment-754952146,https://api.github.com/repos/simonw/datasette/issues/1176,754952146,MDEyOklzc3VlQ29tbWVudDc1NDk1MjE0Ng==,9599,simonw,2021-01-05T22:57:26Z,2021-01-05T22:57:26Z,OWNER,Known public APIs might be worth adding type annotations to as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1176#issuecomment-754952040,https://api.github.com/repos/simonw/datasette/issues/1176,754952040,MDEyOklzc3VlQ29tbWVudDc1NDk1MjA0MA==,9599,simonw,2021-01-05T22:57:09Z,2021-01-05T22:57:09Z,OWNER,It might be neater to move all of the non-public functions into a separate module - `datasette.utils.internal` perhaps.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1176#issuecomment-754951786,https://api.github.com/repos/simonw/datasette/issues/1176,754951786,MDEyOklzc3VlQ29tbWVudDc1NDk1MTc4Ng==,9599,simonw,2021-01-05T22:56:27Z,2021-01-05T22:56:43Z,OWNER,"Idea: introduce a `@documented` decorator which marks specific functions as part of the public, documented API. The unit tests can then confirm that anything with that decorator is both documented and tested.
```python
@documented
def escape_css_string(s):
    return _css_re.sub(
        lambda m: ""\\"" + (f""{ord(m.group()):X}"".zfill(6)),
        s.replace(""\r\n"", ""\n""),
    )
```
Or maybe `@public`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1175#issuecomment-754696725,https://api.github.com/repos/simonw/datasette/issues/1175,754696725,MDEyOklzc3VlQ29tbWVudDc1NDY5NjcyNQ==,9599,simonw,2021-01-05T15:12:30Z,2021-01-05T15:12:30Z,OWNER,Some tips here: https://github.com/tiangolo/fastapi/issues/78,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779156520,Use structlog for logging,
https://github.com/simonw/datasette/issues/1173#issuecomment-754463845,https://api.github.com/repos/simonw/datasette/issues/1173,754463845,MDEyOklzc3VlQ29tbWVudDc1NDQ2Mzg0NQ==,9599,simonw,2021-01-05T07:41:43Z,2021-01-05T07:41:43Z,OWNER,"https://github.com/oleksis/pyinstaller-manylinux looks useful, via https://twitter.com/oleksis/status/1346341987876823040","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778682317,GitHub Actions workflow to build manylinux binary,
https://github.com/simonw/datasette/issues/1171#issuecomment-754296761,https://api.github.com/repos/simonw/datasette/issues/1171,754296761,MDEyOklzc3VlQ29tbWVudDc1NDI5Njc2MQ==,9599,simonw,2021-01-04T23:55:44Z,2021-01-04T23:55:44Z,OWNER,Bit uncomfortable that it looks like you need to include your Apple ID username and password in the CI configuration to do this. I'll use GitHub Secrets for this but I don't like it - I'll definitely setup a dedicated code signing account that's not my access-to-everything AppleID for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1171#issuecomment-754295380,https://api.github.com/repos/simonw/datasette/issues/1171,754295380,MDEyOklzc3VlQ29tbWVudDc1NDI5NTM4MA==,9599,simonw,2021-01-04T23:54:32Z,2021-01-04T23:54:32Z,OWNER,"https://github.com/search?l=YAML&q=gon+json&type=Code reveals some examples of people using `gon` in workflows.

These look useful:

* https://github.com/coherence/hub-server/blob/3b7e9c7c5bce9e244b14b854f1f89d66f53a5a39/.github/workflows/release_build.yml
* https://github.com/simoncozens/pilcrow/blob/5abc145e7fb9577086afe47b48fd730cb8195386/.github/workflows/buildapp.yaml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1171#issuecomment-754287882,https://api.github.com/repos/simonw/datasette/issues/1171,754287882,MDEyOklzc3VlQ29tbWVudDc1NDI4Nzg4Mg==,9599,simonw,2021-01-04T23:40:10Z,2021-01-04T23:42:32Z,OWNER,"This looks VERY useful: https://github.com/mitchellh/gon - "" Sign, notarize, and package macOS CLI tools and applications written in any language. Available as both a CLI and a Go library.""

And it installs like this:

    brew install mitchellh/gon/gon","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1171#issuecomment-754286783,https://api.github.com/repos/simonw/datasette/issues/1171,754286783,MDEyOklzc3VlQ29tbWVudDc1NDI4Njc4Mw==,9599,simonw,2021-01-04T23:38:18Z,2021-01-04T23:38:18Z,OWNER,Oh wow maybe I need to Notarize it too? https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1171#issuecomment-754286618,https://api.github.com/repos/simonw/datasette/issues/1171,754286618,MDEyOklzc3VlQ29tbWVudDc1NDI4NjYxOA==,9599,simonw,2021-01-04T23:37:45Z,2021-01-04T23:37:45Z,OWNER,https://github.com/actions/virtual-environments/issues/1820#issuecomment-719549887 looks useful - not sure if those notes are for iOS or macOS though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/93#issuecomment-754285795,https://api.github.com/repos/simonw/datasette/issues/93,754285795,MDEyOklzc3VlQ29tbWVudDc1NDI4NTc5NQ==,9599,simonw,2021-01-04T23:35:13Z,2021-01-04T23:35:13Z,OWNER,Next step is to automate this all!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/1152#issuecomment-754285588,https://api.github.com/repos/simonw/datasette/issues/1152,754285588,MDEyOklzc3VlQ29tbWVudDc1NDI4NTU4OA==,9599,simonw,2021-01-04T23:34:30Z,2021-01-04T23:34:30Z,OWNER,"I think the way to do this is to have a new plugin hook that returns two SQL where clauses: one returning a list of resources that the user should be able to access (the allow-list) and one returning a list of resources they are explicitly forbidden from accessing (the deny-list). Either of these can be blank.

Datasette can then combine those into a full SQL query and use it to answer the question ""show me a list of resources that the user is allowed to perform action X on"". It can also answer the existing question, ""is user X allowed to perform action Y on resource Z""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/93#issuecomment-754233960,https://api.github.com/repos/simonw/datasette/issues/93,754233960,MDEyOklzc3VlQ29tbWVudDc1NDIzMzk2MA==,9599,simonw,2021-01-04T21:35:37Z,2021-01-04T21:35:37Z,OWNER,"I tested it by running a `tmate` session on the GitHub macOS machines, and it worked!
```
Mac-1609795972770:tmp runner$ wget 'https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip'
--2021-01-04 21:34:10--  https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream [following]
--2021-01-04 21:34:14--  https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.43.164
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.43.164|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8297283 (7.9M) [application/octet-stream]
Saving to: ‘datasette-0.53-macos-binary.zip’

datasette-0.53-maco 100%[===================>]   7.91M  --.-KB/s    in 0.1s    

2021-01-04 21:34:14 (73.4 MB/s) - ‘datasette-0.53-macos-binary.zip’ saved [8297283/8297283]

Mac-1609795972770:tmp runner$ unzip datasette-0.53-macos-binary.zip 
Archive:  datasette-0.53-macos-binary.zip
   creating: datasette-0.53-macos-binary/
  inflating: datasette-0.53-macos-binary/datasette  
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet along...
  uninstall  Uninstall Python packages (e.g.
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --get /-/versions.json
{""python"": {""version"": ""3.9.1"", ""full"": ""3.9.1 (default, Dec 10 2020, 10:36:35) \n[Clang 12.0.0 (clang-1200.0.32.27)]""}, ""datasette"": {""version"": ""0.53""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.13.3"", ""sqlite"": {""version"": ""3.34.0"", ""fts_versions"": [""FTS5"", ""FTS4"", ""FTS3""], ""extensions"": {""json1"": null}, ""compile_options"": [""COMPILER=clang-12.0.0"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_FTS3"", ""ENABLE_FTS3_PARENTHESIS"", ""ENABLE_FTS4"", ""ENABLE_FTS5"", ""ENABLE_GEOPOLY"", ""ENABLE_JSON1"", ""ENABLE_PREUPDATE_HOOK"", ""ENABLE_RTREE"", ""ENABLE_SESSION"", ""MAX_VARIABLE_NUMBER=250000"", ""THREADSAFE=1""]}}
Mac-1609795972770:tmp runner$ 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754229977,https://api.github.com/repos/simonw/datasette/issues/93,754229977,MDEyOklzc3VlQ29tbWVudDc1NDIyOTk3Nw==,9599,simonw,2021-01-04T21:28:01Z,2021-01-04T21:28:01Z,OWNER,"As an experiment, I put the macOS one in a zip file and attached it to the latest release:

```
mkdir datasette-0.53-macos-binary
cp dist/datasette datasette-0.53-macos-binary
zip -r datasette-0.53-macos-binary.zip datasette-0.53-macos-binary
```

It's available here: https://github.com/simonw/datasette/releases/tag/0.53 - download URL is https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754227543,https://api.github.com/repos/simonw/datasette/issues/93,754227543,MDEyOklzc3VlQ29tbWVudDc1NDIyNzU0Mw==,9599,simonw,2021-01-04T21:23:13Z,2021-01-04T21:23:13Z,OWNER,"```
(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# dist/datasette --get /-/databases.json
[{""name"": "":memory:"", ""path"": null, ""size"": 0, ""is_mutable"": true, ""is_memory"": true, ""hash"": null}]
(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# ls -lah dist/datasette 
-rwxr-xr-x 1 root root 8.9M Jan  4 21:05 dist/datasette
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754219002,https://api.github.com/repos/simonw/datasette/issues/93,754219002,MDEyOklzc3VlQ29tbWVudDc1NDIxOTAwMg==,9599,simonw,2021-01-04T21:06:49Z,2021-01-04T21:22:27Z,OWNER,"Works on Linux/Ubuntu too, except I had to do `export BASE=` on a separate line. I also did this:
```
apt-get install python3 python3-venv
python3 -m venv pyinstaller-venv
source pyinstaller-venv/bin/activate
pip install wheel
pip install datasette pyinstaller

export DATASETTE_BASE=$(python -c 'import os; print(os.path.dirname(__import__(""datasette"").__file__))')

pyinstaller -F \
    --add-data ""$DATASETTE_BASE/templates:datasette/templates"" \
    --add-data ""$DATASETTE_BASE/static:datasette/static"" \
    --hidden-import datasette.publish \
    --hidden-import datasette.publish.heroku \
    --hidden-import datasette.publish.cloudrun \
    --hidden-import datasette.facets \
    --hidden-import datasette.sql_functions \
    --hidden-import datasette.actor_auth_cookie \
    --hidden-import datasette.default_permissions \
    --hidden-import datasette.default_magic_parameters \
    --hidden-import datasette.blob_renderer \
    --hidden-import datasette.default_menu_links \
    --hidden-import uvicorn \
    --hidden-import uvicorn.logging \
    --hidden-import uvicorn.loops \
    --hidden-import uvicorn.loops.auto \
    --hidden-import uvicorn.protocols \
    --hidden-import uvicorn.protocols.http \
    --hidden-import uvicorn.protocols.http.auto \
    --hidden-import uvicorn.protocols.websockets \
    --hidden-import uvicorn.protocols.websockets.auto \
    --hidden-import uvicorn.lifespan \
    --hidden-import uvicorn.lifespan.on \
    $(which datasette)
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754218545,https://api.github.com/repos/simonw/datasette/issues/93,754218545,MDEyOklzc3VlQ29tbWVudDc1NDIxODU0NQ==,9599,simonw,2021-01-04T21:05:57Z,2021-01-04T21:05:57Z,OWNER,That BASE= trick seems to work with `zsh` but not with `bash`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754215392,https://api.github.com/repos/simonw/datasette/issues/93,754215392,MDEyOklzc3VlQ29tbWVudDc1NDIxNTM5Mg==,9599,simonw,2021-01-04T20:59:20Z,2021-01-04T21:03:14Z,OWNER,"Updated `pyinstaller` recipe - lots of hidden imports needed now:
```
pip install wheel
pip install datasette pyinstaller

BASE=$(python -c 'import os; print(os.path.dirname(__import__(""datasette"").__file__))') \
    pyinstaller -F \
        --add-data ""$BASE/templates:datasette/templates"" \
        --add-data ""$BASE/static:datasette/static"" \
        --hidden-import datasette.publish \
        --hidden-import datasette.publish.heroku \
        --hidden-import datasette.publish.cloudrun \
        --hidden-import datasette.facets \
        --hidden-import datasette.sql_functions \
        --hidden-import datasette.actor_auth_cookie \
        --hidden-import datasette.default_permissions \
        --hidden-import datasette.default_magic_parameters \
        --hidden-import datasette.blob_renderer \
        --hidden-import datasette.default_menu_links \
        --hidden-import uvicorn \
        --hidden-import uvicorn.logging \
        --hidden-import uvicorn.loops \
        --hidden-import uvicorn.loops.auto \
        --hidden-import uvicorn.protocols \
        --hidden-import uvicorn.protocols.http \
        --hidden-import uvicorn.protocols.http.auto \
        --hidden-import uvicorn.protocols.websockets \
        --hidden-import uvicorn.protocols.websockets.auto \
        --hidden-import uvicorn.lifespan \
        --hidden-import uvicorn.lifespan.on \
        $(which datasette)
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754215793,https://api.github.com/repos/simonw/datasette/issues/93,754215793,MDEyOklzc3VlQ29tbWVudDc1NDIxNTc5Mw==,9599,simonw,2021-01-04T21:00:14Z,2021-01-04T21:00:14Z,OWNER,"```
(pyinstaller-datasette) pyinstaller-datasette % file dist/datasette
dist/datasette: Mach-O 64-bit executable x86_64
(pyinstaller-datasette) pyinstaller-datasette % ls -lah dist/datasette
-rwxr-xr-x  1 simon  wheel   8.0M Jan  4 12:58 dist/datasette
```
I'm surprised it's only 8MB!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/668#issuecomment-754194996,https://api.github.com/repos/simonw/datasette/issues/668,754194996,MDEyOklzc3VlQ29tbWVudDc1NDE5NDk5Ng==,9599,simonw,2021-01-04T20:18:39Z,2021-01-04T20:18:39Z,OWNER,I fixed this in #1115 - you can run `--load-extension=spatialite` now and it will look for the extension in common places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",563347679,Make it easier to load SpatiaLite,
https://github.com/simonw/datasette/issues/436#issuecomment-754193501,https://api.github.com/repos/simonw/datasette/issues/436,754193501,MDEyOklzc3VlQ29tbWVudDc1NDE5MzUwMQ==,9599,simonw,2021-01-04T20:15:41Z,2021-01-04T20:15:41Z,OWNER,"Sadly `publish.datasettes.com` was broken by changes to Zeit, and I don't think I'll be bringing it back.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",435819321,400 Error when trying to register new user via https://publish.datasettes.com/,
https://github.com/simonw/datasette/issues/371#issuecomment-754192873,https://api.github.com/repos/simonw/datasette/issues/371,754192873,MDEyOklzc3VlQ29tbWVudDc1NDE5Mjg3Mw==,9599,simonw,2021-01-04T20:14:28Z,2021-01-04T20:14:28Z,OWNER,"Now that Digital Ocean has App Platform this is less necessary, especially since the documentation covers how to use App Platform here: https://docs.datasette.io/en/stable/deploying.html#deploying-using-buildpacks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",377156339,datasette publish digitalocean plugin,
https://github.com/simonw/datasette/issues/102#issuecomment-754192267,https://api.github.com/repos/simonw/datasette/issues/102,754192267,MDEyOklzc3VlQ29tbWVudDc1NDE5MjI2Nw==,9599,simonw,2021-01-04T20:13:19Z,2021-01-04T20:13:19Z,OWNER,"I'm more likely to do Lambda than Elastic Beanstalk, especially now the size limit for Lambdas has been increased as part of their support for Docker.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274264175,datasette publish elasticbeanstalk,
https://github.com/simonw/datasette/issues/221#issuecomment-754191699,https://api.github.com/repos/simonw/datasette/issues/221,754191699,MDEyOklzc3VlQ29tbWVudDc1NDE5MTY5OQ==,9599,simonw,2021-01-04T20:12:14Z,2021-01-04T20:12:14Z,OWNER,I'm going to close this. Plugins can register their own CLI tools (see https://github.com/simonw/click-app) if they need to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",315142414,Allow plugins to add new cli sub commands ,
https://github.com/simonw/datasette/issues/221#issuecomment-754190952,https://api.github.com/repos/simonw/datasette/issues/221,754190952,MDEyOklzc3VlQ29tbWVudDc1NDE5MDk1Mg==,9599,simonw,2021-01-04T20:10:51Z,2021-01-04T20:10:51Z,OWNER,Is this still a good idea? I don't have any pressing need for it at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",315142414,Allow plugins to add new cli sub commands ,
https://github.com/simonw/datasette/issues/221#issuecomment-754190814,https://api.github.com/repos/simonw/datasette/issues/221,754190814,MDEyOklzc3VlQ29tbWVudDc1NDE5MDgxNA==,9599,simonw,2021-01-04T20:10:34Z,2021-01-04T20:10:34Z,OWNER,"For the `csvs-to-sqlite` case I'm going with `datasette insert` instead, see #1160.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",315142414,Allow plugins to add new cli sub commands ,
https://github.com/simonw/datasette/issues/18#issuecomment-754188383,https://api.github.com/repos/simonw/datasette/issues/18,754188383,MDEyOklzc3VlQ29tbWVudDc1NDE4ODM4Mw==,9599,simonw,2021-01-04T20:05:48Z,2021-01-04T20:05:48Z,OWNER,"I'm not using Sanic any more, but this is still very feasible. If I ever do it I'll write a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267739593,See if I can get a websockets interface working,
https://github.com/simonw/datasette/issues/103#issuecomment-754188099,https://api.github.com/repos/simonw/datasette/issues/103,754188099,MDEyOklzc3VlQ29tbWVudDc1NDE4ODA5OQ==,9599,simonw,2021-01-04T20:05:14Z,2021-01-04T20:05:14Z,OWNER,"Wontfix, Cloud Run is already implemented and is a better fit for Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274265878,datasette publish appengine,
https://github.com/simonw/datasette/issues/913#issuecomment-754187520,https://api.github.com/repos/simonw/datasette/issues/913,754187520,MDEyOklzc3VlQ29tbWVudDc1NDE4NzUyMA==,9599,simonw,2021-01-04T20:04:10Z,2021-01-04T20:04:10Z,OWNER,That's pretty elegant: each plugin gets its own namespace and can register new settings.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/913#issuecomment-754187326,https://api.github.com/repos/simonw/datasette/issues/913,754187326,MDEyOklzc3VlQ29tbWVudDc1NDE4NzMyNg==,9599,simonw,2021-01-04T20:03:50Z,2021-01-04T20:03:50Z,OWNER,"I renamed `--config` to `--setting` and changed it to work like this:

    datasette --setting sql_time_limit_ms 1000

Note the lack of colons.

This actually makes colons cleaner to use for plugins - I could support this:

    datasette --setting datasette-insert:unsafe 1","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/1111#issuecomment-754184287,https://api.github.com/repos/simonw/datasette/issues/1111,754184287,MDEyOklzc3VlQ29tbWVudDc1NDE4NDI4Nw==,9599,simonw,2021-01-04T19:57:53Z,2021-01-04T19:57:53Z,OWNER,Relevant new feature in sqlite-utils: the ability to use triggers to maintain fast counts. This optimization could help a lot here. https://github.com/simonw/sqlite-utils/issues/212,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",751195017,Accessing a database's `.json` is slow for very large SQLite files,
https://github.com/simonw/datasette/issues/1164#issuecomment-754182058,https://api.github.com/repos/simonw/datasette/issues/1164,754182058,MDEyOklzc3VlQ29tbWVudDc1NDE4MjA1OA==,9599,simonw,2021-01-04T19:53:31Z,2021-01-04T19:53:31Z,OWNER,This will be helped by the new `package.json` added in #1170.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/pull/1170#issuecomment-754181646,https://api.github.com/repos/simonw/datasette/issues/1170,754181646,MDEyOklzc3VlQ29tbWVudDc1NDE4MTY0Ng==,9599,simonw,2021-01-04T19:52:40Z,2021-01-04T19:52:40Z,OWNER,Thank you very much!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778126516,Install Prettier via package.json,
https://github.com/simonw/datasette/issues/983#issuecomment-753690280,https://api.github.com/repos/simonw/datasette/issues/983,753690280,MDEyOklzc3VlQ29tbWVudDc1MzY5MDI4MA==,9599,simonw,2021-01-03T23:13:30Z,2021-01-03T23:13:30Z,OWNER,"Oh that's interesting, I hadn't thought about plugins firing events - just responding to events fired by the rest of the application.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671902,https://api.github.com/repos/simonw/sqlite-utils/issues/219,753671902,MDEyOklzc3VlQ29tbWVudDc1MzY3MTkwMg==,9599,simonw,2021-01-03T20:31:04Z,2021-01-03T20:32:13Z,OWNER,A `table.has_count_triggers` property.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777707544,reset_counts() method and command,
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671235,https://api.github.com/repos/simonw/sqlite-utils/issues/219,753671235,MDEyOklzc3VlQ29tbWVudDc1MzY3MTIzNQ==,9599,simonw,2021-01-03T20:25:10Z,2021-01-03T20:25:10Z,OWNER,"To detect tables, look at the names of the triggers - `{table}{counts_table}_insert` and `{table}{counts_table}_delete`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777707544,reset_counts() method and command,
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671009,https://api.github.com/repos/simonw/sqlite-utils/issues/219,753671009,MDEyOklzc3VlQ29tbWVudDc1MzY3MTAwOQ==,9599,simonw,2021-01-03T20:22:53Z,2021-01-03T20:22:53Z,OWNER,I think this should be accompanied by a `sqlite-utils reset-counts` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777707544,reset_counts() method and command,
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753670833,https://api.github.com/repos/simonw/sqlite-utils/issues/219,753670833,MDEyOklzc3VlQ29tbWVudDc1MzY3MDgzMw==,9599,simonw,2021-01-03T20:20:54Z,2021-01-03T20:20:54Z,OWNER,"This is a little tricky. We should assume that the existing values in the `_counts` table cannot be trusted at all when this method is called - so we should probably clear that table entirely and then re-populate it.

But that means we need to figure out which tables in the database have the counts triggers defined.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777707544,reset_counts() method and command,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753668099,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753668099,MDEyOklzc3VlQ29tbWVudDc1MzY2ODA5OQ==,9599,simonw,2021-01-03T19:55:53Z,2021-01-03T19:55:53Z,OWNER,So if you instantiate the `Database()` constructor with `use_counts_table=True` any access to the `.count` properties will go through this table - otherwise regular `count(*)` queries will be executed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753665521,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753665521,MDEyOklzc3VlQ29tbWVudDc1MzY2NTUyMQ==,9599,simonw,2021-01-03T19:31:33Z,2021-01-03T19:31:33Z,OWNER,"I'm having second thoughts about this being the default behaviour. It's pretty weird. I feel like HUGE databases that need this are rare, so having it on by default doesn't make sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753662490,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753662490,MDEyOklzc3VlQ29tbWVudDc1MzY2MjQ5MA==,9599,simonw,2021-01-03T19:05:53Z,2021-01-03T19:05:53Z,OWNER,Idea: a `.execute_count()` method that never uses the cache.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753661292,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753661292,MDEyOklzc3VlQ29tbWVudDc1MzY2MTI5Mg==,9599,simonw,2021-01-03T18:56:06Z,2021-01-03T18:56:23Z,OWNER,"Another option: on creation of the `Database()` object, check to see if the `_counts` table exists and use that as the default for a `use_counts_table` property. Also flip that property to `True` if the user calls `.enable_counts()` at any time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753661158,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753661158,MDEyOklzc3VlQ29tbWVudDc1MzY2MTE1OA==,9599,simonw,2021-01-03T18:55:16Z,2021-01-03T18:55:16Z,OWNER,"Alternative implementation: provided `db.should_trust_counts` is `True`, try running the query:
```sql
select count from _counts where [table] = ?
```
If the query fails to return a result OR throws an error because the table doesn't exist, run the `count(*)` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753660814,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753660814,MDEyOklzc3VlQ29tbWVudDc1MzY2MDgxNA==,9599,simonw,2021-01-03T18:53:05Z,2021-01-03T18:53:05Z,OWNER,"Here's the current `.count` property: https://github.com/simonw/sqlite-utils/blob/036ec6d32313487527c66dea613a3e7118b97459/sqlite_utils/db.py#L597-L609

It's implemented on `Queryable` which means it's available on both `Table` and `View` - the optimization doesn't make sense for views.

I'm a bit cautious about making that property so much more complex. In order to decide if it should try the `_counts` table first it needs to know:

- Should it be trusting the counts? I'm thinking a `.should_trust_counts` property on `Database` which defaults to `True` would be good - then advanced users can turn that off if they know the counts should not be trusted.
- Does the `_counts` table exist?
- Are the triggers defined?

Then it can do the query, and if the query fails it can fall back on the `count(*)`. That's quite a lot of extra activity though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753660379,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753660379,MDEyOklzc3VlQ29tbWVudDc1MzY2MDM3OQ==,9599,simonw,2021-01-03T18:50:15Z,2021-01-03T18:50:15Z,OWNER,"```python
    def cached_counts(self, tables=None):
        sql = ""select [table], count from {}"".format(self._counts_table_name)
        if tables:
            sql += "" where [table] in ({})"".format("", "".join(""?"" for table in tables))
        return {r[0]: r[1] for r in self.execute(sql, tables).fetchall()}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/206#issuecomment-753659260,https://api.github.com/repos/simonw/sqlite-utils/issues/206,753659260,MDEyOklzc3VlQ29tbWVudDc1MzY1OTI2MA==,9599,simonw,2021-01-03T18:42:01Z,2021-01-03T18:42:01Z,OWNER,"```
% sqlite-utils insert blah.db blah global_power_plant_database.csv
Error: Invalid JSON - use --csv for CSV or --tsv for TSV files
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",761915790,sqlite-utils should suggest --csv if JSON parsing fails,
https://github.com/simonw/datasette/issues/1169#issuecomment-753657180,https://api.github.com/repos/simonw/datasette/issues/1169,753657180,MDEyOklzc3VlQ29tbWVudDc1MzY1NzE4MA==,9599,simonw,2021-01-03T18:23:30Z,2021-01-03T18:23:30Z,OWNER,"Also welcome in that PR would be a bit of documentation for contributors, see #1167 - but no problem if you leave that out, I'm happy to add it later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777677671,Prettier package not actually being cached,
https://github.com/simonw/datasette/issues/1169#issuecomment-753653260,https://api.github.com/repos/simonw/datasette/issues/1169,753653260,MDEyOklzc3VlQ29tbWVudDc1MzY1MzI2MA==,9599,simonw,2021-01-03T17:54:40Z,2021-01-03T17:54:40Z,OWNER,And @benpickles yes I would land that pull request straight away as-is. Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777677671,Prettier package not actually being cached,
https://github.com/simonw/datasette/issues/1169#issuecomment-753653033,https://api.github.com/repos/simonw/datasette/issues/1169,753653033,MDEyOklzc3VlQ29tbWVudDc1MzY1MzAzMw==,9599,simonw,2021-01-03T17:52:53Z,2021-01-03T17:52:53Z,OWNER,"Oh that's so frustrating! I was worried about that - I spotted a few runs that seemed faster and hoped that it meant that the package was coming out of the `~/.npm` cache, but evidently that's not the case.

You've convinced me that Datasette itself should have a `package.json` - the Dependabot argument is a really good one.

But... I'd really love to figure out a general pattern for using `npx` scripts in GitHub Actions workflows in a cache-friendly way. I have plenty of other projects that I'd love to run Prettier or Uglify or `puppeteer-cli` in without adding a `package.json` to them.

Any ideas? The best I can think of is for the workflow itself to write out a `package.json` file (using `echo '{ ... }' > package.json`) as part of the run - that way the cache should work (I think) but I don't get a misleading `package.json` file sitting in the repo.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777677671,Prettier package not actually being cached,
https://github.com/simonw/datasette/issues/983#issuecomment-753570710,https://api.github.com/repos/simonw/datasette/issues/983,753570710,MDEyOklzc3VlQ29tbWVudDc1MzU3MDcxMA==,9599,simonw,2021-01-03T05:29:56Z,2021-01-03T05:29:56Z,OWNER,"I thought about using browser events, but they don't quite match the API that I'm looking to provide. In particular, the great thing about Pluggy is that if you have multiple handlers registered for a specific plugin hook each of those handlers can return a value, and Pluggy will combine those values into a list of replies.

This is great for things like plugin hooks that add extra menu items - each plugin can return a menu item (maybe as a label/URL/click-callback object) and the calling code can then add all of those items to the menu. See https://docs.datasette.io/en/stable/plugin_hooks.html#table-actions-datasette-actor-database-table for a Python example.

I'm on the fence about relying on JavaScript modules. I need to think about browser compatibility for them - but I'm already commited to requiring support for `() => {}` arrow functions so maybe I'm committed to module support too already?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1160#issuecomment-753568428,https://api.github.com/repos/simonw/datasette/issues/1160,753568428,MDEyOklzc3VlQ29tbWVudDc1MzU2ODQyOA==,9599,simonw,2021-01-03T05:02:32Z,2021-01-03T05:02:32Z,OWNER,"Should this command include a `--fts` option for configuring full-text search on one-or-more columns?

I thought about doing that for `sqlite-utils insert` in https://github.com/simonw/sqlite-utils/issues/202 and decided not to because of the need to include extra options covering the FTS version, porter stemming options and whether or not to create triggers.

But maybe I can set sensible defaults for that with `datasette insert ... -f title -f body`? Worth thinking about a bit more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/sqlite-utils/issues/202#issuecomment-753568264,https://api.github.com/repos/simonw/sqlite-utils/issues/202,753568264,MDEyOklzc3VlQ29tbWVudDc1MzU2ODI2NA==,9599,simonw,2021-01-03T05:00:24Z,2021-01-03T05:00:24Z,OWNER,"I'm not going to implement this, because it actually needs several additional options that already exist on `sqlite-utils enable-fts`:
```
  --fts4                 Use FTS4
  --fts5                 Use FTS5
  --tokenize TEXT        Tokenizer to use, e.g. porter
  --create-triggers      Create triggers to update the FTS tables when the
                         parent table changes.
```
I'd rather not add all four of those options to `sqlite-utils insert` just to support this shortcut.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",738514367,sqlite-utils insert -f colname - for configuring full-text search,
https://github.com/simonw/sqlite-utils/issues/202#issuecomment-753567969,https://api.github.com/repos/simonw/sqlite-utils/issues/202,753567969,MDEyOklzc3VlQ29tbWVudDc1MzU2Nzk2OQ==,9599,simonw,2021-01-03T04:55:17Z,2021-01-03T04:55:43Z,OWNER,"The long version of this can be `--fts`, same as in `csvs-to-sqlite`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",738514367,sqlite-utils insert -f colname - for configuring full-text search,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567932,MDEyOklzc3VlQ29tbWVudDc1MzU2NzkzMg==,9599,simonw,2021-01-03T04:54:43Z,2021-01-03T04:54:43Z,OWNER,"Another option: expand the `ForeignKey` object to have `.columns` and `.other_columns` properties in addition to the existing `.column` and `.other_column` properties. These new plural properties would always return a tuple, which would be a one-item tuple for a non-compound-foreign-key.

The question then is what should `.column` and `.other_column` return for compound foreign keys?

I'd be inclined to say they should return `None` - which would trigger errors in code that encounters a compound foreign key for the first time, but those errors would at least be a strong indicator as to what had gone wrong.

We can label `.column` and `.other_column` as deprecated and then remove them in `sqlite-utils 4.0`.

Since this would still be a breaking change in some minor edge-cases I'm thinking maybe 4.0 needs to happen in order to land this feature. I'm not opposed to doing that, I was just hoping it might be avoidable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567744,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567744,MDEyOklzc3VlQ29tbWVudDc1MzU2Nzc0NA==,9599,simonw,2021-01-03T04:51:44Z,2021-01-03T04:51:44Z,OWNER,"One way that this could avoid a breaking change would be to have `fk.column` and `fk.other_column` remain as strings for non-compound-foreign-keys, but turn into tuples for a compound foreign key.

This is a bit of an ugly API design, and it could still break existing code that encounters a compound foreign key for the first time - but it would leave code working for the more common case of a non-compound-foreign-key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567508,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567508,MDEyOklzc3VlQ29tbWVudDc1MzU2NzUwOA==,9599,simonw,2021-01-03T04:48:17Z,2021-01-03T04:48:17Z,OWNER,"Sorry for taking so long to review this!

This approach looks great to me - being able to optionally pass a tuple anywhere the API currently expects a column is smart, and it's consistent with how the `pk=` parameter works elsewhere.

There's just one problem I can see with this: the way it changes the `ForeignKey(...)` interface to always return a tuple for `.column` and `.other_column`, even if that tuple only contains a single item.

This represents a breaking change to the existing API - any code that expects `ForeignKey.column` to be a single string (which is any code that has been written against that) will break.

As such, I'd have to bump the major version of `sqlite-utils` to `4.0` in order to ship this.

Ideally I'd like to make this change in a way that doesn't represent an API compatibility break. I need to think a bit harder about how that might be achieved.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/issues/217#issuecomment-753566184,https://api.github.com/repos/simonw/sqlite-utils/issues/217,753566184,MDEyOklzc3VlQ29tbWVudDc1MzU2NjE4NA==,9599,simonw,2021-01-03T04:27:38Z,2021-01-03T04:27:38Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-strings-for-use-in-sql,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777543336,Rename .escape() to .quote(),
https://github.com/simonw/sqlite-utils/issues/216#issuecomment-753566156,https://api.github.com/repos/simonw/sqlite-utils/issues/216,753566156,MDEyOklzc3VlQ29tbWVudDc1MzU2NjE1Ng==,9599,simonw,2021-01-03T04:27:14Z,2021-01-03T04:27:14Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#introspection,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777540352,database.triggers_dict introspection property,
https://github.com/simonw/sqlite-utils/issues/218#issuecomment-753563757,https://api.github.com/repos/simonw/sqlite-utils/issues/218,753563757,MDEyOklzc3VlQ29tbWVudDc1MzU2Mzc1Nw==,9599,simonw,2021-01-03T03:49:51Z,2021-01-03T03:49:51Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-triggers,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777560474,"""sqlite-utils triggers"" command",
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545757,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753545757,MDEyOklzc3VlQ29tbWVudDc1MzU0NTc1Nw==,9599,simonw,2021-01-02T23:58:07Z,2021-01-02T23:58:07Z,OWNER,"Thought: maybe there should be a `.reset_counts()` method too, for if the table gets out of date with the triggers.

One way that could happen is if a table is dropped and recreated - the counts in the `_counts` table would likely no longer match the number of rows in that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545381,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753545381,MDEyOklzc3VlQ29tbWVudDc1MzU0NTM4MQ==,9599,simonw,2021-01-02T23:52:52Z,2021-01-02T23:52:52Z,OWNER,Idea: a `db.cached_counts()` method that returns a dictionary of data from the `_counts` table. Call it with a list of tables to get back the counts for just those tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/217#issuecomment-753544914,https://api.github.com/repos/simonw/sqlite-utils/issues/217,753544914,MDEyOklzc3VlQ29tbWVudDc1MzU0NDkxNA==,9599,simonw,2021-01-02T23:47:42Z,2021-01-02T23:47:42Z,OWNER,https://github.com/simonw/sqlite-utils/blob/9a5c92b63e7917c93cc502478493c51c781b2ecc/sqlite_utils/db.py#L231-L239,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777543336,Rename .escape() to .quote(),
https://github.com/simonw/sqlite-utils/issues/213#issuecomment-753535488,https://api.github.com/repos/simonw/sqlite-utils/issues/213,753535488,MDEyOklzc3VlQ29tbWVudDc1MzUzNTQ4OA==,9599,simonw,2021-01-02T22:03:48Z,2021-01-02T22:03:48Z,OWNER,"I got this error while prototyping this:

    too many levels of trigger recursion

It looks like that's because SQLite doesn't like triggers on a table that themselves then update that table - so I'm going to exclude the `_counts` table from this mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777529979,db.enable_counts() method,
https://github.com/simonw/sqlite-utils/issues/213#issuecomment-753533775,https://api.github.com/repos/simonw/sqlite-utils/issues/213,753533775,MDEyOklzc3VlQ29tbWVudDc1MzUzMzc3NQ==,9599,simonw,2021-01-02T21:47:10Z,2021-01-02T21:47:10Z,OWNER,"I'm going to skip virtual tables, which I can identify using this property: https://github.com/simonw/sqlite-utils/blob/1cad7fad3e7a5b734088f5cc545b69a055e636da/sqlite_utils/db.py#L720-L726","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777529979,db.enable_counts() method,
https://github.com/simonw/datasette/issues/1168#issuecomment-753524779,https://api.github.com/repos/simonw/datasette/issues/1168,753524779,MDEyOklzc3VlQ29tbWVudDc1MzUyNDc3OQ==,9599,simonw,2021-01-02T20:19:26Z,2021-01-02T20:19:26Z,OWNER,Idea: version the metadata scheme. If the table is called `_metadata_v1` it gives me a clear path to designing a new scheme in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/sqlite-utils/issues/212#issuecomment-753422324,https://api.github.com/repos/simonw/sqlite-utils/issues/212,753422324,MDEyOklzc3VlQ29tbWVudDc1MzQyMjMyNA==,9599,simonw,2021-01-02T03:00:34Z,2021-01-02T03:00:34Z,OWNER,"Here's a prototype:
```python
with db.conn:
    db.conn.executescript(""""""
CREATE TABLE IF NOT EXISTS [_counts] ([table] TEXT PRIMARY KEY, [count] INTEGER DEFAULT 0);
CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ai] AFTER INSERT ON [Street_Tree_List] BEGIN
    INSERT OR REPLACE INTO _counts
        VALUES ('Street_Tree_List', COALESCE(
            (SELECT count FROM _counts
                WHERE [table]='Street_Tree_List'),
            0) + 1);
END;
CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ad] AFTER DELETE ON [Street_Tree_List] BEGIN
    INSERT OR REPLACE INTO _counts
        VALUES ('Street_Tree_List', COALESCE(
            (SELECT count FROM _counts
                WHERE [table]='Street_Tree_List'),
            0) - 1);
END;
INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', (select count(*) from [Street_Tree_List]));
"""""")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777392020,Mechanism for maintaining cache of table counts using triggers,
https://github.com/simonw/sqlite-utils/issues/210#issuecomment-753406744,https://api.github.com/repos/simonw/sqlite-utils/issues/210,753406744,MDEyOklzc3VlQ29tbWVudDc1MzQwNjc0NA==,9599,simonw,2021-01-02T00:02:39Z,2021-01-02T00:02:39Z,OWNER,"It looks like https://github.com/ofajardo/pyreadr is a good library for this.

I won't add this to `sqlite-utils` because it's quite a bulky dependency for a relatively small feature.

Normally I'd write a `rdata-to-sqlite` tool similar to https://pypi.org/project/dbf-to-sqlite/ - but I'm actually working on a new plugin hook for Datasette that might be an even better fit for this. The idea is to allow Datasette plugins to define input formats - such as RData - which would then result in being able to import them on the command-line with `datasette insert my.db file.rdata` or by uploading a file through the Datasette web interface.

That work is happening over here: https://github.com/simonw/datasette/issues/1160 - I'll close this issue in favour of a sometime-in-the-future `datasette-import-rdata` plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",767685961,Support of RData files,
https://github.com/simonw/sqlite-utils/issues/209#issuecomment-753405835,https://api.github.com/repos/simonw/sqlite-utils/issues/209,753405835,MDEyOklzc3VlQ29tbWVudDc1MzQwNTgzNQ==,9599,simonw,2021-01-01T23:52:06Z,2021-01-01T23:52:06Z,OWNER,I just hit this one too. Such a weird bug!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",766156875,Test failure with sqlite 3.34 in test_cli.py::test_optimize,
https://github.com/simonw/datasette/issues/1168#issuecomment-753402423,https://api.github.com/repos/simonw/datasette/issues/1168,753402423,MDEyOklzc3VlQ29tbWVudDc1MzQwMjQyMw==,9599,simonw,2021-01-01T23:16:05Z,2021-01-01T23:16:05Z,OWNER,"One catch: solving the ""show me all metadata for everything in this Datasette instance"" problem.

Ideally there would be a SQLite table that can be queried for this. But the need to resolve the potentially complex set of precedence rules means that table would be difficult if not impossible to provide at run-time.

Ideally a denormalized table would be available that featured the results of running those precedence rule calculations. But how to handle keeping this up-to-date? It would need to be recalculated any time a `_metadata` table in any of the attached databases had an update.

This is a much larger problem - but one potential fix would be to use triggers to maintain a ""version number"" for the `_metadata` table - similar to SQLite's own built-in `schema_version` mechanism. Triggers could increment a counter any time a record in that table was added, deleted or updated.

Such a mechanism would have applications outside of just this `_metadata` system. The ability to attach a version number to any table and have it automatically incremented when that table changes (via triggers) could help with all kinds of other Datasette-at-scale problems, including things like cached table counts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753401001,https://api.github.com/repos/simonw/datasette/issues/1168,753401001,MDEyOklzc3VlQ29tbWVudDc1MzQwMTAwMQ==,9599,simonw,2021-01-01T23:01:45Z,2021-01-01T23:01:45Z,OWNER,I need to prototype this. Could I do that as a plugin? I think so - I could try out the algorithm for loading metadata and display it on pages using some custom templates.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753400420,https://api.github.com/repos/simonw/datasette/issues/1168,753400420,MDEyOklzc3VlQ29tbWVudDc1MzQwMDQyMA==,9599,simonw,2021-01-01T22:53:58Z,2021-01-01T22:53:58Z,OWNER,"Precedence idea:
- First priority is non-_internal metadata from other databases - if those conflict then pick then the alphabetically-ordered-first database name wins
- Next priority: `_internal` metadata, which should have been loaded from `metadata.json`
- Last priority: the `_metadata` table from that database itself, i.e. the default ""baked in"" metadata","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753400306,https://api.github.com/repos/simonw/datasette/issues/1168,753400306,MDEyOklzc3VlQ29tbWVudDc1MzQwMDMwNg==,9599,simonw,2021-01-01T22:52:44Z,2021-01-01T22:52:44Z,OWNER,"Also: probably load column metadata as part of the table metadata rather than loading column metadata individually, since it's going to be rare to want the metadata for a single column rather than for an entire table full of columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753400265,https://api.github.com/repos/simonw/datasette/issues/1168,753400265,MDEyOklzc3VlQ29tbWVudDc1MzQwMDI2NQ==,9599,simonw,2021-01-01T22:52:09Z,2021-01-01T22:52:09Z,OWNER,"From an implementation perspective, I think the way this works is SQL queries read the relevant metadata from ALL available metadata tables, then Python code solves the precedence rules to produce the final, combined metadata for a database/table/column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753399635,https://api.github.com/repos/simonw/datasette/issues/1168,753399635,MDEyOklzc3VlQ29tbWVudDc1MzM5OTYzNQ==,9599,simonw,2021-01-01T22:45:21Z,2021-01-01T22:50:21Z,OWNER,"Would also need to figure out the precedence rules:

- What happens if the database has a `_metadata` table with data that conflicts with a remote metadata record from another database? I think the other database should win, because that allows plugins to over-ride the default metadata for something.
- Do JSON values get merged together? So if one table provides a description and another provides a title do both values get returned?
- If a database has a `license`, does that ""cascade"" down to the tables? What about `source` and `about`?
- What if there are two databases (or more) that provide conflicting metadata for a table in some other database? Also, `_internal` may have loaded data from `metadata.json` that conflicts with some other remote table metadata definition.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753399428,https://api.github.com/repos/simonw/datasette/issues/1168,753399428,MDEyOklzc3VlQ29tbWVudDc1MzM5OTQyOA==,9599,simonw,2021-01-01T22:43:14Z,2021-01-01T22:43:22Z,OWNER,"Could this use a compound primary key on `database, table, column`? Does that work with null values?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753399366,https://api.github.com/repos/simonw/datasette/issues/1168,753399366,MDEyOklzc3VlQ29tbWVudDc1MzM5OTM2Ng==,9599,simonw,2021-01-01T22:42:37Z,2021-01-01T22:42:37Z,OWNER,"So what would the database schema for this look like?

I'm leaning towards a single table called `_metadata`, because that's a neater fit for baking the metadata into the database file along with the data that it is describing. Alternatively I could have multiple tables sharing that prefix - `_metadata_database` and `_metadata_tables` and `_metadata_columns` perhaps.

If it's just a single `_metadata` table, the schema could look like this:

| database | table | column | metadata |
| --- | --- | --- | --- |
|  | mytable |  | {""title"": ""My Table"" } |
|  | mytable | mycolumn | {""description"": ""Column description"" } |
| otherdb | othertable | | {""description"": ""Table in another DB"" } |

If the `database` column is `null` it means ""this is describing a table in the same database file as this `_metadata` table"".

The alternative to the `metadata` JSON column would be separate columns for each potential metadata value - `license`, `source`, `about`, `about_url` etc. But that makes it harder for people to create custom metadata fields.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753398542,https://api.github.com/repos/simonw/datasette/issues/1168,753398542,MDEyOklzc3VlQ29tbWVudDc1MzM5ODU0Mg==,9599,simonw,2021-01-01T22:37:24Z,2021-01-01T22:37:24Z,OWNER,"The direction I'm leaning in now is the following:

- Metadata always lives in SQLite tables
- These tables can be co-located with the database they describe (same DB file)
- ... or they can be in a different DB file and reference the other database that they are describing
- Metadata provided on startup in a `metadata.json` file is loaded into an in-memory metadata table using that same mechanism

Plugins that want to provide metadata can do so by populating a table. They could even maintain their own in-memory database for this, or they could write to the `_internal` in-memory database, or they could write to a table in a database on disk.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753392102,https://api.github.com/repos/simonw/datasette/issues/1168,753392102,MDEyOklzc3VlQ29tbWVudDc1MzM5MjEwMg==,9599,simonw,2021-01-01T22:06:33Z,2021-01-01T22:06:33Z,OWNER,"Some SQLite databases include SQL comments in the schema definition which tell you what each column means:

```sql
CREATE TABLE User
        -- A table comment
(
        uid INTEGER,    -- A field comment
        flags INTEGER   -- Another field comment
);
```
The problem with these is that they're not exposed to SQLite in any mechanism other than parsing the `CREATE TABLE` statement from the `sqlite_master` table to extract those columns.

I had an idea to build a plugin that could return these. That would be easy with a ""get metadata for this column"" plugin hook - in the absence of one a plugin could still run that reads the schemas on startup and uses them to populate a metadata database table somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753391869,https://api.github.com/repos/simonw/datasette/issues/1168,753391869,MDEyOklzc3VlQ29tbWVudDc1MzM5MTg2OQ==,9599,simonw,2021-01-01T22:04:30Z,2021-01-01T22:04:30Z,OWNER,"The sticking point here seems to be the plugin hook. Allowing plugins to over-ride the way the question ""give me the metadata for this database/table/column"" is answered makes the database-backed metadata mechanisms much more complicated to think about.

What if plugins didn't get to over-ride metadata in this way, but could instead update the metadata in a persistent Datasette-managed storage mechanism?

Then maybe Datasette could do the following:

- Maintain metadata in `_internal` that has been loaded from `metadata.json`
- Know how to check a database for baked-in metadata (maybe in a `_metadata` table)
- Know how to fall back on the `_internal` metadata if no baked-in metadata is available

If database files were optionally allowed to store metadata about tables that live in another database file this could perhaps solve the plugin needs - since an ""edit metadata"" plugin would be able to edit records in a separate, dedicated `metadata.db` database to store new information about tables in other files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753390791,https://api.github.com/repos/simonw/datasette/issues/1168,753390791,MDEyOklzc3VlQ29tbWVudDc1MzM5MDc5MQ==,9599,simonw,2021-01-01T22:00:42Z,2021-01-01T22:00:42Z,OWNER,"Here are the requirements I'm currently trying to satisfy:

- It should be possible to query the metadata for ALL attached tables in one place, potentially with pagination and filtering
- Metadata should be able to exist in the current `metadata.json` file
- It should also be possible to bundle metadata in a table in the SQLite database files themselves
- Plugins should be able to define their own special mechanisms for metadata. This is particularly interesting for providing a UI that allows users to edit the metadata for their existing tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753390262,https://api.github.com/repos/simonw/datasette/issues/1168,753390262,MDEyOklzc3VlQ29tbWVudDc1MzM5MDI2Mg==,9599,simonw,2021-01-01T21:58:11Z,2021-01-01T21:58:11Z,OWNER,"One possibility: plugins could write directly to that in-memory database table. But how would they know to write again should the server restart? Maybe they would write to it once when called by the `startup` plugin hook, and then update it (and their own backing store) when metadata changes for some reason. Feels a bit messy though.

Also: if I want to support metadata optionally living in a `_metadata` table colocated with the data in a SQLite database file itself, how would that affect the `metadata` columns in `_internal`? How often would Datasette denormalize and copy data across from the on-disk `_metadata` tables to the `_internal` in-memory columns?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753389938,https://api.github.com/repos/simonw/datasette/issues/1168,753389938,MDEyOklzc3VlQ29tbWVudDc1MzM4OTkzOA==,9599,simonw,2021-01-01T21:54:15Z,2021-01-01T21:54:15Z,OWNER,"So what if the `databases`, `tables` and `columns` tables in `_internal` each grew a new `metadata` text column?

These columns could be populated by Datasette on startup through reading the `metadata.json` file. But how would plugins interact with them?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753389477,https://api.github.com/repos/simonw/datasette/issues/1168,753389477,MDEyOklzc3VlQ29tbWVudDc1MzM4OTQ3Nw==,9599,simonw,2021-01-01T21:49:57Z,2021-01-01T21:49:57Z,OWNER,"What if metadata was stored in a JSON text column in the existing `_internal` tables? This would allow for users to invent additional metadata fields in the future beyond the current `license`, `license_url` etc fields - without needing a schema change.

The downside of JSON columns generally is that they're harder to run indexed queries against. For metadata I don't think that matters - even with 10,000 tables each with their own metadata a SQL query asking for e.g. ""everything that has Apache 2 as the license"" would return in just a few ms.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753388809,https://api.github.com/repos/simonw/datasette/issues/1168,753388809,MDEyOklzc3VlQ29tbWVudDc1MzM4ODgwOQ==,9599,simonw,2021-01-01T21:47:51Z,2021-01-01T21:47:51Z,OWNER,"A database that exposes metadata will have the same restriction as the new `_internal` database that exposes columns and tables, in that it needs to take permissions into account. A user should not be able to view metadata for tables that they are not able to see.

As such, I'd rather bundle any metadata tables into the existing `_internal` database so I don't have to solve that permissions problem in two places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753366024,https://api.github.com/repos/simonw/datasette/issues/1168,753366024,MDEyOklzc3VlQ29tbWVudDc1MzM2NjAyNA==,9599,simonw,2021-01-01T18:48:34Z,2021-01-01T18:48:34Z,OWNER,Also: in #188 I proposed bundling metadata in the SQLite database itself alongside the data. This is a great way of ensuring metadata travels with the data when it is downloaded as a SQLite `.db` file. But how would that play with the idea of an in-memory `_metadata` table? Could that table perhaps offer views that join data across multiple attached physical databases?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1166#issuecomment-753224351,https://api.github.com/repos/simonw/datasette/issues/1166,753224351,MDEyOklzc3VlQ29tbWVudDc1MzIyNDM1MQ==,9599,simonw,2020-12-31T23:23:29Z,2020-12-31T23:23:29Z,OWNER,I should configure the action to only run if changes have been made within the `datasette/static` directory.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/983#issuecomment-753221646,https://api.github.com/repos/simonw/datasette/issues/983,753221646,MDEyOklzc3VlQ29tbWVudDc1MzIyMTY0Ng==,9599,simonw,2020-12-31T22:58:47Z,2020-12-31T22:58:47Z,OWNER,"https://github.com/mishoo/UglifyJS/issues/1905#issuecomment-300485490 says:

> `sourceMappingURL` aren't added by default in `3.x` due to one of the feature requests not to - some users are putting them within HTTP response headers instead.
> 
> So the command line for that would be:
> 
> ```js
> $ uglifyjs main.js -cmo main.min.js --source-map url=main.min.js.map
> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1164#issuecomment-753221362,https://api.github.com/repos/simonw/datasette/issues/1164,753221362,MDEyOklzc3VlQ29tbWVudDc1MzIyMTM2Mg==,9599,simonw,2020-12-31T22:55:57Z,2020-12-31T22:55:57Z,OWNER,"I had to add this as the first line in `table.min.js` for the source mapping to work:
```
//# sourceMappingURL=/-/static/table.min.js.map
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-753220665,https://api.github.com/repos/simonw/datasette/issues/1164,753220665,MDEyOklzc3VlQ29tbWVudDc1MzIyMDY2NQ==,9599,simonw,2020-12-31T22:49:36Z,2020-12-31T22:49:36Z,OWNER,"I started with a 7K `table.js` file.

`npx uglifyjs table.js --source-map -o table.min.js` gave me a 5.6K `table.min.js` file. 

`npx uglifyjs table.js --source-map -o table.min.js --compress --mangle` gave me 4.5K.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-753220412,https://api.github.com/repos/simonw/datasette/issues/1164,753220412,MDEyOklzc3VlQ29tbWVudDc1MzIyMDQxMg==,9599,simonw,2020-12-31T22:47:36Z,2020-12-31T22:47:36Z,OWNER,"I'm trying to minify `table.js` and I ran into a problem:

    Uglification failed. Unexpected character '`'

It turns out `uglify-js` doesn't support ES6 syntax!

But `uglify-es` does:

    npm install uglify-es

Annoyingly it looks like `uglify-es` uses the same CLI command, `uglifyjs`. So after installing it this seemed to work:

    npx uglifyjs table.js --source-map -o table.min.js

I really don't like how `npx uglifyjs` could mean different things depending on which package was installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/983#issuecomment-753219521,https://api.github.com/repos/simonw/datasette/issues/983,753219521,MDEyOklzc3VlQ29tbWVudDc1MzIxOTUyMQ==,9599,simonw,2020-12-31T22:39:52Z,2020-12-31T22:39:52Z,OWNER,For inlining the `plugins.min.js` file into the Jinja templates I could use the trick described here: https://stackoverflow.com/a/41404611 - which adds a `{{ include_file('file.txt') }}` function to Jinja.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753219407,https://api.github.com/repos/simonw/datasette/issues/983,753219407,MDEyOklzc3VlQ29tbWVudDc1MzIxOTQwNw==,9599,simonw,2020-12-31T22:38:45Z,2020-12-31T22:39:10Z,OWNER,"You'll be able to add JavaScript plugins using a bunch of different mechanisms:

- In a custom template, dropping the code in to a `<script>` block
- A bookmarklet that injects an extra script (I'm really excited to try this out)
- A separate `script.js` file that's loaded into Datasette using the `""extra_js_urls""` metadata option, documented here: https://docs.datasette.io/en/stable/custom_templates.html#custom-css-and-javascript
- A plugin you can install, like `datasette-vega` or `datasette-cluster-map` - since plugins can bundle their own script files that then get loaded on pages via this hook: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette
","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753217917,https://api.github.com/repos/simonw/datasette/issues/983,753217917,MDEyOklzc3VlQ29tbWVudDc1MzIxNzkxNw==,9599,simonw,2020-12-31T22:23:29Z,2020-12-31T22:23:36Z,OWNER,"If I'm going to do that, it would be good if subsequent plugins that register against the `load` event are executed straight away. That's a bit of a weird edge-case in plugin world - it would involve the bulkier code that gets loaded redefining how `datasette.plugins.register` works to special-case the `'load'` hook.

Maybe the tiny bootstrap code could define a `datasette.plugins.onload(callbackFunction)` method which gets upgraded later into something that fires straight away? Would add more bytes though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753217714,https://api.github.com/repos/simonw/datasette/issues/983,753217714,MDEyOklzc3VlQ29tbWVudDc1MzIxNzcxNA==,9599,simonw,2020-12-31T22:21:33Z,2020-12-31T22:21:33Z,OWNER,"Eventually I'd like to provide a whole bunch of other `datasette.X` utility functions that plugins can use - things like `datasette.addTabbedContentPane()` or similar.

But I don't want to inline those into the page.

So... I think the basic plugin system remains inline - maybe from an inlined file called `plugins-bootstrap.js`. Then a separate `plugins.js` contains the rest of the API functionality.

If a plugin wants to take advantage of those APIs, maybe it registers itself using `datasette.plugins.register('load', () => ...)` - that `load` hook can then be fired once the bulkier plugin code has been loaded.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/987#issuecomment-753217127,https://api.github.com/repos/simonw/datasette/issues/987,753217127,MDEyOklzc3VlQ29tbWVudDc1MzIxNzEyNw==,9599,simonw,2020-12-31T22:16:46Z,2020-12-31T22:16:46Z,OWNER,"I'm going to use `class=""plugin-content-pre-table""` rather than `id=` - just because I still want to be able to display all of this stuff on the single https://latest.datasette.io/-/patterns page so duplicate IDs are best avoided.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/983#issuecomment-753215761,https://api.github.com/repos/simonw/datasette/issues/983,753215761,MDEyOklzc3VlQ29tbWVudDc1MzIxNTc2MQ==,9599,simonw,2020-12-31T22:07:31Z,2020-12-31T22:07:31Z,OWNER,"I think I need to keep the mechanism whereby a plugin can return `undefined` in order to indicate that it has nothing to say for that specific item - that's borrowed from Pluggy and I've used it a bunch in my Python plugins. That makes the code a bit longer.

I'll write some example plugins to help me decide if the filtering-out-of-undefined mechanism is needed or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753215545,https://api.github.com/repos/simonw/datasette/issues/983,753215545,MDEyOklzc3VlQ29tbWVudDc1MzIxNTU0NQ==,9599,simonw,2020-12-31T22:05:41Z,2020-12-31T22:05:41Z,OWNER,Using object destructuring like that is a great idea. I'm going to play with your version  - it's delightfully succinct.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1166#issuecomment-753214664,https://api.github.com/repos/simonw/datasette/issues/1166,753214664,MDEyOklzc3VlQ29tbWVudDc1MzIxNDY2NA==,9599,simonw,2020-12-31T21:58:04Z,2020-12-31T21:58:04Z,OWNER,Wrote a TIL about this: https://til.simonwillison.net/github-actions/prettier-github-actions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753211535,https://api.github.com/repos/simonw/datasette/issues/1166,753211535,MDEyOklzc3VlQ29tbWVudDc1MzIxMTUzNQ==,9599,simonw,2020-12-31T21:46:04Z,2020-12-31T21:46:04Z,OWNER,"https://github.com/simonw/datasette/runs/1631682372?check_suite_focus=true failed!

<img width=""693"" alt=""Trying_out_bad_formatting__refs__1166_·_simonw_datasette_8087091"" src=""https://user-images.githubusercontent.com/9599/103426449-841e5c00-4b6e-11eb-95e6-26432a7b27dd.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753210536,https://api.github.com/repos/simonw/datasette/issues/1166,753210536,MDEyOklzc3VlQ29tbWVudDc1MzIxMDUzNg==,9599,simonw,2020-12-31T21:45:19Z,2020-12-31T21:45:19Z,OWNER,"Oops, committed that bad formatting test to `main` instead of a branch!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753209192,https://api.github.com/repos/simonw/datasette/issues/1166,753209192,MDEyOklzc3VlQ29tbWVudDc1MzIwOTE5Mg==,9599,simonw,2020-12-31T21:44:22Z,2020-12-31T21:44:22Z,OWNER,"Tests passed in https://github.com/simonw/datasette/runs/1631677726?check_suite_focus=true

I'm going to try submitting a pull request with badly formatted JavaScript to see if it gets caught.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753200580,https://api.github.com/repos/simonw/datasette/issues/1166,753200580,MDEyOklzc3VlQ29tbWVudDc1MzIwMDU4MA==,9599,simonw,2020-12-31T21:38:06Z,2020-12-31T21:38:06Z,OWNER,"I think this should work:
```
- uses: actions/cache@v2
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/prettier.yml' }}
```
I'll use the `prettier.yml` workflow that I'm about to create as the cache key for the NPM cache.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753197957,https://api.github.com/repos/simonw/datasette/issues/1166,753197957,MDEyOklzc3VlQ29tbWVudDc1MzE5Nzk1Nw==,9599,simonw,2020-12-31T21:36:14Z,2020-12-31T21:36:14Z,OWNER,"Maybe not that action actually - I wanted to use a pre-built action to avoid installing Prettier every time, but that's what it seems to do: https://github.com/creyD/prettier_action/blob/bb361e2979cff283ca7684908deac8f95400e779/entrypoint.sh#L28-L37","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753195905,https://api.github.com/repos/simonw/datasette/issues/1166,753195905,MDEyOklzc3VlQ29tbWVudDc1MzE5NTkwNQ==,9599,simonw,2020-12-31T21:34:46Z,2020-12-31T21:34:46Z,OWNER,This action looks good - tag 3.2 is equivalent to this commit hash: https://github.com/creyD/prettier_action/tree/bb361e2979cff283ca7684908deac8f95400e779,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753193475,https://api.github.com/repos/simonw/datasette/issues/1166,753193475,MDEyOklzc3VlQ29tbWVudDc1MzE5MzQ3NQ==,9599,simonw,2020-12-31T21:33:00Z,2020-12-31T21:33:00Z,OWNER,"I want a CI check that confirms that files conform to prettier - but only `datasette/static/*.js` files that are not already  minified.

This seems to do the job:

    npx prettier --check 'datasette/static/*[!.min].js'
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1165#issuecomment-752846267,https://api.github.com/repos/simonw/datasette/issues/1165,752846267,MDEyOklzc3VlQ29tbWVudDc1Mjg0NjI2Nw==,9599,simonw,2020-12-31T05:10:41Z,2020-12-31T05:13:14Z,OWNER,"https://github.com/PostHog/posthog/tree/master/cypress/integration has some useful examples, linked from this article: https://posthog.com/blog/cypress-end-to-end-tests

Also useful: their workflow https://github.com/PostHog/posthog/blob/master/.github/workflows/e2e.yml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752839433,https://api.github.com/repos/simonw/datasette/issues/1165,752839433,MDEyOklzc3VlQ29tbWVudDc1MjgzOTQzMw==,9599,simonw,2020-12-31T04:29:40Z,2020-12-31T04:29:40Z,OWNER,Important to absorb the slightly bizarre assertion syntax from Chai - docs here https://www.chaijs.com/api/bdd/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752828851,https://api.github.com/repos/simonw/datasette/issues/1165,752828851,MDEyOklzc3VlQ29tbWVudDc1MjgyODg1MQ==,9599,simonw,2020-12-31T03:19:38Z,2020-12-31T03:19:38Z,OWNER,"I got Cypress working! I added the `datasette.plugins` code to the table template and ran a test called `plugins.spec.js` using the following:
```javascript
context('datasette.plugins API', () => {
    beforeEach(() => {
      cy.visit('/fixtures/compound_three_primary_keys')
    });
    it('should exist', () => {
        let datasette;
        cy.window().then(win => {
            datasette = win.datasette;
        }).then(() => {
            expect(datasette).to.exist;
            expect(datasette.plugins).to.exist;
        });
    });
    it('should register and execute plugins', () => {
        let datasette;
        cy.window().then(win => {
            datasette = win.datasette;
        }).then(() => {
            expect(datasette.plugins.call('numbers')).to.deep.equal([]);
            // Register a plugin
            datasette.plugins.register(""numbers"", (a, b) => a + b, ['a', 'b']);
            var result = datasette.plugins.call(""numbers"", {a: 1, b: 2});
            expect(result).to.deep.equal([3]);
            // Second plugin
            datasette.plugins.register(""numbers"", (a, b) => a * b, ['a', 'b']);
            var result2 = datasette.plugins.call(""numbers"", {a: 1, b: 2});
            expect(result2).to.deep.equal([3, 2]);
        });
    });
});
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752780000,https://api.github.com/repos/simonw/datasette/issues/1165,752780000,MDEyOklzc3VlQ29tbWVudDc1Mjc4MDAwMA==,9599,simonw,2020-12-30T22:41:25Z,2020-12-30T22:41:25Z,OWNER,Jest works with Puppeteer: https://jestjs.io/docs/en/puppeteer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752779820,https://api.github.com/repos/simonw/datasette/issues/1165,752779820,MDEyOklzc3VlQ29tbWVudDc1Mjc3OTgyMA==,9599,simonw,2020-12-30T22:40:28Z,2020-12-30T22:40:28Z,OWNER,"I don't know if Jest on the command-line is the right tool for this. It works for the `plugins.js` script but I'm increasingly going to want to start adding tests for browser JavaScript features - like the https://github.com/simonw/datasette/blob/0.53/datasette/static/table.js script - which will need to run in a browser.

So maybe I should just find a browser testing solution and figure out how to run that under CI in GitHub Actions. Maybe https://www.cypress.io/ ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752779490,https://api.github.com/repos/simonw/datasette/issues/1165,752779490,MDEyOklzc3VlQ29tbWVudDc1Mjc3OTQ5MA==,9599,simonw,2020-12-30T22:38:43Z,2020-12-30T22:38:43Z,OWNER,Turned that into a TIL: https://til.simonwillison.net/javascript/jest-without-package-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752777744,https://api.github.com/repos/simonw/datasette/issues/1165,752777744,MDEyOklzc3VlQ29tbWVudDc1Mjc3Nzc0NA==,9599,simonw,2020-12-30T22:30:24Z,2020-12-30T22:30:24Z,OWNER,"https://www.valentinog.com/blog/jest/ was useful.

I created a `static/__tests__` folder and added this file as `plugins.spec.js`:

```javascript
const datasette = require(""../plugins.js"");

describe(""Datasette Plugins"", () => {
  test(""it should have datasette.plugins"", () => {
    expect(!!datasette.plugins).toEqual(true);
  });
  test(""registering a plugin should work"", () => {
    datasette.plugins.register(""numbers"", (a, b) => a + b, [""a"", ""b""]);
    var result = datasette.plugins.call(""numbers"", { a: 1, b: 2 });
    expect(result).toEqual([3]);
    datasette.plugins.register(""numbers"", (a, b) => a * b, [""a"", ""b""]);
    var result2 = datasette.plugins.call(""numbers"", { a: 1, b: 2 });
    expect(result2).toEqual([3, 2]);
  });
});
```

In `static/plugins.js` I put this:
```javascript
var datasette = datasette || {};
datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn, parameters) => {
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        call: (hook, args) => {
            args = args || {};
            var results = [];
            (registry[hook] || []).forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var result = fn.apply(fn, parameters.map(parameter => args[parameter]));
                if (result !== undefined) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();

module.exports = datasette;
```
Note the `module.exports` line at the end.

Then inside `static/` I ran the following command:

```
% npx jest -c '{}'
 PASS  __tests__/plugins.spec.js
  Datasette Plugins
    ✓ it should have datasette.plugins (3 ms)
    ✓ registering a plugin should work (1 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        1.163 s
Ran all test suites.
```
The `-c {}` was necessary because I didn't have a Jest configuration or a `package.json`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/983#issuecomment-752773508,https://api.github.com/repos/simonw/datasette/issues/983,752773508,MDEyOklzc3VlQ29tbWVudDc1Mjc3MzUwOA==,9599,simonw,2020-12-30T22:10:08Z,2020-12-30T22:11:34Z,OWNER,"https://twitter.com/dracos/status/1344402639476424706 points out that plugins returning 0 will be ignored.

This should probably check for `result !== undefined` instead - knocks the size up to 250.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752770488,https://api.github.com/repos/simonw/datasette/issues/983,752770488,MDEyOklzc3VlQ29tbWVudDc1Mjc3MDQ4OA==,9599,simonw,2020-12-30T21:55:35Z,2020-12-30T21:58:26Z,OWNER,"This one minifies to 241:
```javascript
var datasette = datasette || {};
datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn, parameters) => {
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        call: (hook, args) => {
            args = args || {};
            var results = [];
            (registry[hook] || []).forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var result = fn.apply(fn, parameters.map(parameter => args[parameter]));
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();
```
`var datasette=datasette||{};datasette.plugins=(()=>{var a={};return{register:(t,r,e)=>{a[t]||(a[t]=[]),a[t].push([r,e])},call:(t,r)=>{r=r||{};var e=[];return(a[t]||[]).forEach(([a,t])=>{var s=a.apply(a,t.map(a=>r[a]));s&&e.push(s)}),e}}})();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752770133,https://api.github.com/repos/simonw/datasette/issues/983,752770133,MDEyOklzc3VlQ29tbWVudDc1Mjc3MDEzMw==,9599,simonw,2020-12-30T21:53:45Z,2020-12-30T21:54:22Z,OWNER,"FixMyStreet inlines some JavaScript, and it's always a good idea to copy what they're doing when it comes to web performance: https://github.com/mysociety/fixmystreet/blob/23e9564b58a86b783ce47f3c0bf837cbd4fe7282/templates/web/base/common_header_tags.html#L19-L25

Note `var fixmystreet=fixmystreet||{};` which is shorter - https://twitter.com/dracos/status/1344399909794045954","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1164#issuecomment-752769452,https://api.github.com/repos/simonw/datasette/issues/1164,752769452,MDEyOklzc3VlQ29tbWVudDc1Mjc2OTQ1Mg==,9599,simonw,2020-12-30T21:50:16Z,2020-12-30T21:50:16Z,OWNER,If I implement this I can automate the CodeMirror minification and remove the bit about running `uglify-js` against it from the documentation here: https://docs.datasette.io/en/0.53/contributing.html#upgrading-codemirror,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-752768785,https://api.github.com/repos/simonw/datasette/issues/1164,752768785,MDEyOklzc3VlQ29tbWVudDc1Mjc2ODc4NQ==,9599,simonw,2020-12-30T21:47:06Z,2020-12-30T21:47:06Z,OWNER,If I'm going to minify `table.js` I'd like to offer a source map for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-752768652,https://api.github.com/repos/simonw/datasette/issues/1164,752768652,MDEyOklzc3VlQ29tbWVudDc1Mjc2ODY1Mg==,9599,simonw,2020-12-30T21:46:29Z,2020-12-30T21:46:29Z,OWNER,Running https://skalman.github.io/UglifyJS-online/ against https://github.com/simonw/datasette/blob/0.53/datasette/static/table.js knocks it down from 7810 characters to 4643.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/983#issuecomment-752767500,https://api.github.com/repos/simonw/datasette/issues/983,752767500,MDEyOklzc3VlQ29tbWVudDc1Mjc2NzUwMA==,9599,simonw,2020-12-30T21:42:07Z,2020-12-30T21:42:07Z,OWNER,"Another option: have both ""dev"" and ""production"" versions of the plugin mechanism script. Make it easy to switch between the two. Build JavaScript unit tests that exercise the ""production"" APIs against the development version, and have extra tests that just work against the features in the development version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752767174,https://api.github.com/repos/simonw/datasette/issues/983,752767174,MDEyOklzc3VlQ29tbWVudDc1Mjc2NzE3NA==,9599,simonw,2020-12-30T21:40:44Z,2020-12-30T21:40:44Z,OWNER,Started a Twitter thread about this here: https://twitter.com/simonw/status/1344392603794477056,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752751490,https://api.github.com/repos/simonw/datasette/issues/983,752751490,MDEyOklzc3VlQ29tbWVudDc1Mjc1MTQ5MA==,9599,simonw,2020-12-30T20:40:04Z,2020-12-30T21:34:22Z,OWNER,"This one is 683 bytes with Uglify - I like how https://skalman.github.io/UglifyJS-online/ shows you the minified character count as you edit the script:
```javascript
window.datasette = window.datasette || {};
window.datasette.plugins = (() => {
    var registry = {};
    var definitions = {};
    var stringify = JSON.stringify;

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }

    function isSubSet(a, b) {
        return a.every(parameter => b.includes(parameter))
    }

    return {
        _r: registry,
        define: (hook, parameters) => {
            definitions[hook] = parameters || [];
        },
        register: (hook, fn, parameters) => {
            parameters = parameters || extractParameters(fn);
            if (!definitions[hook]) {
                throw 'Hook ""' + hook + '"" not defined';
            }
            /* Check parameters is a subset of definitions[hook] */
            var validParameters = definitions[hook];
            if (!isSubSet(parameters, validParameters)) {
                throw '""' + hook + '"" valid args: ' + stringify(validParameters);
            }
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        
        call: (hook, args) => {
            args = args || {};
            if (!definitions[hook]) {
                throw '""' + hook + '"" hook not defined';
            }
            if (!isSubSet(Object.keys(args), definitions[hook])) {
                throw '""' + hook + '"" valid args: ' + stringify(definitions[hook]);
            }
            
            var implementations = registry[hook] || [];
            var results = [];
            implementations.forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var callWith = parameters.map(parameter => args[parameter]);
                var result = fn.apply(fn, callWith);
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }       
    };
})();
```
`window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var t={},r={},e=JSON.stringify;function i(t,r){return t.every(t=>r.includes(t))}return{_r:t,define:(t,e)=>{r[t]=e||[]},register:(a,n,o)=>{if(o=o||function(t){var r=/\((.*)\)/.exec(t.toString());return r&&r[1].trim()?r[1].split("","").map(t=>t.trim()):[]}(n),!r[a])throw'Hook ""'+a+'"" not defined';var d=r[a];if(!i(o,d))throw'""'+a+'"" valid args: '+e(d);t[a]||(t[a]=[]),t[a].push([n,o])},call:(a,n)=>{if(n=n||{},!r[a])throw'""'+a+'"" hook not defined';if(!i(Object.keys(n),r[a]))throw'""'+a+'"" valid args: '+e(r[a]);var o=t[a]||[],d=[];return o.forEach(([t,r])=>{var e=r.map(t=>n[t]),i=t.apply(t,e);i&&d.push(i)}),d}}})();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752760815,https://api.github.com/repos/simonw/datasette/issues/983,752760815,MDEyOklzc3VlQ29tbWVudDc1Mjc2MDgxNQ==,9599,simonw,2020-12-30T21:15:41Z,2020-12-30T21:15:41Z,OWNER,"I'm going to write a few example plugins and try them out against the longer and shorter versions of the script, to get a better feel for how useful the longer versions with the error handling and explicit definition actually are.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752760054,https://api.github.com/repos/simonw/datasette/issues/983,752760054,MDEyOklzc3VlQ29tbWVudDc1Mjc2MDA1NA==,9599,simonw,2020-12-30T21:12:36Z,2020-12-30T21:14:05Z,OWNER,"I gotta admit that 262 byte version is pretty tempting, if it's going to end up in the `<head>` of every single page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752759885,https://api.github.com/repos/simonw/datasette/issues/983,752759885,MDEyOklzc3VlQ29tbWVudDc1Mjc1OTg4NQ==,9599,simonw,2020-12-30T21:11:52Z,2020-12-30T21:14:00Z,OWNER,"262 bytes if I remove the parameter introspection code, instead requiring plugin authors to specify the arguments they take:
```javascript
window.datasette = window.datasette || {};
window.datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn, parameters) => {
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        call: (hook, args) => {
            args = args || {};
            var results = [];
            (registry[hook] || []).forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var callWith = parameters.map(parameter => args[parameter]);
                var result = fn.apply(fn, callWith);
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();
```
`window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var a={};return{register:(t,e,r)=>{a[t]||(a[t]=[]),a[t].push([e,r])},call:(t,e)=>{e=e||{};var r=[];return(a[t]||[]).forEach(([a,t])=>{var s=t.map(a=>e[a]),d=a.apply(a,s);d&&r.push(d)}),r}}})();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752758802,https://api.github.com/repos/simonw/datasette/issues/983,752758802,MDEyOklzc3VlQ29tbWVudDc1Mjc1ODgwMg==,9599,simonw,2020-12-30T21:07:33Z,2020-12-30T21:10:10Z,OWNER,"Removing the `datasette.plugin.define()` method and associated error handling reduces the uglified version from 683 bytes to 380 bytes. I think the error checking is worth the extra 303 bytes per page load, even if it's only really needed for a better developer experience.
```javascript
window.datasette = window.datasette || {};
window.datasette.plugins = (() => {
    var registry = {};

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }
    return {
        register: (hook, fn, parameters) => {
            parameters = parameters || extractParameters(fn);
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        
        call: (hook, args) => {
            args = args || {};
            var implementations = registry[hook] || [];
            var results = [];
            implementations.forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var callWith = parameters.map(parameter => args[parameter]);
                var result = fn.apply(fn, callWith);
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();
```
`window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var t={};return{register:(r,a,e)=>{e=e||function(t){var r=/\((.*)\)/.exec(t.toString());return r&&r[1].trim()?r[1].split("","").map(t=>t.trim()):[]}(a),t[r]||(t[r]=[]),t[r].push([a,e])},call:(r,a)=>{a=a||{};var e=t[r]||[],i=[];return e.forEach(([t,r])=>{var e=r.map(t=>a[t]),n=t.apply(t,e);n&&i.push(n)}),i}}})();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1165#issuecomment-752757910,https://api.github.com/repos/simonw/datasette/issues/1165,752757910,MDEyOklzc3VlQ29tbWVudDc1Mjc1NzkxMA==,9599,simonw,2020-12-30T21:04:18Z,2020-12-30T21:04:18Z,OWNER,https://jestjs.io/ looks worth trying here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/983#issuecomment-752757289,https://api.github.com/repos/simonw/datasette/issues/983,752757289,MDEyOklzc3VlQ29tbWVudDc1Mjc1NzI4OQ==,9599,simonw,2020-12-30T21:02:20Z,2020-12-30T21:02:20Z,OWNER,I'm going to need to add JavaScript unit tests for this new plugin system.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1164#issuecomment-752757075,https://api.github.com/repos/simonw/datasette/issues/1164,752757075,MDEyOklzc3VlQ29tbWVudDc1Mjc1NzA3NQ==,9599,simonw,2020-12-30T21:01:27Z,2020-12-30T21:01:27Z,OWNER,"I don't want Datasette contributors to need a working Node.js install to run the tests or work on Datasette unless they are explicitly working on the JavaScript.

I think I'm going to do this with a unit test that runs only if `upglify-js` is available on the path and confirms that the `*.min.js` version of each script in the repository correctly matches the results from running `uglify-js` against it.

That way if anyone checks in a change to JavaScript but forgets to run the minifier the tests will fail in CI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-752756612,https://api.github.com/repos/simonw/datasette/issues/1164,752756612,MDEyOklzc3VlQ29tbWVudDc1Mjc1NjYxMg==,9599,simonw,2020-12-30T20:59:54Z,2020-12-30T20:59:54Z,OWNER,"I tried a few different pure-Python JavaScript minifying libraries and none of them produced results as good as https://www.npmjs.com/package/uglify-js for the plugin code I'm considering in #983.

So I think I'll need to rely on a Node.js tool for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/983#issuecomment-752750551,https://api.github.com/repos/simonw/datasette/issues/983,752750551,MDEyOklzc3VlQ29tbWVudDc1Mjc1MDU1MQ==,9599,simonw,2020-12-30T20:36:38Z,2020-12-30T20:37:48Z,OWNER,"This version minifies to 702 characters:
```javascript
window.datasette = window.datasette || {};
window.datasette.plugins = (() => {
    var registry = {};
    var definitions = {};
    var stringify = JSON.stringify;

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }

    function isSubSet(a, b) {
        return a.every(parameter => b.includes(parameter))
    }

    return {
        _registry: registry,
        define: (hook, parameters) => {
            definitions[hook] = parameters || [];
        },
        register: (hook, fn, parameters) => {
            parameters = parameters || extractParameters(fn);
            if (!definitions[hook]) {
                throw '""' + hook + '"" is not a defined hook';
            }
            /* Check parameters is a subset of definitions[hook] */
            var validParameters = definitions[hook];
            if (!isSubSet(parameters, validParameters)) {
                throw '""' + hook + '"" valid args are ' + stringify(validParameters);
            }
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        
        call: (hook, args) => {
            args = args || {};
            if (!definitions[hook]) {
                throw '""' + hook + '"" hook is not defined';
            }
            if (!isSubSet(Object.keys(args), definitions[hook])) {
                throw '""' + hook + '"" valid args: ' + stringify(definitions[hook]);
            }
            
            var implementations = registry[hook] || [];
            var results = [];
            implementations.forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var callWith = parameters.map(parameter => args[parameter]);
                var result = fn.apply(fn, callWith);
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }       
    };
})();
```
Or 701 characters using https://skalman.github.io/UglifyJS-online/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752749189,https://api.github.com/repos/simonw/datasette/issues/983,752749189,MDEyOklzc3VlQ29tbWVudDc1Mjc0OTE4OQ==,9599,simonw,2020-12-30T20:31:28Z,2020-12-30T20:31:28Z,OWNER,"Using raw string exceptions, `throw '""' + hook + '"" hook has not been defined';`, knocks it down to 795 characters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752748496,https://api.github.com/repos/simonw/datasette/issues/983,752748496,MDEyOklzc3VlQ29tbWVudDc1Mjc0ODQ5Ng==,9599,simonw,2020-12-30T20:28:48Z,2020-12-30T20:28:48Z,OWNER,If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752747999,https://api.github.com/repos/simonw/datasette/issues/983,752747999,MDEyOklzc3VlQ29tbWVudDc1Mjc0Nzk5OQ==,9599,simonw,2020-12-30T20:27:00Z,2020-12-30T20:27:00Z,OWNER,"I need to decide how this code is going to be loaded. Putting it in a blocking `<script>` element in the head would work, but I'd rather not block loading of the rest of the page. Using a `<script async>` method would be nicer, but then I have to worry about plugins attempting to register themselves before the page has fully loaded.

Running it through https://javascript-minifier.com/ produces this, which is 855 characters - so maybe I could inline that into the header of the page?

`window.datasette={},window.datasette.plugins=function(){var r={},n={};function e(r,n){return r.every(r=>n.includes(r))}return{define:function(r,e){n[r]=e||[]},register:function(t,i,o){if(o=o||function(r){var n=/\((.*)\)/.exec(r.toString());return n&&n[1].trim()?n[1].split("","").map(r=>r.trim()):[]}(i),!n[t])throw new Error('""'+t+'"" is not a defined plugin hook');if(!n[t])throw new Error('""'+t+'"" is not a defined plugin hook');var a=n[t];if(!e(o,a))throw new Error('""'+t+'"" valid parameters are '+JSON.stringify(a));r[t]||(r[t]=[]),r[t].push([i,o])},_registry:r,call:function(t,i){if(i=i||{},!n[t])throw new Error('""'+t+'"" hook has not been defined');if(!e(Object.keys(i),n[t]))throw new Error('""'+t+'"" valid arguments are '+JSON.stringify(n[t]));var o=r[t]||[],a=[];return o.forEach(([r,n])=>{var e=n.map(r=>i[r]),t=r.apply(r,e);t&&a.push(t)}),a}}}();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752747169,https://api.github.com/repos/simonw/datasette/issues/983,752747169,MDEyOklzc3VlQ29tbWVudDc1Mjc0NzE2OQ==,9599,simonw,2020-12-30T20:24:07Z,2020-12-30T20:24:07Z,OWNER,"This version adds `datasette.plugins.define()` plus extra validation of both `.register()` and `.call()`:
```javascript
window.datasette = {};
window.datasette.plugins = (function() {
    var registry = {};
    var definitions = {};

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }

    function define(hook, parameters) {
        definitions[hook] = parameters || [];
    }

    function isSubSet(a, b) {
        return a.every(parameter => b.includes(parameter))
    }

    function register(hook, fn, parameters) {
        parameters = parameters || extractParameters(fn);
        if (!definitions[hook]) {
            throw new Error('""' + hook + '"" is not a defined plugin hook');
        }
        if (!definitions[hook]) {
            throw new Error('""' + hook + '"" is not a defined plugin hook');
        }
        /* Check parameters is a subset of definitions[hook] */
        var validParameters = definitions[hook];
        if (!isSubSet(parameters, validParameters)) {
            throw new Error('""' + hook + '"" valid parameters are ' + JSON.stringify(validParameters));
        }
        if (!registry[hook]) {
            registry[hook] = [];
        }
        registry[hook].push([fn, parameters]);
    }

    function call(hook, args) {
        args = args || {};
        if (!definitions[hook]) {
            throw new Error('""' + hook + '"" hook has not been defined');
        }
        if (!isSubSet(Object.keys(args), definitions[hook])) {
            throw new Error('""' + hook + '"" valid arguments are ' + JSON.stringify(definitions[hook]));
        }
        
        var implementations = registry[hook] || [];
        var results = [];
        implementations.forEach(([fn, parameters]) => {
            /* Call with the correct arguments */
            var callWith = parameters.map(parameter => args[parameter]);
            var result = fn.apply(fn, callWith);
            if (result) {
                results.push(result);
            }
        });
        return results;
    }
    return {
        define: define,
        register: register,
        _registry: registry,
        call: call
    };
})();
```
Usage:
```javascript
datasette.plugins.define('numbers', ['a', 'b'])
datasette.plugins.register('numbers', (a, b) => a + b)
datasette.plugins.register('numbers', (a, b) => a * b)
datasette.plugins.call('numbers', {a: 4, b: 6})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752744311,https://api.github.com/repos/simonw/datasette/issues/983,752744311,MDEyOklzc3VlQ29tbWVudDc1Mjc0NDMxMQ==,9599,simonw,2020-12-30T20:12:50Z,2020-12-30T20:13:02Z,OWNER,"This could work to define a plugin hook:
```javascript
datasette.plugins.define('numbers', ['a' ,'b'])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752744195,https://api.github.com/repos/simonw/datasette/issues/983,752744195,MDEyOklzc3VlQ29tbWVudDc1Mjc0NDE5NQ==,9599,simonw,2020-12-30T20:12:26Z,2020-12-30T20:12:26Z,OWNER,"This implementation doesn't have an equivalent of ""hookspecs"" which can identify if a registered plugin implementation matches a known signature. I should add that, it will provide a better developer experience if someone has a typo.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752742669,https://api.github.com/repos/simonw/datasette/issues/983,752742669,MDEyOklzc3VlQ29tbWVudDc1Mjc0MjY2OQ==,9599,simonw,2020-12-30T20:07:05Z,2020-12-30T20:07:18Z,OWNER,"Initial prototype:
```javascript
window.datasette = {};
window.datasette.plugins = (function() {
    var registry = {};

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }

    function register(hook, fn, parameters) {
        parameters = parameters || extractParameters(fn);
        if (!registry[hook]) {
            registry[hook] = [];
        }
        registry[hook].push([fn, parameters]);
    }

    function call(hook, args) {
        args = args || {};
        var implementations = registry[hook] || [];
        var results = [];
        implementations.forEach(([fn, parameters]) => {
            /* Call with the correct arguments */
            var callWith = parameters.map(parameter => args[parameter]);
            var result = fn.apply(fn, callWith);
            if (result) {
                results.push(result);
            }
        });
        return results;
    }
    return {
        register: register,
        _registry: registry,
        call: call
    };
})();
```
Usage example:
```javascript
datasette.plugins.register('numbers', (a, b) => a + b)
datasette.plugins.register('numbers', (a, b) => a * b)
datasette.plugins.call('numbers', {a: 4, b: 6})
/* Returns [10, 24] */
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752729035,https://api.github.com/repos/simonw/datasette/issues/983,752729035,MDEyOklzc3VlQ29tbWVudDc1MjcyOTAzNQ==,9599,simonw,2020-12-30T19:15:56Z,2020-12-30T19:16:44Z,OWNER,"The `column_actions` hook is the obvious first place to try this out. What are some demo plugins I could build for it?

- Word cloud for this column
- Count values (essentially linking to the SQL query for that column, as an extended version of the facet counts) - would be great if this could include pagination somehow, via #856.
- Extract this column into a separate table
- Add an index to this column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752722863,https://api.github.com/repos/simonw/datasette/issues/983,752722863,MDEyOklzc3VlQ29tbWVudDc1MjcyMjg2Mw==,9599,simonw,2020-12-30T18:52:39Z,2020-12-30T18:52:39Z,OWNER,"Then to call the plugins:
```javascript
datasette.plugins.call('column_actions', {database: 'database', table: 'table'})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752721840,https://api.github.com/repos/simonw/datasette/issues/983,752721840,MDEyOklzc3VlQ29tbWVudDc1MjcyMTg0MA==,9599,simonw,2020-12-30T18:48:53Z,2020-12-30T18:51:51Z,OWNER,"Potential design:
```javascript
datasette.plugins.register('column_actions', function(database, table, column, actor) {
    /* ... *l
})
```
Or if you want to be explicit to survive minification:
```javascript
datasette.plugins.register('column_actions', function(database, table, column, actor) {
    /* ... *l
}, ['database', 'table', 'column', 'actor'])
```
I'm making that list of parameter names an optional third argument to the `register()` function. If that argument isn't passed, introspection will be used to figure out the parameter names.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752721069,https://api.github.com/repos/simonw/datasette/issues/983,752721069,MDEyOklzc3VlQ29tbWVudDc1MjcyMTA2OQ==,9599,simonw,2020-12-30T18:46:10Z,2020-12-30T18:46:10Z,OWNER,"Pluggy does dependency injection by introspecting the named arguments to the Python function, which I really like.

That's tricker in JavaScript. It looks like the only way to introspect a function is to look at the `.toString()` representation of it and parse the `(parameter, list)` using a regular expression.

Even more challenging: JavaScript developers love minifying their code, and minification can shorten the function parameter names.

From https://code-maven.com/dependency-injection-in-angularjs it looks like Angular.js does dependency injection and solves this by letting you optionally provide a separate list of the arguments your function uses:

```javascript
   angular.module('DemoApp', [])
   .controller('DemoController', ['$scope', '$log', function($scope, $log) {
       $scope.message = ""Hello World"";
       $log.debug('logging hello');
   }]);
```

I can copy that approach: I'll introspect by default, but provide a documented mechanism for explicitly listing your parameter names so that if you know your plugin code will be minified you can use that instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752715412,https://api.github.com/repos/simonw/datasette/issues/983,752715412,MDEyOklzc3VlQ29tbWVudDc1MjcxNTQxMg==,9599,simonw,2020-12-30T18:25:31Z,2020-12-30T18:25:31Z,OWNER,I'm going to introduce a global `datasette` object which holds all the documented JavaScript API for plugin authors.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752715236,https://api.github.com/repos/simonw/datasette/issues/983,752715236,MDEyOklzc3VlQ29tbWVudDc1MjcxNTIzNg==,9599,simonw,2020-12-30T18:24:54Z,2020-12-30T18:24:54Z,OWNER,"I think I'm going to try building a very lightweight clone of the core API design of Pluggy - not the advanced features, just the idea that plugins can register and a call to `plugin.nameOfHook()` will return the concatenated results of all of the registered hooks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/987#issuecomment-752714747,https://api.github.com/repos/simonw/datasette/issues/987,752714747,MDEyOklzc3VlQ29tbWVudDc1MjcxNDc0Nw==,9599,simonw,2020-12-30T18:23:08Z,2020-12-30T18:23:20Z,OWNER,"In terms of ""places to put your plugin content"", the simplest solution I can think of is something like this:
```html
<div id=""plugin-content-pre-table""></div>
```
Alternative designs:

- A documented JavaScript function that returns the CSS selector where plugins should put their content
- A documented JavaScript function that returns a DOM node where plugins should put their content. This would allow the JavaScript to create the element if it does not already exist (though it wouldn't be obvious WHERE that element should be created)
- Documented JavaScript functions for things like ""append this node/HTML to the place-where-plugins-go""

I think the original option - an empty `<div>` with a known `id` attribute - is the right one to go with here. It's the simplest, it's very easy for custom template authors to understand and it acknowledges that plugins may have all kinds of extra crazy stuff they want to do - like checking in that div to see if another plugin has written to it already, for example.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/987#issuecomment-752697279,https://api.github.com/repos/simonw/datasette/issues/987,752697279,MDEyOklzc3VlQ29tbWVudDc1MjY5NzI3OQ==,9599,simonw,2020-12-30T17:23:27Z,2020-12-30T17:23:32Z,OWNER,Related problem: right now if you're writing custom template it's not at all obvious how to write them such that visualization plugins like `datasette-cluster-map` will continue to work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/987#issuecomment-752696499,https://api.github.com/repos/simonw/datasette/issues/987,752696499,MDEyOklzc3VlQ29tbWVudDc1MjY5NjQ5OQ==,9599,simonw,2020-12-30T17:21:08Z,2020-12-30T17:21:08Z,OWNER,"More generally, I need to document certain areas of the page that JavaScript plugins are invited to append their content to - such that plugin authors can use them and feel confident that future changes to the Datasette templates won't break their plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/1160#issuecomment-752275611,https://api.github.com/repos/simonw/datasette/issues/1160,752275611,MDEyOklzc3VlQ29tbWVudDc1MjI3NTYxMQ==,9599,simonw,2020-12-29T23:32:04Z,2020-12-29T23:32:04Z,OWNER,"If I can get this working for CSV, TSV, JSON and JSON-NL that should be enough to exercise the API design pretty well across both streaming and non-streaming formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752274509,https://api.github.com/repos/simonw/datasette/issues/1160,752274509,MDEyOklzc3VlQ29tbWVudDc1MjI3NDUwOQ==,9599,simonw,2020-12-29T23:26:02Z,2020-12-29T23:26:02Z,OWNER,"The documentation for this plugin hook is going to be pretty detailed, since it involves writing custom classes.

I'll stick it all on the existing hooks page for the moment, but I should think about breaking up the plugin hook documentation into a page-per-hook in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1163#issuecomment-752274340,https://api.github.com/repos/simonw/datasette/issues/1163,752274340,MDEyOklzc3VlQ29tbWVudDc1MjI3NDM0MA==,9599,simonw,2020-12-29T23:25:02Z,2020-12-29T23:25:02Z,OWNER,This will be built on top of `httpx` since that's already a dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776128565,"""datasette insert data.db url-to-csv""",
https://github.com/simonw/datasette/issues/1160#issuecomment-752274078,https://api.github.com/repos/simonw/datasette/issues/1160,752274078,MDEyOklzc3VlQ29tbWVudDc1MjI3NDA3OA==,9599,simonw,2020-12-29T23:23:39Z,2020-12-29T23:23:39Z,OWNER,"If I design this right I can ship a full version of the command-line `datasette insert` command in a release without doing any work at all on the Web UI version of it - that UI can then come later, without needing any changes to be made to the plugin hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752273873,https://api.github.com/repos/simonw/datasette/issues/1160,752273873,MDEyOklzc3VlQ29tbWVudDc1MjI3Mzg3Mw==,9599,simonw,2020-12-29T23:22:30Z,2020-12-29T23:22:30Z,OWNER,"How much of this should I get done in a branch before merging into `main`?

The challenge here is the plugin hook design: ideally I don't want an incomplete plugin hook design in `main` since that could be a blocker for a release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752273400,https://api.github.com/repos/simonw/datasette/issues/1160,752273400,MDEyOklzc3VlQ29tbWVudDc1MjI3MzQwMA==,9599,simonw,2020-12-29T23:19:46Z,2020-12-29T23:19:46Z,OWNER,I'm going to break out some separate tickets.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752273306,https://api.github.com/repos/simonw/datasette/issues/1160,752273306,MDEyOklzc3VlQ29tbWVudDc1MjI3MzMwNg==,9599,simonw,2020-12-29T23:19:15Z,2020-12-29T23:19:15Z,OWNER,It would be nice if this abstraction could support progress bars as well. These won't necessarily work for every format - or they might work for things loaded from files but not things loaded over URLs (if the `content-length` HTTP header is missing) - but if they ARE possible it would be good to provide them - both for the CLI interface and the web insert UI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752267905,https://api.github.com/repos/simonw/datasette/issues/1160,752267905,MDEyOklzc3VlQ29tbWVudDc1MjI2NzkwNQ==,9599,simonw,2020-12-29T22:52:09Z,2020-12-29T22:52:09Z,OWNER,"What's the simplest thing that could possible work? I think it's `datasette insert blah.db data.csv` - no URL handling, no other formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752266076,https://api.github.com/repos/simonw/datasette/issues/1160,752266076,MDEyOklzc3VlQ29tbWVudDc1MjI2NjA3Ng==,9599,simonw,2020-12-29T22:42:23Z,2020-12-29T22:42:59Z,OWNER,"Aside: maybe `datasette insert` works against simple files, but a later mechanism called `datasette import` allows plugins to register sub-commands, like `datasette import github ...` or `datasette import jira ...` or whatever.

This would be useful for import mechanisms that are likely to need their own custom set of command-line options unique to that source.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752265600,https://api.github.com/repos/simonw/datasette/issues/1160,752265600,MDEyOklzc3VlQ29tbWVudDc1MjI2NTYwMA==,9599,simonw,2020-12-29T22:39:56Z,2020-12-29T22:39:56Z,OWNER,"Does it definitely make sense to break this operation up into the code that turns the incoming format into a iterator of dictionaries, then the code that inserts those into the database using `sqlite-utils`?

That seems right for simple imports, where the incoming file represents a sequence of records in a single table. But what about more complex formats? What if a format needs to be represented as multiple tables?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752259345,https://api.github.com/repos/simonw/datasette/issues/1160,752259345,MDEyOklzc3VlQ29tbWVudDc1MjI1OTM0NQ==,9599,simonw,2020-12-29T22:11:54Z,2020-12-29T22:11:54Z,OWNER,"Important detail from https://docs.python.org/3/library/csv.html#csv.reader

> If *csvfile* is a file object, it should be opened with `newline=''`. [1]
>
> [...]
>
> If `newline=''` is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use `\r\n` linendings on write an extra `\r` will be added. It should always be safe to specify `newline=''`, since the csv module does its own ([universal](https://docs.python.org/3/glossary.html#term-universal-newlines)) newline handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752257666,https://api.github.com/repos/simonw/datasette/issues/1160,752257666,MDEyOklzc3VlQ29tbWVudDc1MjI1NzY2Ng==,9599,simonw,2020-12-29T22:09:18Z,2020-12-29T22:09:18Z,OWNER,"### Figuring out the API design

I want to be able to support different formats, and be able to parse them into tables either streaming or in one go depending on if the format supports that.

Ideally I want to be able to pull the first 1,024 bytes for the purpose of detecting the format, then replay those bytes again later. I'm considering this a stretch goal though.

CSV is easy to parse as a stream - here’s [how sqlite-utils does it](https://github.com/simonw/sqlite-utils/blob/f1277f638f3a54a821db6e03cb980adad2f2fa35/sqlite_utils/cli.py#L630):

        dialect = ""excel-tab"" if tsv else ""excel""
        with file_progress(json_file, silent=silent) as json_file:
            reader = csv_std.reader(json_file, dialect=dialect)
            headers = next(reader)
            docs = (dict(zip(headers, row)) for row in reader)

Problem: using `db.insert_all()` could block for a long time on a big set of rows. Probably easiest to batch the records before calling `insert_all()` and then run a batch at a time using a `db.execute_write_fn()` call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1161#issuecomment-752253095,https://api.github.com/repos/simonw/datasette/issues/1161,752253095,MDEyOklzc3VlQ29tbWVudDc1MjI1MzA5NQ==,9599,simonw,2020-12-29T21:49:57Z,2020-12-29T21:49:57Z,OWNER,https://github.com/Homebrew/homebrew-core/pull/67983,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776101101,Update a whole bunch of links to datasette.io instead of datasette.readthedocs.io,
https://github.com/simonw/datasette/issues/1160#issuecomment-752236520,https://api.github.com/repos/simonw/datasette/issues/1160,752236520,MDEyOklzc3VlQ29tbWVudDc1MjIzNjUyMA==,9599,simonw,2020-12-29T20:48:51Z,2020-12-29T20:48:51Z,OWNER,It would be neat if `datasette insert` could accept a `--plugins-dir` option which allowed one-off format plugins to be registered. Bit tricky to implement since the `--format` Click option will already be populated by that plugin hook call.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751925934,https://api.github.com/repos/simonw/datasette/issues/1160,751925934,MDEyOklzc3VlQ29tbWVudDc1MTkyNTkzNA==,9599,simonw,2020-12-29T02:40:13Z,2020-12-29T20:25:57Z,OWNER,"Basic command design:

    datasette insert data.db blah.csv

The options can include:

- `--format` to specify the exact format - without this it will be guessed based on the filename
- `--table` to specify the table (otherwise the filename is used)
- `--pk` to specify one or more primary key columns
- `--replace` to specify that existing rows with a matching primary key should be replaced
- `--upsert` to specify that existing matching rows should be upserted
- `--ignore` to ignore matching rows
- `--alter` to alter the table to add missing columns
- `--type column type` to specify the type of a column - useful when working with CSV or TSV files","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752208036,https://api.github.com/repos/simonw/datasette/issues/1160,752208036,MDEyOklzc3VlQ29tbWVudDc1MjIwODAzNg==,9599,simonw,2020-12-29T19:06:35Z,2020-12-29T19:06:35Z,OWNER,"If I'm going to execute 1000s of writes in an `async def` operation it may make sense to break that up into smaller chunks, so as not to block the event loop for too long.

https://stackoverflow.com/a/36648102 and https://github.com/python/asyncio/issues/284 confirm that `await asyncio.sleep(0)` is the recommended way of doing this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752203909,https://api.github.com/repos/simonw/datasette/issues/1160,752203909,MDEyOklzc3VlQ29tbWVudDc1MjIwMzkwOQ==,9599,simonw,2020-12-29T18:54:19Z,2020-12-29T18:54:19Z,OWNER,More thoughts on this: the key mechanism that populates the tables needs to be an `aysnc def` method of some sort so that it can run as part of the async loop in core Datasette - for importing from web uploads.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751947991,https://api.github.com/repos/simonw/datasette/issues/1160,751947991,MDEyOklzc3VlQ29tbWVudDc1MTk0Nzk5MQ==,9599,simonw,2020-12-29T05:06:50Z,2020-12-29T05:07:03Z,OWNER,"Given the URL option could it be possible for plugins to ""subscribe"" to URLs that keep on streaming?

    datasette insert db.db https://example.con/streaming-api \
      --format api-stream","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751946262,https://api.github.com/repos/simonw/datasette/issues/1160,751946262,MDEyOklzc3VlQ29tbWVudDc1MTk0NjI2Mg==,9599,simonw,2020-12-29T04:56:12Z,2020-12-29T04:56:32Z,OWNER,"Potential design for this: a `datasette memory` command which takes most of the same arguments as `datasette serve` but starts an in-memory database and treats the command arguments as things that should be inserted into that in-memory database.

    tail -f access.log | datasette memory - \
      --format clf -p 8002 -o","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751945094,https://api.github.com/repos/simonw/datasette/issues/1160,751945094,MDEyOklzc3VlQ29tbWVudDc1MTk0NTA5NA==,9599,simonw,2020-12-29T04:48:11Z,2020-12-29T04:48:11Z,OWNER,"It would be pretty cool if you could launch Datasette directly against an insert-compatible file or URL without first having to load it into a SQLite database file.

Or imagine being able to tail a log file and like that directly into a new Datasette process, which then runs a web server with the UI while simultaneously continuing to load new entries from that log into the in-memory SQLite database that it is serving...

Not quite sure what that CLI interface would look like. Maybe treat that as a future stretch goal for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751943837,https://api.github.com/repos/simonw/datasette/issues/1160,751943837,MDEyOklzc3VlQ29tbWVudDc1MTk0MzgzNw==,9599,simonw,2020-12-29T04:40:30Z,2020-12-29T04:40:30Z,OWNER,"The `insert` command should also accept URLs - anything starting with `http://` or `https://`.

It should accept more than one file name at a time for bulk inserts.

if using a URL that URL will be passed to the method that decides if a plugin implementation can handle the import or not. This will allow plugins to register themselves for specific websites.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751926437,https://api.github.com/repos/simonw/datasette/issues/1160,751926437,MDEyOklzc3VlQ29tbWVudDc1MTkyNjQzNw==,9599,simonw,2020-12-29T02:43:21Z,2020-12-29T02:43:37Z,OWNER,"Default formats to support:

- CSV
- TSV
- JSON and newline-delimited JSON
- YAML

Each of these will be implemented as a default plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751926218,https://api.github.com/repos/simonw/datasette/issues/1160,751926218,MDEyOklzc3VlQ29tbWVudDc1MTkyNjIxOA==,9599,simonw,2020-12-29T02:41:57Z,2020-12-29T02:41:57Z,OWNER,"Other names I considered:

- `datasette load`
- `datasette import` - I decided to keep this name available for any future work that might involve plugins that help import data from APIs as opposed to inserting it from files","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751926095,https://api.github.com/repos/simonw/datasette/issues/1160,751926095,MDEyOklzc3VlQ29tbWVudDc1MTkyNjA5NQ==,9599,simonw,2020-12-29T02:41:15Z,2020-12-29T02:41:15Z,OWNER,"The UI can live at `/-/insert` and be available by default to the `root` user only. It can offer the following:

- Upload a file and have the import type detected (equivalent to `datasette insert data.db thatfile.csv`)
- Copy and paste the data to be inserted into a textarea
- API equivalents of these","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/675#issuecomment-751826749,https://api.github.com/repos/simonw/datasette/issues/675,751826749,MDEyOklzc3VlQ29tbWVudDc1MTgyNjc0OQ==,9599,simonw,2020-12-28T18:49:21Z,2020-12-28T18:49:21Z,OWNER,"That `--exec` could help solve all sorts of other problems too, like needing to `apt-get install` extra packages or download files from somewhere using `wget`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/675#issuecomment-751826621,https://api.github.com/repos/simonw/datasette/issues/675,751826621,MDEyOklzc3VlQ29tbWVudDc1MTgyNjYyMQ==,9599,simonw,2020-12-28T18:48:51Z,2020-12-28T18:48:51Z,OWNER,"I could make `--include` work if I also had a mechanism for running some shell commands inside the container at the end of the build - which users could then use to move files into the correct place.

    datasette publish cloudrun my.db --include src/ --exec 'mv /app/src/config.yml /etc/conf/config.yml'","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/417#issuecomment-751127485,https://api.github.com/repos/simonw/datasette/issues/417,751127485,MDEyOklzc3VlQ29tbWVudDc1MTEyNzQ4NQ==,9599,simonw,2020-12-24T22:58:05Z,2020-12-24T22:58:05Z,OWNER,"That's a great idea. I'd ruled that out because working with the different operating system versions of those is tricky, but if `watchdog` can handle those differences for me this could be a really good option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",421546944,Datasette Library,
https://github.com/simonw/datasette/pull/1158#issuecomment-750390741,https://api.github.com/repos/simonw/datasette/issues/1158,750390741,MDEyOklzc3VlQ29tbWVudDc1MDM5MDc0MQ==,9599,simonw,2020-12-23T17:05:32Z,2020-12-23T17:05:32Z,OWNER,"Thanks for this!

I'm fine keeping the `os.path` stuff as is.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",773913793,Modernize code to Python 3.6+,
https://github.com/simonw/datasette/pull/1158#issuecomment-750373603,https://api.github.com/repos/simonw/datasette/issues/1158,750373603,MDEyOklzc3VlQ29tbWVudDc1MDM3MzYwMw==,9599,simonw,2020-12-23T16:26:21Z,2020-12-23T16:26:21Z,OWNER,"Did you use a tool to find these or are they all from manual code inspection?

They look good - I particularly like the set/dict  literals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",773913793,Modernize code to Python 3.6+,
https://github.com/simonw/datasette/issues/1099#issuecomment-749845797,https://api.github.com/repos/simonw/datasette/issues/1099,749845797,MDEyOklzc3VlQ29tbWVudDc0OTg0NTc5Nw==,9599,simonw,2020-12-23T00:13:29Z,2020-12-23T00:14:25Z,OWNER,"Also need to solve displaying these links in the opposite direction:

https://latest.datasette.io/_internal/tables/fixtures,facet_cities

<img width=""617"" alt=""_internal__tables"" src=""https://user-images.githubusercontent.com/9599/102944742-84cd3900-4470-11eb-95a0-bfaf00b17e82.png"">

That page should link to lists of records in columns, foreign_keys and indexes - like this example: https://latest.datasette.io/fixtures/roadside_attractions/1

<img width=""712"" alt=""fixtures__roadside_attractions"" src=""https://user-images.githubusercontent.com/9599/102944816-c1009980-4470-11eb-9dbd-50d0ba4fd137.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1152#issuecomment-748206874,https://api.github.com/repos/simonw/datasette/issues/1152,748206874,MDEyOklzc3VlQ29tbWVudDc0ODIwNjg3NA==,9599,simonw,2020-12-18T17:03:00Z,2020-12-22T23:58:04Z,OWNER,"Another permissions thought: what if ALL Datasette permissions were default-deny, and plugins could only grant permission to things, not block permission?

Right now a plugin can reply `False` to block, `True` to allow or `None` for ""I have no opinion on this, ask someone else"" - but even I'm confused by the interactions between block and allow and I implemented the system!

If everything in Datasette was default-deny then the user could use `--public-view` as an option when starting the server to default-allow view actions.

More importantly: plugins could return SQL statements that select a list of databases/tables the user is allowed access to. These could then be combined with `UNION` to create a full list of available resources.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747920515,https://api.github.com/repos/simonw/datasette/issues/1152,747920515,MDEyOklzc3VlQ29tbWVudDc0NzkyMDUxNQ==,9599,simonw,2020-12-18T07:29:21Z,2020-12-22T23:57:29Z,OWNER,Could I solve this using a configured canned query against the `_internal` tables with the actor's properties as inputs?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1099#issuecomment-749771231,https://api.github.com/repos/simonw/datasette/issues/1099,749771231,MDEyOklzc3VlQ29tbWVudDc0OTc3MTIzMQ==,9599,simonw,2020-12-22T20:54:25Z,2020-12-22T20:54:25Z,OWNER,"https://latest.datasette.io/_internal/foreign_keys (use https://latest.datasette.io/login-as-root first) is now a compound foreign key table:
```sql
CREATE TABLE foreign_keys (
        ""database_name"" TEXT,
        ""table_name"" TEXT,
        ""id"" INTEGER,
        ""seq"" INTEGER,
        ""table"" TEXT,
        ""from"" TEXT,
        ""to"" TEXT,
        ""on_update"" TEXT,
        ""on_delete"" TEXT,
        ""match"" TEXT,
        PRIMARY KEY (database_name, table_name, id, seq),
        FOREIGN KEY (database_name) REFERENCES databases(database_name),
        FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name)
    );
```
Currently the `database_name` column becomes a link (because it's a single foreign key) but the `table_name` one remains a non-link:

<img width=""1079"" alt=""_internal__foreign_keys__24_rows"" src=""https://user-images.githubusercontent.com/9599/102932110-87ba3080-4454-11eb-9ad5-b70a65129588.png"">

My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1152#issuecomment-749750995,https://api.github.com/repos/simonw/datasette/issues/1152,749750995,MDEyOklzc3VlQ29tbWVudDc0OTc1MDk5NQ==,9599,simonw,2020-12-22T20:05:30Z,2020-12-22T20:05:30Z,OWNER,"#1150 is landed now, which means there's a new, hidden `_internal` SQLite in-memory database containing all of the tables and databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/509#issuecomment-749749948,https://api.github.com/repos/simonw/datasette/issues/509,749749948,MDEyOklzc3VlQ29tbWVudDc0OTc0OTk0OA==,9599,simonw,2020-12-22T20:03:10Z,2020-12-22T20:03:10Z,OWNER,"If you open multiple files with the same filename, e.g. like this:

    datasette fixtures.db templates/fixtures.db plugins/fixtures.db

You'll now get this:

<img width=""702"" alt=""Datasette__fixtures__fixtures_2__fixtures_3"" src=""https://user-images.githubusercontent.com/9599/102928494-a4069f00-444d-11eb-9c1e-382a4e27bb51.png"">
","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",456568880,Support opening multiple databases with the same stem,
https://github.com/simonw/datasette/issues/509#issuecomment-749738241,https://api.github.com/repos/simonw/datasette/issues/509,749738241,MDEyOklzc3VlQ29tbWVudDc0OTczODI0MQ==,9599,simonw,2020-12-22T19:38:14Z,2020-12-22T19:38:14Z,OWNER,I'm fixing this in #1155.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456568880,Support opening multiple databases with the same stem,
https://github.com/simonw/datasette/issues/1155#issuecomment-749723557,https://api.github.com/repos/simonw/datasette/issues/1155,749723557,MDEyOklzc3VlQ29tbWVudDc0OTcyMzU1Nw==,9599,simonw,2020-12-22T19:08:27Z,2020-12-22T19:08:27Z,OWNER,"I'm going to have the `.add_database()` method select the name used in the path, de-duping against any existing names. It will then set database.name to that so that the database has access to its own name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748356492,https://api.github.com/repos/simonw/datasette/issues/1155,748356492,MDEyOklzc3VlQ29tbWVudDc0ODM1NjQ5Mg==,9599,simonw,2020-12-18T22:49:32Z,2020-12-22T01:13:05Z,OWNER,"There's some messy code that needs fixing here. The `datasette.databases` dictionary right now has a key that corresponds to the `/_internal` URL in the path, and a value that's a `Database()` object. BUT... the `Database()` object doesn't know what its key is.

While fixing this I should fix the issue where Datasette gets confused by multiple databases with the same stem: https://github.com/simonw/datasette/issues/509","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1157#issuecomment-749179460,https://api.github.com/repos/simonw/datasette/issues/1157,749179460,MDEyOklzc3VlQ29tbWVudDc0OTE3OTQ2MA==,9599,simonw,2020-12-21T20:24:19Z,2020-12-21T20:24:19Z,OWNER,"Three places to fix:

https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/tracer.py#L40-L42

https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/utils/__init__.py#L139-L152

https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/views/base.py#L460-L461","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",772438273,Use time.perf_counter() instead of time.time() to measure performance,
https://github.com/simonw/datasette/issues/1155#issuecomment-749176936,https://api.github.com/repos/simonw/datasette/issues/1155,749176936,MDEyOklzc3VlQ29tbWVudDc0OTE3NjkzNg==,9599,simonw,2020-12-21T20:18:15Z,2020-12-21T20:18:15Z,OWNER,"Fun query:
```sql
select table_name, group_concat(name, ', ') from columns group by database_name, table_name
```
https://latest.datasette.io/_internal?sql=select+table_name%2C+group_concat%28name%2C+%27%2C+%27%29+from+columns+group+by+database_name%2C+table_name","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-749170608,https://api.github.com/repos/simonw/datasette/issues/1155,749170608,MDEyOklzc3VlQ29tbWVudDc0OTE3MDYwOA==,9599,simonw,2020-12-21T20:01:47Z,2020-12-21T20:01:47Z,OWNER,I removed that `MEMORY` object() in dcdfb2c301341d45b66683e3e3be72f9c7585b2f,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1156#issuecomment-749158111,https://api.github.com/repos/simonw/datasette/issues/1156,749158111,MDEyOklzc3VlQ29tbWVudDc0OTE1ODExMQ==,9599,simonw,2020-12-21T19:33:45Z,2020-12-21T19:33:45Z,OWNER,"One reason for this change: it means I can use that database for more stuff. I've been thinking about moving metadata storage there for example, which fits a database called `_internal` but not one called `_schemas`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",772408750,Rename _schemas to _internal,
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426877,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426877,MDEyOklzc3VlQ29tbWVudDc0ODQyNjg3Nw==,9599,simonw,2020-12-19T06:16:11Z,2020-12-19T06:16:11Z,MEMBER,"Here's why:

    if ""fts5"" in str(e):

But the error being raised here is:

    sqlite3.OperationalError: no such column: to

I'm going to attempt the escaped on on every error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426663,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426663,MDEyOklzc3VlQ29tbWVudDc0ODQyNjY2Mw==,9599,simonw,2020-12-19T06:14:06Z,2020-12-19T06:14:06Z,MEMBER,Looks like I already do that here: https://github.com/dogsheep/dogsheep-beta/blob/9ba4401017ac24ffa3bc1db38e0910ea49de7616/dogsheep_beta/__init__.py#L141-L146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426581,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426581,MDEyOklzc3VlQ29tbWVudDc0ODQyNjU4MQ==,9599,simonw,2020-12-19T06:13:17Z,2020-12-19T06:13:17Z,MEMBER,"One fix for this could be to try running the raw query, but if it throws an error run it again with the query escaped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426501,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426501,MDEyOklzc3VlQ29tbWVudDc0ODQyNjUwMQ==,9599,simonw,2020-12-19T06:12:22Z,2020-12-19T06:12:22Z,MEMBER,I deliberately added support for advanced FTS in https://github.com/dogsheep/dogsheep-beta/commit/cbb2491b85d7ff416d6d429b60109e6c2d6d50b9 for #13 but that's the cause of this bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/simonw/datasette/issues/1155#issuecomment-748397998,https://api.github.com/repos/simonw/datasette/issues/1155,748397998,MDEyOklzc3VlQ29tbWVudDc0ODM5Nzk5OA==,9599,simonw,2020-12-19T01:28:18Z,2020-12-19T01:28:18Z,OWNER,"`datasette-graphql` returns an error due to this issue:

<img width=""878"" alt=""GraphiQL_and_fixtures__compound_three_primary_keys__1_001_rows"" src=""https://user-images.githubusercontent.com/9599/102677257-61e01380-4156-11eb-8f88-fe26b9c4a40b.png"">

On the console:
```
INFO:     127.0.0.1:63116 - ""POST /graphql/_internal HTTP/1.1"" 500 Internal Server Error
Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/promise/promise.py"", line 844, in handle_future_result
    resolve(future.result())
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/datasette_graphql/utils.py"", line 603, in resolve_table
    data, _, _ = await view.data(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 304, in data
    db = self.ds.databases[database]
graphql.error.located_error.GraphQLLocatedError: ':memory:c6dd5abe1a757a7de00d99b699175bd33d9a575f05b5751bf856b8656fb07edd'
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748368660,https://api.github.com/repos/simonw/datasette/issues/1155,748368660,MDEyOklzc3VlQ29tbWVudDc0ODM2ODY2MA==,9599,simonw,2020-12-18T23:18:04Z,2020-12-19T01:12:00Z,OWNER,"A `Database` should have a `.name` which is unique across the Datasette instance and is used in the URL. The `path` should be optional, only set for file databases. A new `.memory_name` property can be used for shared memory databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748368938,https://api.github.com/repos/simonw/datasette/issues/1155,748368938,MDEyOklzc3VlQ29tbWVudDc0ODM2ODkzOA==,9599,simonw,2020-12-18T23:19:04Z,2020-12-18T23:19:04Z,OWNER,`Database` internal class is documented here: https://docs.datasette.io/en/latest/internals.html#database-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748368384,https://api.github.com/repos/simonw/datasette/issues/1155,748368384,MDEyOklzc3VlQ29tbWVudDc0ODM2ODM4NA==,9599,simonw,2020-12-18T23:17:00Z,2020-12-18T23:17:00Z,OWNER,Here's the commit where I added it. https://github.com/simonw/datasette/commit/9743e1d91b5f0a2b3c1c0bd6ffce8739341f43c4 - I didn't yet have the `.add_database()` mechanism. Today the `MEMORY` object bit is no longer needed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748367922,https://api.github.com/repos/simonw/datasette/issues/1155,748367922,MDEyOklzc3VlQ29tbWVudDc0ODM2NzkyMg==,9599,simonw,2020-12-18T23:15:24Z,2020-12-18T23:15:24Z,OWNER,"The code for building up that `.databases` dictionary is a bit convoluted. Here's the code that adds a `:memory:` database if the user specified `--memory` OR if there are no files to be attached:

https://github.com/simonw/datasette/blob/ebc7aa287c99fe6114b79aeab8efb8d4489a6182/datasette/app.py#L221-L241

I'm not sure why I wrote it this way, instead of just calling `.add_database("":memory:"", Database(..., is_memory=True)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/509#issuecomment-748356637,https://api.github.com/repos/simonw/datasette/issues/509,748356637,MDEyOklzc3VlQ29tbWVudDc0ODM1NjYzNw==,9599,simonw,2020-12-18T22:50:03Z,2020-12-18T22:50:03Z,OWNER,Related problem caused by the new `_schemas` database - if a user attempts to open their own `_schemas.db` file it will fail. I'd like to open and mount that as `/_schemas_` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456568880,Support opening multiple databases with the same stem,
https://github.com/simonw/datasette/issues/1150#issuecomment-748354841,https://api.github.com/repos/simonw/datasette/issues/1150,748354841,MDEyOklzc3VlQ29tbWVudDc0ODM1NDg0MQ==,9599,simonw,2020-12-18T22:43:49Z,2020-12-18T22:43:49Z,OWNER,"For a demo, visit https://latest.datasette.io/login-as-root and then hit https://latest.datasette.io/_schemas","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-748352106,https://api.github.com/repos/simonw/datasette/issues/1150,748352106,MDEyOklzc3VlQ29tbWVudDc0ODM1MjEwNg==,9599,simonw,2020-12-18T22:34:40Z,2020-12-18T22:34:40Z,OWNER,"Needs documentation, but I can wait to write that until I've tested out the feature a bit more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-748351350,https://api.github.com/repos/simonw/datasette/issues/1150,748351350,MDEyOklzc3VlQ29tbWVudDc0ODM1MTM1MA==,9599,simonw,2020-12-18T22:32:13Z,2020-12-18T22:32:13Z,OWNER,Getting all the tests to pass is tricky because this adds a whole extra database to Datasette - and there's various code that loops through `ds.databases` as part of the tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-748260875,https://api.github.com/repos/simonw/datasette/issues/1150,748260875,MDEyOklzc3VlQ29tbWVudDc0ODI2MDg3NQ==,9599,simonw,2020-12-18T18:55:12Z,2020-12-18T18:55:12Z,OWNER,"I'm going to move the code into a `utils/schemas.py` module, to avoid further extending the `Datasette` class definition and to make it more easily testable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-748260118,https://api.github.com/repos/simonw/datasette/issues/1150,748260118,MDEyOklzc3VlQ29tbWVudDc0ODI2MDExOA==,9599,simonw,2020-12-18T18:54:12Z,2020-12-18T18:54:12Z,OWNER,"I'm going to tidy this up and land it. A couple of additional decisions:

- The database will be called `/_schemas`
- By default it will only be visible to `root` - thus avoiding having to solve the permissions problem with regards to users seeing schemas for tables that are otherwise invisible to them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/493#issuecomment-747966232,https://api.github.com/repos/simonw/datasette/issues/493,747966232,MDEyOklzc3VlQ29tbWVudDc0Nzk2NjIzMg==,9599,simonw,2020-12-18T09:19:41Z,2020-12-18T09:19:41Z,OWNER,Is there any reason to keep `--setting` rather than moving those items into a `configure.json` file with all the configuration options that currently live in `metadata.json`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/issues/1152#issuecomment-747921195,https://api.github.com/repos/simonw/datasette/issues/1152,747921195,MDEyOklzc3VlQ29tbWVudDc0NzkyMTE5NQ==,9599,simonw,2020-12-18T07:31:25Z,2020-12-18T07:31:25Z,OWNER,It's also a really good fit for the new mechanism that's coming together in #1150.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747920852,https://api.github.com/repos/simonw/datasette/issues/1152,747920852,MDEyOklzc3VlQ29tbWVudDc0NzkyMDg1Mg==,9599,simonw,2020-12-18T07:30:22Z,2020-12-18T07:30:22Z,OWNER,Redefining all Datasette permissions in terms of SQL queries that return the set of databases and tables that the user is allowed to interact with does feel VERY Datasette-y.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747920087,https://api.github.com/repos/simonw/datasette/issues/1152,747920087,MDEyOklzc3VlQ29tbWVudDc0NzkyMDA4Nw==,9599,simonw,2020-12-18T07:27:58Z,2020-12-18T07:28:30Z,OWNER,"I want to keep the existing `metadata.json` ""allow"" blocks mechanism working. Note that if you have 1,000 tables and a permissions policy you won't be using ""allow"" blocks, you'll be using a more sophisticated permissions plugin instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747919782,https://api.github.com/repos/simonw/datasette/issues/1152,747919782,MDEyOklzc3VlQ29tbWVudDc0NzkxOTc4Mg==,9599,simonw,2020-12-18T07:27:01Z,2020-12-18T07:27:01Z,OWNER,"Perhaps this can be solved by keeping the existing plugin hooks and adding new, optional ones for bulk lookups.

If your plugin doesn't implement the bulk lookup hooks Datasette will do an inefficient loop through everything checking permissions on each one.

If you DO implement it you can speed things up dramatically.

Not sure if this would solve the homepage problem though, where you might need to run 1,000 table permission checks. That's more a case where you want to think in terms of a SQL where clause.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1150#issuecomment-747893704,https://api.github.com/repos/simonw/datasette/issues/1150,747893704,MDEyOklzc3VlQ29tbWVudDc0Nzg5MzcwNA==,9599,simonw,2020-12-18T06:19:13Z,2020-12-18T06:19:13Z,OWNER,I'm not going to block this issue on permissions - I will tackle the efficient bulk permissions problem in #1152.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1152#issuecomment-747893423,https://api.github.com/repos/simonw/datasette/issues/1152,747893423,MDEyOklzc3VlQ29tbWVudDc0Nzg5MzQyMw==,9599,simonw,2020-12-18T06:18:24Z,2020-12-18T06:18:24Z,OWNER,"What would Datasette's permission hooks look like if they all dealt with sets of items rather than individual items? So plugins could return a set of items that the user has permission to access, or even a WHERE clause?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747892731,https://api.github.com/repos/simonw/datasette/issues/1152,747892731,MDEyOklzc3VlQ29tbWVudDc0Nzg5MjczMQ==,9599,simonw,2020-12-18T06:16:29Z,2020-12-18T06:16:29Z,OWNER,"One enormous advantage I have is that after #1150 I will have a database table full of databases and tables that I can execute queries against.

This means I could calculate visible tables using SQL where clauses, which should be easily fast enough even against ten thousand plus tables.

The catch is the permissions hooks. Since I haven't hit Datasette 1.0 yet maybe I should redesign those hooks to work against the new in-memory database schema stuff?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747891854,https://api.github.com/repos/simonw/datasette/issues/1152,747891854,MDEyOklzc3VlQ29tbWVudDc0Nzg5MTg1NA==,9599,simonw,2020-12-18T06:14:09Z,2020-12-18T06:14:15Z,OWNER,"This is a classic challenge in permissions systems. If I want Datasette to be able to handle thousands of tables I need a reasonable solution for it.

Twitter conversation: https://twitter.com/simonw/status/1339791768842248192","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1150#issuecomment-747864831,https://api.github.com/repos/simonw/datasette/issues/1150,747864831,MDEyOklzc3VlQ29tbWVudDc0Nzg2NDgzMQ==,9599,simonw,2020-12-18T04:46:18Z,2020-12-18T04:46:18Z,OWNER,"The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75

A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747864080,https://api.github.com/repos/simonw/datasette/issues/1150,747864080,MDEyOklzc3VlQ29tbWVudDc0Nzg2NDA4MA==,9599,simonw,2020-12-18T04:43:29Z,2020-12-18T04:43:29Z,OWNER,"I may be overthinking that problem. Many queries are fast in SQLite. If a Datasette instance has 1,000 connected tables will even that be a performance problem for permission checks? I should benchmark to find out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747862001,https://api.github.com/repos/simonw/datasette/issues/1150,747862001,MDEyOklzc3VlQ29tbWVudDc0Nzg2MjAwMQ==,9599,simonw,2020-12-18T04:35:34Z,2020-12-18T04:35:34Z,OWNER,"I do need to solve the permissions problem properly though, because one of the goals of this system is to provide a paginated, searchable list of databases and tables for the homepage of the instance - #991.

As such, the homepage will need to be able to display only the tables and databases that the user has permission to view.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747861556,https://api.github.com/repos/simonw/datasette/issues/1150,747861556,MDEyOklzc3VlQ29tbWVudDc0Nzg2MTU1Ng==,9599,simonw,2020-12-18T04:33:45Z,2020-12-18T04:33:45Z,OWNER,"One solution on permissions: if Datasette had an efficient way of saying ""list the tables that this user has access to"" I could use that as a filter any time the user views the schema information. The implementation could be tricky though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747861357,https://api.github.com/repos/simonw/datasette/issues/1150,747861357,MDEyOklzc3VlQ29tbWVudDc0Nzg2MTM1Nw==,9599,simonw,2020-12-18T04:32:52Z,2020-12-18T04:32:52Z,OWNER,"I need to figure out how this will interact with Datasette permissions.

If some tables are private, but others are public, should users be able to see the private tables listed in the schema metadata?

If not, how can that mechanism work?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747833639,https://api.github.com/repos/simonw/datasette/issues/1150,747833639,MDEyOklzc3VlQ29tbWVudDc0NzgzMzYzOQ==,9599,simonw,2020-12-18T02:49:40Z,2020-12-18T03:52:12Z,OWNER,"I'm going to use five tables to start off with:

- `databases` - a list of databases. Each one has a `name`, `path` (if it's on disk), `is_memory`, `schema_version`
- `tables` - a list of tables. Each row is `database_name`, `table_name`, `sql` (the create table statement) - may add more tables in the future, in particular maybe a `last_row_count` to cache results of counting the rows.
- `columns` - a list of columns. It's the output of `pragma_table_xinfo` with the `database_name` and `table_name` columns added at the beginning.
- `foreign_keys` - a list of foreign keys - `pragma_foreign_key_list` output plus `database_name` and `table_name`.
- `indexes` - a list of indexes - `pragma_table_xinfo` output plus `database_name` and `table_name`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747847405,https://api.github.com/repos/simonw/datasette/issues/1150,747847405,MDEyOklzc3VlQ29tbWVudDc0Nzg0NzQwNQ==,9599,simonw,2020-12-18T03:36:04Z,2020-12-18T03:36:04Z,OWNER,I could have another table that stores the combined rows from `sqlite_máster` on every connected database so I have a copy of the schema SQL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747847180,https://api.github.com/repos/simonw/datasette/issues/1150,747847180,MDEyOklzc3VlQ29tbWVudDc0Nzg0NzE4MA==,9599,simonw,2020-12-18T03:35:15Z,2020-12-18T03:35:15Z,OWNER,"Simpler implementation idea: a Datasette method `.refresh_schemas()` which loops through all known databases, checks their schema version and updates the in-memory schemas database if they have changed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747834762,https://api.github.com/repos/simonw/datasette/issues/1150,747834762,MDEyOklzc3VlQ29tbWVudDc0NzgzNDc2Mg==,9599,simonw,2020-12-18T02:53:22Z,2020-12-18T02:53:22Z,OWNER,I think I'm going to have to build this without using the `pragma_x()` SQL functions as they were only added in 3.16 in 2017-01-02 and I've seen plenty of Datasette instances running on older versions of SQLite.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747834462,https://api.github.com/repos/simonw/datasette/issues/1150,747834462,MDEyOklzc3VlQ29tbWVudDc0NzgzNDQ2Mg==,9599,simonw,2020-12-18T02:52:19Z,2020-12-18T02:52:26Z,OWNER,Maintaining this database will be the responsibility of a subclass of `Database` called `_SchemaDatabase` which will be managed by the `Datasette` instance.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747834113,https://api.github.com/repos/simonw/datasette/issues/1150,747834113,MDEyOklzc3VlQ29tbWVudDc0NzgzNDExMw==,9599,simonw,2020-12-18T02:51:13Z,2020-12-18T02:51:20Z,OWNER,"SQLite uses `indexes` rather than `indices` as the plural, so I'll go with that: https://sqlite.org/lang_createindex.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747809670,https://api.github.com/repos/simonw/datasette/issues/1150,747809670,MDEyOklzc3VlQ29tbWVudDc0NzgwOTY3MA==,9599,simonw,2020-12-18T01:29:30Z,2020-12-18T01:29:30Z,OWNER,I've been rediscovering the pattern I already documented in this TIL: https://github.com/simonw/til/blob/main/sqlite/list-all-columns-in-a-database.md#better-alternative-using-a-join,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747807891,https://api.github.com/repos/simonw/datasette/issues/1150,747807891,MDEyOklzc3VlQ29tbWVudDc0NzgwNzg5MQ==,9599,simonw,2020-12-18T01:23:59Z,2020-12-18T01:23:59Z,OWNER,"https://www.sqlite.org/pragma.html#pragfunc says:
>  * This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. 
>  * The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747807289,https://api.github.com/repos/simonw/datasette/issues/1150,747807289,MDEyOklzc3VlQ29tbWVudDc0NzgwNzI4OQ==,9599,simonw,2020-12-18T01:22:05Z,2020-12-18T01:22:05Z,OWNER,"Here's a simpler query pattern (not using CTEs so should work on older versions of SQLite) - this one lists all indexes for all tables:
```sql
select
  sqlite_master.name as 'table',
  indexes.*
from
  sqlite_master
  join pragma_index_list(sqlite_master.name) indexes
where
  sqlite_master.type = 'table'
```
https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+%27table%27%2C%0D%0A++indexes.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_index_list%28sqlite_master.name%29+indexes%0D%0Awhere%0D%0A++sqlite_master.type+%3D+%27table%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747805275,https://api.github.com/repos/simonw/datasette/issues/1150,747805275,MDEyOklzc3VlQ29tbWVudDc0NzgwNTI3NQ==,9599,simonw,2020-12-18T01:15:27Z,2020-12-18T01:16:17Z,OWNER,"This query uses a join to pull foreign key information for every table: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_foreign_key_list%28tables.name%29+foo

```sql
with tables as (
  select
    name
  from
    sqlite_master
  where
    type = 'table'
)
select
  tables.name as 'table',
  foo.*
from
  tables
  join pragma_foreign_key_list(tables.name) foo
```

Same query for `pragma_table_xinfo`: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_table_xinfo%28tables.name%29+foo","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747804254,https://api.github.com/repos/simonw/datasette/issues/1150,747804254,MDEyOklzc3VlQ29tbWVudDc0NzgwNDI1NA==,9599,simonw,2020-12-18T01:12:13Z,2020-12-18T01:12:13Z,OWNER,"Prototype: https://latest.datasette.io/fixtures?sql=select+%27facetable%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27facetable%27%29%0D%0Aunion%0D%0Aselect+%27searchable%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27searchable%27%29%0D%0Aunion%0D%0Aselect+%27compound_three_primary_keys%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27compound_three_primary_keys%27%29

```sql
select 'facetable' as 'table', * from pragma_table_xinfo('facetable')
union
select 'searchable' as 'table', * from pragma_table_xinfo('searchable')
union
select 'compound_three_primary_keys' as 'table', * from pragma_table_xinfo('compound_three_primary_keys')
```


table | cid | name | type | notnull | dflt_value | pk | hidden
-- | -- | -- | -- | -- | -- | -- | --
compound_three_primary_keys | 0 | pk1 | varchar(30) | 0 |   | 1 | 0
compound_three_primary_keys | 1 | pk2 | varchar(30) | 0 |   | 2 | 0
compound_three_primary_keys | 2 | pk3 | varchar(30) | 0 |   | 3 | 0
compound_three_primary_keys | 3 | content | text | 0 |   | 0 | 0
facetable | 0 | pk | integer | 0 |   | 1 | 0
facetable | 1 | created | text | 0 |   | 0 | 0
facetable | 2 | planet_int | integer | 0 |   | 0 | 0
facetable | 3 | on_earth | integer | 0 |   | 0 | 0
facetable | 4 | state | text | 0 |   | 0 | 0
facetable | 5 | city_id | integer | 0 |   | 0 | 0
facetable | 6 | neighborhood | text | 0 |   | 0 | 0
facetable | 7 | tags | text | 0 |   | 0 | 0
facetable | 8 | complex_array | text | 0 |   | 0 | 0
facetable | 9 | distinct_some_null |   | 0 |   | 0 | 0
searchable | 0 | pk | integer | 0 |   | 1 | 0
searchable | 1 | text1 | text | 0 |   | 0 | 0
searchable | 2 | text2 | text | 0 |   | 0 | 0
searchable | 3 | name with . and spaces | text | 0 |   | 0 | 0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747803268,https://api.github.com/repos/simonw/datasette/issues/1150,747803268,MDEyOklzc3VlQ29tbWVudDc0NzgwMzI2OA==,9599,simonw,2020-12-18T01:08:40Z,2020-12-18T01:08:40Z,OWNER,"Next step: design a schema for the in-memory database table that exposes all of the tables.

I want to support things like:

- Show me all of the tables
- Show me the columns in a table
- Show me all tables that contain a `tags` column
- Show me the indexes
- Show me every table configured for full-text search

Maybe a starting point would be to build concrete tables using the results of things like `PRAGMA foreign_key_list(table)` and `PRAGMA table_xinfo(table)` - note though that `table_xinfo` is SQLite 3.26.0 or higher, as shown here: https://github.com/simonw/datasette/blob/5e9895c67f08e9f42acedd3d6d29512ac446e15f/datasette/utils/__init__.py#L563-L579","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1151#issuecomment-747801751,https://api.github.com/repos/simonw/datasette/issues/1151,747801751,MDEyOklzc3VlQ29tbWVudDc0NzgwMTc1MQ==,9599,simonw,2020-12-18T01:03:39Z,2020-12-18T01:03:39Z,OWNER,"This feature is illustrated by the tests: https://github.com/simonw/datasette/blob/5e9895c67f08e9f42acedd3d6d29512ac446e15f/tests/test_internals_database.py#L469-L496

I added new documentation for the `Datasette()` constructor here as well: https://docs.datasette.io/en/latest/internals.html#database-ds-path-none-is-mutable-false-is-memory-false-memory-name-none","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747801084,https://api.github.com/repos/simonw/datasette/issues/1151,747801084,MDEyOklzc3VlQ29tbWVudDc0NzgwMTA4NA==,9599,simonw,2020-12-18T01:01:26Z,2020-12-18T01:01:26Z,OWNER,"I tested this with a one-off plugin and it worked!

```python
from datasette import hookimpl
from datasette.database import Database


@hookimpl
def startup(datasette):
    datasette.add_database(""statistics"", Database(
        datasette,
        memory_name=""statistics""
    ))
```
This created a `/statistics` database when I ran `datasette` - and if I installed https://github.com/simonw/datasette-write I could then create tables in it which persisted until I restarted the server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747784199,https://api.github.com/repos/simonw/datasette/issues/1151,747784199,MDEyOklzc3VlQ29tbWVudDc0Nzc4NDE5OQ==,9599,simonw,2020-12-18T00:09:36Z,2020-12-18T00:09:36Z,OWNER,"Is it possible to connect to a memory database in read-only mode?

`file:foo?mode=memory&cache=shared&mode=ro` isn't valid because it features `mode=` more than once.

https://stackoverflow.com/a/40548682 suggests using `PRAGMA query_only` on the connection instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747775245,https://api.github.com/repos/simonw/datasette/issues/1151,747775245,MDEyOklzc3VlQ29tbWVudDc0Nzc3NTI0NQ==,9599,simonw,2020-12-17T23:43:41Z,2020-12-17T23:56:27Z,OWNER,"I'm going to add an argument to the `Database()` constructor which means ""connect to named in-memory database called X"".

```python
db = Database(ds, memory_name=""datasette"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747779056,https://api.github.com/repos/simonw/datasette/issues/1151,747779056,MDEyOklzc3VlQ29tbWVudDc0Nzc3OTA1Ng==,9599,simonw,2020-12-17T23:55:57Z,2020-12-17T23:55:57Z,OWNER,Wait I do use it - if you run `datasette --memory` - which is useful for trying things out in SQL that doesn't need to run against a table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747775792,https://api.github.com/repos/simonw/datasette/issues/1151,747775792,MDEyOklzc3VlQ29tbWVudDc0Nzc3NTc5Mg==,9599,simonw,2020-12-17T23:45:20Z,2020-12-17T23:45:20Z,OWNER,"Do I use the current `is_memory=` boolean anywhere at the moment?

https://ripgrep.datasette.io/-/ripgrep?pattern=is_memory - doesn't look like it.

I may remove that feature, since it's not actually useful, and replace it with a mechanism for creating shared named memory databases instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747774855,https://api.github.com/repos/simonw/datasette/issues/1151,747774855,MDEyOklzc3VlQ29tbWVudDc0Nzc3NDg1NQ==,9599,simonw,2020-12-17T23:42:34Z,2020-12-17T23:42:34Z,OWNER,"This worked as a prototype:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 412e0c5..a90e617 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -24,11 +24,12 @@ connections = threading.local()
 
 
 class Database:
-    def __init__(self, ds, path=None, is_mutable=False, is_memory=False):
+    def __init__(self, ds, path=None, is_mutable=False, is_memory=False, uri=None):
         self.ds = ds
         self.path = path
         self.is_mutable = is_mutable
         self.is_memory = is_memory
+        self.uri = uri
         self.hash = None
         self.cached_size = None
         self.cached_table_counts = None
@@ -46,6 +47,8 @@ class Database:
                 }
 
     def connect(self, write=False):
+        if self.uri:
+            return sqlite3.connect(self.uri, uri=True, check_same_thread=False)
         if self.is_memory:
             return sqlite3.connect("":memory:"")
         # mode=ro or immutable=1?
```
Then in `ipython`:
```
from datasette.app import Datasette
from datasette.database import Database
ds = Datasette([])
db = Database(ds, uri=""file:datasette?mode=memory&cache=shared"", is_memory=True)
await db.execute_write(""create table foo (bar text)"")
await db.table_names()
# Outputs [""foo""]
db2 = Database(ds, uri=""file:datasette?mode=memory&cache=shared"", is_memory=True)
await db2.table_names()
# Also outputs [""foo""]
```

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747770581,https://api.github.com/repos/simonw/datasette/issues/1151,747770581,MDEyOklzc3VlQ29tbWVudDc0Nzc3MDU4MQ==,9599,simonw,2020-12-17T23:31:18Z,2020-12-17T23:32:07Z,OWNER,"This works in `ipython`:
```
In [1]: import sqlite3

In [2]: c1 = sqlite3.connect(""file:datasette?mode=memory&cache=shared"", uri=True)

In [3]: c2 = sqlite3.connect(""file:datasette?mode=memory&cache=shared"", uri=True)

In [4]: c1.executescript(""CREATE TABLE hello (world TEXT)"")
Out[4]: <sqlite3.Cursor at 0x1104addc0>

In [5]: c1.execute(""select * from sqlite_master"").fetchall()
Out[5]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

In [6]: c2.execute(""select * from sqlite_master"").fetchall()
Out[6]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

In [7]: c3 = sqlite3.connect(""file:datasette?mode=memory&cache=shared"", uri=True)

In [9]: c3.execute(""select * from sqlite_master"").fetchall()
Out[9]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

In [10]: c4 = sqlite3.connect(""file:datasette?mode=memory"", uri=True)

In [11]: c4.execute(""select * from sqlite_master"").fetchall()
Out[11]: []
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747770082,https://api.github.com/repos/simonw/datasette/issues/1151,747770082,MDEyOklzc3VlQ29tbWVudDc0Nzc3MDA4Mg==,9599,simonw,2020-12-17T23:29:53Z,2020-12-17T23:29:53Z,OWNER,I'm going to try with `file:datasette?mode=memory&cache=shared`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747769830,https://api.github.com/repos/simonw/datasette/issues/1151,747769830,MDEyOklzc3VlQ29tbWVudDc0Nzc2OTgzMA==,9599,simonw,2020-12-17T23:29:08Z,2020-12-17T23:29:08Z,OWNER,"https://sqlite.org/inmemorydb.html

>  The database ceases to exist as soon as the database connection is closed. Every :memory: database is distinct from every other. So, opening two database connections each with the filename "":memory:"" will create two independent in-memory databases.
>
> [...]
>
> The special `"":memory:""` filename also works when using URI filenames. For example:
>
>      rc = sqlite3_open(""file::memory:"", &db);
>
> [...]
>
>  However, the same in-memory database can be opened by two or more database connections as follows:
>
>      rc = sqlite3_open(""file::memory:?cache=shared"", &db);
>
> [...]
> If two or more distinct but shareable in-memory databases are needed in a single process, then the mode=memory query parameter can be used with a URI filename to create a named in-memory database:
>
>     rc = sqlite3_open(""file:memdb1?mode=memory&cache=shared"", &db);
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1150#issuecomment-747768112,https://api.github.com/repos/simonw/datasette/issues/1150,747768112,MDEyOklzc3VlQ29tbWVudDc0Nzc2ODExMg==,9599,simonw,2020-12-17T23:25:21Z,2020-12-17T23:25:21Z,OWNER,"Next challenge: figure out how to use the `Database` class from https://github.com/simonw/datasette/blob/0.53/datasette/database.py for an in-memory database which persists data for the duration of the lifetime of the server, and allows access to that in-memory database from multiple threads in a way that lets them see each other's changes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747767598,https://api.github.com/repos/simonw/datasette/issues/1150,747767598,MDEyOklzc3VlQ29tbWVudDc0Nzc2NzU5OA==,9599,simonw,2020-12-17T23:24:03Z,2020-12-17T23:24:03Z,OWNER,"I'm going to assume that even the heaviest user will have trouble going beyond a few hundred database files, so this is fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747767499,https://api.github.com/repos/simonw/datasette/issues/1150,747767499,MDEyOklzc3VlQ29tbWVudDc0Nzc2NzQ5OQ==,9599,simonw,2020-12-17T23:23:44Z,2020-12-17T23:23:44Z,OWNER,Grabbing the schema version of 380 files in the root directory takes 70ms.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747767055,https://api.github.com/repos/simonw/datasette/issues/1150,747767055,MDEyOklzc3VlQ29tbWVudDc0Nzc2NzA1NQ==,9599,simonw,2020-12-17T23:22:41Z,2020-12-17T23:22:41Z,OWNER,"It's just recursion that's expensive. I created 380 empty SQLite databases in a folder and timed `list(pathlib.Path(""/tmp"").glob(""*.db""));` and it took 0.002s.

So maybe I tell users that all SQLite databases have to be in the root folder.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747766310,https://api.github.com/repos/simonw/datasette/issues/1150,747766310,MDEyOklzc3VlQ29tbWVudDc0Nzc2NjMxMA==,9599,simonw,2020-12-17T23:20:49Z,2020-12-17T23:20:49Z,OWNER,"I tried against my entire `~/Development/Dropbox` folder - deeply nested with 381 SQLite database files in sub-folders - and it took 25s! But it turned out 23.9s of that was the call to `pathlib.Path(""/Users/simon/Dropbox/Development"").glob('**/*.db')`.

So it looks like connecting to a SQLite database file and getting the schema version is extremely fast. Scanning directories is slower.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747764712,https://api.github.com/repos/simonw/datasette/issues/1150,747764712,MDEyOklzc3VlQ29tbWVudDc0Nzc2NDcxMg==,9599,simonw,2020-12-17T23:16:31Z,2020-12-17T23:16:31Z,OWNER,"Quick micro-benchmark, run against a folder with 46 database files adding up to 1.4GB total:
```python
import pathlib, sqlite3, time

paths = list(pathlib.Path(""."").glob('*.db'))

def schema_version(path):
    db = sqlite3.connect(path)
    version = db.execute(""PRAGMA schema_version"").fetchall()[0]
    db.close()
    return version

def all():
    versions = {}
    for path in paths:
        versions[path.name] = schema_version(path)
    return versions

start = time.time(); all(); print(time.time() - start)
# 0.012346982955932617
```
So that's 12ms.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747754229,https://api.github.com/repos/simonw/datasette/issues/1150,747754229,MDEyOklzc3VlQ29tbWVudDc0Nzc1NDIyOQ==,9599,simonw,2020-12-17T23:04:38Z,2020-12-17T23:04:38Z,OWNER,"Open question: will this work for hundreds of database files, or is the overhead of connecting to each of 100 databases in turn to run `PRAGMA schema_version` too high?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747754082,https://api.github.com/repos/simonw/datasette/issues/1150,747754082,MDEyOklzc3VlQ29tbWVudDc0Nzc1NDA4Mg==,9599,simonw,2020-12-17T23:04:13Z,2020-12-17T23:04:13Z,OWNER,"Pages that need a list of all databases - the index page and /-/databases for example - could trigger a ""check for new directories in the configured directories"" scan.

That scan would run at most once every 5 (n) seconds - the check is triggered if it’s run more recently than that it doesn’t run.

Hopefully this means it could be done as a blocking operation, rather than trying to run it in a thread.

When it runs it scans for *.db or *.sqlite files (maybe one or two other extensions) that it hasn’t seen before. It also checks that the existing list of known database files still exists.

If it finds any new ones it connects to them once to run `.schema`. It also runs `PRAGMA schema_version` on each known database so that it can compare the schema version number to the last one it saw. That's how it detects if there are new tables or if the cached schema needs to be updated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/461#issuecomment-747734273,https://api.github.com/repos/simonw/datasette/issues/461,747734273,MDEyOklzc3VlQ29tbWVudDc0NzczNDI3Mw==,9599,simonw,2020-12-17T22:14:46Z,2020-12-17T22:14:46Z,OWNER,"I've been thinking about this a bunch. For Datasette to be useful as a private repository of data (Datasette Library, #417) it's crucial that it can handle a much, much larger number of databases.

This makes me worry about how many connections (and open file handles) it makes sense to have open at one time.

I realize now that this is much less of a problem for private instances. Public instances on the internet could get traffic to any database at any time, so connections could easily get out of control. A private instance with only a few users could instead get away with only opening connections to databases in ""active use"".

This does however make it even more important for Datasette to maintain a cached set of metadata about the tables - which is also needed to power this feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",443021509,Paginate + search for databases/tables on the homepage,
https://github.com/simonw/datasette/issues/1005#issuecomment-747209115,https://api.github.com/repos/simonw/datasette/issues/1005,747209115,MDEyOklzc3VlQ29tbWVudDc0NzIwOTExNQ==,9599,simonw,2020-12-17T05:11:04Z,2020-12-17T05:11:04Z,OWNER,Tracking ticket for the next HTTPX release is https://github.com/encode/httpx/pull/1403,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718259202,Remove xfail tests when new httpx is released,
https://github.com/simonw/datasette/issues/741#issuecomment-747208543,https://api.github.com/repos/simonw/datasette/issues/741,747208543,MDEyOklzc3VlQ29tbWVudDc0NzIwODU0Mw==,9599,simonw,2020-12-17T05:09:03Z,2020-12-17T05:09:03Z,OWNER,I really like this in `datasette-publish-vercel` - I'm definitely going to bring this to the other publish implementations as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/1149#issuecomment-747207787,https://api.github.com/repos/simonw/datasette/issues/1149,747207787,MDEyOklzc3VlQ29tbWVudDc0NzIwNzc4Nw==,9599,simonw,2020-12-17T05:06:16Z,2020-12-17T05:06:16Z,OWNER,"So, an idea: what if Datasette's default CSS applied only to elements with classes - or maybe to childen of a `body class=""datasette""` element? In such a way that you could write your own custom HTML that reused elements of Datasette's CSS - the cog menu styling for example - but only on an opt-in basis?","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769520939,Make it easier to theme Datasette with CSS,
https://github.com/simonw/datasette/issues/1149#issuecomment-747207487,https://api.github.com/repos/simonw/datasette/issues/1149,747207487,MDEyOklzc3VlQ29tbWVudDc0NzIwNzQ4Nw==,9599,simonw,2020-12-17T05:05:08Z,2020-12-17T05:05:08Z,OWNER,"I think what I want is for it to be easy to reuse portions of Datasette's CSS - the bit that styles the cog menu for example - without pulling in the whole thing.

I tried linking in the `<link rel=""stylesheet"" href=""/-/static/app.css"">` stylesheet and the page broke, wildly:

<img width=""1030"" alt=""content__categories__3_rows_and_Datasette_search__pytest"" src=""https://user-images.githubusercontent.com/9599/102446222-2290b600-3fe2-11eb-9d3a-e32ca72d1dac.png"">

That's because Datasette's [built-in CSS](https://github.com/simonw/datasette/blob/0.53/datasette/static/app.css) applies styles directly to a whole bunch of different tags - `body`, `header`, `footer` etc - which means that if you import that stylesheet it can play havoc with the site you have already built.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769520939,Make it easier to theme Datasette with CSS,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747126777,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2,747126777,MDEyOklzc3VlQ29tbWVudDc0NzEyNjc3Nw==,9599,simonw,2020-12-17T00:36:52Z,2020-12-17T00:36:52Z,MEMBER,The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769376447,killed by oomkiller on large location-history,
https://github.com/simonw/datasette/issues/675#issuecomment-747070709,https://api.github.com/repos/simonw/datasette/issues/675,747070709,MDEyOklzc3VlQ29tbWVudDc0NzA3MDcwOQ==,9599,simonw,2020-12-16T22:09:15Z,2020-12-16T22:09:15Z,OWNER,"The other way this could work is passing a single argument - the file (or directory) to be copied in - and assuming it should always go in the `/app` root. Something like:

    datasette publish cloudrun my.db --include src/ --include dogsheep-beta.yml

Which would add `/app/src/...` and `/app/dogsheep-beta.yml`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/675#issuecomment-747068624,https://api.github.com/repos/simonw/datasette/issues/675,747068624,MDEyOklzc3VlQ29tbWVudDc0NzA2ODYyNA==,9599,simonw,2020-12-16T22:04:42Z,2020-12-16T22:04:42Z,OWNER,"I can't just use `COPY /path/to/blah.yml /app` in the `Dockerfile` because it runs on the Google Cloud Build servers, not on the user's laptop - so I need to first copy the files they specify to that temporary directory that gets uploaded to the cloud, then rewrite the `COPY` lines in the `Dockerfile` to copy from there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/675#issuecomment-747067864,https://api.github.com/repos/simonw/datasette/issues/675,747067864,MDEyOklzc3VlQ29tbWVudDc0NzA2Nzg2NA==,9599,simonw,2020-12-16T22:02:55Z,2020-12-16T22:02:55Z,OWNER,"But since we're already running `COPY . /app` anything that's made it into the temporary directory will get copied into `/app`.

But... I feel the usability of the command will be better if users can use absolute paths on the `target` side:

    datasette publish cloudrun my.db --cp dogsheep-beta.yml /app","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/675#issuecomment-747066629,https://api.github.com/repos/simonw/datasette/issues/675,747066629,MDEyOklzc3VlQ29tbWVudDc0NzA2NjYyOQ==,9599,simonw,2020-12-16T21:59:58Z,2020-12-16T22:00:48Z,OWNER,"Note that `datasette publish cloudrun` uses a working directory of `/app` - so users will need to copy their files into `/app` if that's where they need to live.

https://github.com/simonw/datasette/blob/17cbbb1f7f230b39650afac62dd16476626001b5/datasette/utils/__init__.py#L348-L357","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/1148#issuecomment-747065487,https://api.github.com/repos/simonw/datasette/issues/1148,747065487,MDEyOklzc3VlQ29tbWVudDc0NzA2NTQ4Nw==,9599,simonw,2020-12-16T21:57:29Z,2020-12-16T21:57:29Z,OWNER,I filed a new public bug in their issue tracker here: https://github.com/vercel/vercel/issues/5575,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",767561886,Syntax error with + symbol when deployed to Vercel,
https://github.com/simonw/datasette/issues/1148#issuecomment-747062909,https://api.github.com/repos/simonw/datasette/issues/1148,747062909,MDEyOklzc3VlQ29tbWVudDc0NzA2MjkwOQ==,9599,simonw,2020-12-16T21:51:54Z,2020-12-16T21:51:54Z,OWNER,"This is a really frustrating bug with Vercel: https://github.com/simonw/datasette-publish-vercel/issues/28

`+` characters in URLs get translated into spaces before they get to Datasette. They know about the bug and said they were working on a fix a few months ago, but looks like it's still a problem.

A workaround is to avoid `+` and use `-` instead - I think this SQL query does the same thing as yours:

https://aws-partners-singapore.vercel.app/partners?sql=select%0D%0A++A.launch_rank%2C%0D%0A++A.partner_info%0D%0Afrom%0D%0A++summary+A%0D%0A++INNER+JOIN+summary+B+ON+A.launch_rank+%3E%3D+B.launch_rank+-+3%0D%0A++AND+A.launch_rank+-4+%3C%3D+B.launch_rank%0D%0AWHERE%0D%0A++B.%22partner_info%22+LIKE+%27%25Palo+Alto%25%27

```sql
select
  A.launch_rank,
  A.partner_info
from
  summary A
  INNER JOIN summary B ON A.launch_rank >= B.launch_rank - 3
  AND A.launch_rank -4 <= B.launch_rank
WHERE
  B.""partner_info"" LIKE '%Palo Alto%'
```
I've been moving projects from Vercel to Cloud Run when they run into this, but that's not a great situation to be in.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",767561886,Syntax error with + symbol when deployed to Vercel,
https://github.com/simonw/datasette/issues/675#issuecomment-747059277,https://api.github.com/repos/simonw/datasette/issues/675,747059277,MDEyOklzc3VlQ29tbWVudDc0NzA1OTI3Nw==,9599,simonw,2020-12-16T21:43:52Z,2020-12-16T21:43:52Z,OWNER,"It turns out I need this for a couple of projects:

- [datasette-ripgrep](https://github.com/simonw/datasette-ripgrep) needs to ship a whole bunch of source code files up in a known location. I worked around this with a nasty hack involving `--static` but it would be better if I wasn't doing that.
- [dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) uses an additional `dogsheep-beta.yml` configuration file in the project root (a sibling to `metadata.yml`) which needs to be included when publishing - see https://github.com/simonw/datasette.io/issues/21#issuecomment-747058067

I want this for `datasette publish cloudrun`, not just for `datasette package`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747034481,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747034481,MDEyOklzc3VlQ29tbWVudDc0NzAzNDQ4MQ==,9599,simonw,2020-12-16T21:17:05Z,2020-12-16T21:17:05Z,MEMBER,I'm just going to add `q` for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747031608,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747031608,MDEyOklzc3VlQ29tbWVudDc0NzAzMTYwOA==,9599,simonw,2020-12-16T21:15:18Z,2020-12-16T21:15:18Z,MEMBER,Should I pass any other details to the `display_sql` here as well?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747030964,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747030964,MDEyOklzc3VlQ29tbWVudDc0NzAzMDk2NA==,9599,simonw,2020-12-16T21:14:54Z,2020-12-16T21:14:54Z,MEMBER,"To do this I'll need the search term to be passed to the `display_sql` SQL query:
https://github.com/dogsheep/dogsheep-beta/blob/4890ec87b5e2ec48940f32c9ad1f5aae25c75a4d/dogsheep_beta/__init__.py#L164-L171","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747029636,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747029636,MDEyOklzc3VlQ29tbWVudDc0NzAyOTYzNg==,9599,simonw,2020-12-16T21:14:03Z,2020-12-16T21:14:03Z,MEMBER,"I think I can do this as a cunning trick in `display_sql`. Consider this example query: https://til.simonwillison.net/tils?sql=select%0D%0A++path%2C%0D%0A++snippet%28til_fts%2C+-1%2C+%27b4de2a49c8%27%2C+%278c94a2ed4b%27%2C+%27...%27%2C+60%29+as+snippet%0D%0Afrom%0D%0A++til%0D%0A++join+til_fts+on+til.rowid+%3D+til_fts.rowid%0D%0Awhere%0D%0A++til_fts+match+escape_fts%28%3Aq%29%0D%0A++and+path+%3D+%27asgi_lifespan-test-httpx.md%27%0D%0A&q=pytest

```sql
select
  path,
  snippet(til_fts, -1, 'b4de2a49c8', '8c94a2ed4b', '...', 60) as snippet
from
  til
  join til_fts on til.rowid = til_fts.rowid
where
  til_fts match escape_fts(:q)
  and path = 'asgi_lifespan-test-httpx.md'
```
The `and path = 'asgi_lifespan-test-httpx.md'` bit means we only get back a specific document - but the snippet highlighting is applied to it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/simonw/datasette/issues/1143#issuecomment-746827083,https://api.github.com/repos/simonw/datasette/issues/1143,746827083,MDEyOklzc3VlQ29tbWVudDc0NjgyNzA4Mw==,9599,simonw,2020-12-16T18:56:07Z,2020-12-16T18:56:07Z,OWNER,"I think the right way to do this is to support multiple optional `--cors-origin=` pattern values, like you suggested.","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746735889,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58,746735889,MDEyOklzc3VlQ29tbWVudDc0NjczNTg4OQ==,9599,simonw,2020-12-16T17:59:50Z,2020-12-16T17:59:50Z,MEMBER,"I don't want to add a full HTML parser (like BeautifulSoup) as a dependency for this feature. Since the HTML comes from a single, trusted source (GitHub) I could probably handle this using [regular expressions](https://stackoverflow.com/a/1732454).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769150394,Readme HTML has broken internal links,
https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746734412,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58,746734412,MDEyOklzc3VlQ29tbWVudDc0NjczNDQxMg==,9599,simonw,2020-12-16T17:58:56Z,2020-12-16T17:58:56Z,MEMBER,"I'm going to rewrite those `<a href=""#filtering-tables"">` links to `<a href=""#user-content-filtering-tables"">` - but only if a corresponding `id=""user-content-filtering-tables""` element exists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769150394,Readme HTML has broken internal links,
https://github.com/simonw/datasette/issues/1143#issuecomment-744757558,https://api.github.com/repos/simonw/datasette/issues/1143,744757558,MDEyOklzc3VlQ29tbWVudDc0NDc1NzU1OA==,9599,simonw,2020-12-14T22:42:10Z,2020-12-14T22:42:10Z,OWNER,"This may involve a breaking change to the CLI settings interface, so I'm adding this to the 1.0 milestone.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/simonw/datasette/issues/1143#issuecomment-744756861,https://api.github.com/repos/simonw/datasette/issues/1143,744756861,MDEyOklzc3VlQ29tbWVudDc0NDc1Njg2MQ==,9599,simonw,2020-12-14T22:40:28Z,2020-12-14T22:40:28Z,OWNER,"That's a very convincing argument. I'm keen on making sure Datasette is ""secure by default"" so you're right, encouraging finely grains CORS rules in core rather than leaving that to a plugin sounds like the right call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/simonw/datasette/issues/1142#issuecomment-744576894,https://api.github.com/repos/simonw/datasette/issues/1142,744576894,MDEyOklzc3VlQ29tbWVudDc0NDU3Njg5NA==,9599,simonw,2020-12-14T17:03:13Z,2020-12-14T17:03:13Z,OWNER,"I'm not sure about the radio boxes for JSON, just because you can't right-click on a radio box and copy it to your clipboard like you can with links. Worth trying it out though.

The radio boxes for that CSV option are definitely the right way to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1142#issuecomment-744563209,https://api.github.com/repos/simonw/datasette/issues/1142,744563209,MDEyOklzc3VlQ29tbWVudDc0NDU2MzIwOQ==,9599,simonw,2020-12-14T16:41:11Z,2020-12-14T16:41:11Z,OWNER,"To check out and start the server:

    /tmp % git clone git@github.com:nitinpaul/datasette
    Cloning into 'datasette'...
    remote: Enumerating objects: 124, done.
    # ...
    datasette % python3 -m venv venv
    datasette % source venv/bin/activate
    (venv) datasette % pip install -e '.[test]'
    Obtaining file:///private/tmp/datasette
    Collecting asgiref<3.4.0,>=3.2.10
    Using cached asgiref-3.3.1-py3-none-any.whl (19 kB)
    # ...
    (venv) datasette % datasette
    INFO:     Started server process [24002]
    INFO:     Waiting for application startup.
    INFO:     Application startup complete.
    INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

And to run the tests:

    (venv) datasette % pytest
    ======================================================================== test session starts ========================================================================
    platform darwin -- Python 3.9.1, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
    SQLite: 3.34.0
    rootdir: /private/tmp/datasette, configfile: pytest.ini
    plugins: asyncio-0.14.0, timeout-1.4.2
    collected 841 items                                                                                                                                                 

    tests/test_package.py ..                                                                                                                                      [  0%]
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1142#issuecomment-744251252,https://api.github.com/repos/simonw/datasette/issues/1142,744251252,MDEyOklzc3VlQ29tbWVudDc0NDI1MTI1Mg==,9599,simonw,2020-12-14T07:56:38Z,2020-12-14T07:56:38Z,OWNER,That's a really solid design for this! I'd be very happy to review a pull request - you should be able to implement this with just template edits and some CSS changes I think.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1143#issuecomment-744249157,https://api.github.com/repos/simonw/datasette/issues/1143,744249157,MDEyOklzc3VlQ29tbWVudDc0NDI0OTE1Nw==,9599,simonw,2020-12-14T07:53:15Z,2020-12-14T07:53:15Z,OWNER,"Does this plugin do everything you need? https://github.com/simonw/datasette-cors

I'm open to arguments as to why this should be in core rather than in a plugin - I'm on the fence about that at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/simonw/datasette/issues/741#issuecomment-744142692,https://api.github.com/repos/simonw/datasette/issues/741,744142692,MDEyOklzc3VlQ29tbWVudDc0NDE0MjY5Mg==,9599,simonw,2020-12-14T03:28:56Z,2020-12-14T03:28:56Z,OWNER,I'm going to try this out on `datasette-publish-vercel` first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/983#issuecomment-744066249,https://api.github.com/repos/simonw/datasette/issues/983,744066249,MDEyOklzc3VlQ29tbWVudDc0NDA2NjI0OQ==,9599,simonw,2020-12-13T20:47:52Z,2020-12-13T20:47:52Z,OWNER,"@yozlet just spotted this comment. Wow that is interesting!

With the right plugin hooks on the page (see also #987) one relatively simple way to do that could be with bookmarklets - users could install bookmarklets which, when executed against a Datasette page in their browser, use the existing JavaScript plugin integration points to add all kinds of functionality.

Doing full sandboxing is certainly daunting, but it looks like Figma figured it out so TIL it's technically feasible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743966801,https://api.github.com/repos/simonw/sqlite-utils/issues/207,743966801,MDEyOklzc3VlQ29tbWVudDc0Mzk2NjgwMQ==,9599,simonw,2020-12-13T07:25:23Z,2020-12-13T07:25:23Z,OWNER,"CLI documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#analyzing-tables

Python library documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#analyzing-a-column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763283616,sqlite-utils analyze-tables command,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-743966289,https://api.github.com/repos/simonw/sqlite-utils/issues/203,743966289,MDEyOklzc3VlQ29tbWVudDc0Mzk2NjI4OQ==,9599,simonw,2020-12-13T07:20:51Z,2020-12-13T07:20:51Z,OWNER,Sorry for not reviewing this yet! I'll try to carve out time to look at it in the next few days.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743956666,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743956666,MDEyOklzc3VlQ29tbWVudDc0Mzk1NjY2Ng==,9599,simonw,2020-12-13T05:44:49Z,2020-12-13T05:44:49Z,OWNER,"Example output:
```
% sqlite-utils analyze-tables github.db tags             
tags.repo: (1/3)

  Total rows: 261
  Null rows: 0
  Blank rows: 0

  Distinct values: 14

  Most common:
    88: 107914493
    75: 140912432
    27: 206156866
    21: 207052882
    17: 197431109
    8: 197882382
    5: 256834907
    5: 205429375
    4: 248903544
    3: 206202864

  Least common:
    1: 209590345
    2: 206649770
    2: 303218369
    3: 206202864
    3: 213286752
    4: 248903544
    5: 205429375
    5: 256834907
    8: 197882382
    17: 197431109

tags.name: (2/3)

  Total rows: 261
  Null rows: 0
  Blank rows: 0

  Distinct values: 175

  Most common:
    10: 0.2
    9: 0.1
    7: 0.3
    6: 0.4
    5: 0.7
    5: 0.5
    5: 0.1a
    4: 0.9
    4: 0.8
    4: 0.6

  Least common:
    1: 0.1.1
    1: 0.11.1
    1: 0.1a2
    1: 0.20.1
    1: 0.21.1
    1: 0.21.2
    1: 0.21.3
    1: 0.22
    1: 0.22.1
    1: 0.23

tags.sha: (3/3)

  Total rows: 261
  Null rows: 0
  Blank rows: 0

  Distinct values: 261
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/datasette/issues/1142#issuecomment-743913004,https://api.github.com/repos/simonw/datasette/issues/1142,743913004,MDEyOklzc3VlQ29tbWVudDc0MzkxMzAwNA==,9599,simonw,2020-12-12T22:17:46Z,2020-12-12T22:17:46Z,OWNER,"You're actually choosing between two options here: the 100 rows you can see on the screen, or the x,000 rows that match the current query.

Maybe a radio box would be more obvious?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1142#issuecomment-743912875,https://api.github.com/repos/simonw/datasette/issues/1142,743912875,MDEyOklzc3VlQ29tbWVudDc0MzkxMjg3NQ==,9599,simonw,2020-12-12T22:16:38Z,2020-12-12T22:16:38Z,OWNER,"Yeah, maybe with the number of rows to make it completely clear. `Include all 2,455 rows` perhaps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708524,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708524,MDEyOklzc3VlQ29tbWVudDc0MzcwODUyNA==,9599,simonw,2020-12-12T05:48:20Z,2020-12-12T05:48:32Z,OWNER,"```
% sqlite-utils analyze-tables ../datasette/fixtures.db facetable --column pk
1/1: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708325,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708325,MDEyOklzc3VlQ29tbWVudDc0MzcwODMyNQ==,9599,simonw,2020-12-12T05:46:27Z,2020-12-12T05:46:27Z,OWNER,"It would be neat if you could optionally specify a subset of columns to analyze, using `-c` or `--column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708169,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708169,MDEyOklzc3VlQ29tbWVudDc0MzcwODE2OQ==,9599,simonw,2020-12-12T05:44:46Z,2020-12-12T05:44:46Z,OWNER,"If there are less than ten values is it worth outputting them twice, once in `most_common` and then in reverse in `least_common`? Feels redundant - I think I should leave `least_common` empty in that case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708080,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708080,MDEyOklzc3VlQ29tbWVudDc0MzcwODA4MA==,9599,simonw,2020-12-12T05:43:45Z,2020-12-12T05:43:45Z,OWNER,"CLI output looks like this at the moment, which is bad:
```
 % sqlite-utils analyze-tables ../datasette/fixtures.db facetable
1/10: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None)
2/10: ColumnDetails(table='facetable', column='created', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[('2019-01-17 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-14 08:00:00', 4), ('2019-01-16 08:00:00', 3)], least_common=[('2019-01-16 08:00:00', 3), ('2019-01-14 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-17 08:00:00', 4)])
3/10: ColumnDetails(table='facetable', column='planet_int', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (2, 1)], least_common=[(2, 1), (1, 14)])
4/10: ColumnDetails(table='facetable', column='on_earth', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (0, 1)], least_common=[(0, 1), (1, 14)])
5/10: ColumnDetails(table='facetable', column='state', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('CA', 10), ('MI', 4), ('MC', 1)], least_common=[('MC', 1), ('MI', 4), ('CA', 10)])
6/10: ColumnDetails(table='facetable', column='city_id', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[(1, 6), (3, 4), (2, 4), (4, 1)], least_common=[(4, 1), (2, 4), (3, 4), (1, 6)])
7/10: ColumnDetails(table='facetable', column='neighborhood', total_rows=15, num_null=0, num_blank=0, num_distinct=14, most_common=[('Downtown', 2), ('Tenderloin', 1), ('SOMA', 1), ('Mission', 1), ('Mexicantown', 1), ('Los Feliz', 1), ('Koreatown', 1), ('Hollywood', 1), ('Hayes Valley', 1), ('Greektown', 1)], least_common=[('Arcadia Planitia', 1), ('Bernal Heights', 1), ('Corktown', 1), ('Dogpatch', 1), ('Greektown', 1), ('Hayes Valley', 1), ('Hollywood', 1), ('Koreatown', 1), ('Los Feliz', 1), ('Mexicantown', 1)])
8/10: ColumnDetails(table='facetable', column='tags', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('[]', 13), ('[""tag1"", ""tag3""]', 1), ('[""tag1"", ""tag2""]', 1)], least_common=[('[""tag1"", ""tag2""]', 1), ('[""tag1"", ""tag3""]', 1), ('[]', 13)])
9/10: ColumnDetails(table='facetable', column='complex_array', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[('[]', 14), ('[{""foo"": ""bar""}]', 1)], least_common=[('[{""foo"": ""bar""}]', 1), ('[]', 14)])
10/10: ColumnDetails(table='facetable', column='distinct_some_null', total_rows=15, num_null=13, num_blank=0, num_distinct=2, most_common=[(None, 13), ('two', 1), ('one', 1)], least_common=[('one', 1), ('two', 1), (None, 13)])
(sqlite-utils) sqlite-utils % 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743707969,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743707969,MDEyOklzc3VlQ29tbWVudDc0MzcwNzk2OQ==,9599,simonw,2020-12-12T05:42:26Z,2020-12-12T05:43:06Z,OWNER,"Should truncate values in the least/most common JSON array to a sensible length, otherwise you end up with stuff like this:
```json
[
    [
        ""b'\\x00\\x05barry\\x03\\x01\\x02\\x00\\x00\\x03cat\\x03\\x01\\x03\\x00\\x00\\x03dog\\x08\\x01\\x01\\x01\\x03\\x00\\x01\\x03\\x00\\x00\\x07panther\\x05\\x01\\x01\\x02\\x02\\x00\\x01\\x03uma\\x05\\x02\\x01\\x02\\x02\\x00\\x00\\x04sara\\x05\\x02\\x01\\x01\\x02\\x00\\x00\\x05terry\\x08\\x01\\x01\\x01\\x02\\x00\\x01\\x02\\x00\\x00\\x06weasel\\x05\\x02\\x01\\x01\\x03\\x00'"",
        1
    ]
]
```
This example also shows that binary values (like those in `_fts` tables) look a bit weird, but I think I'm OK with that since binary data can't be represented neatly in JSON anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701697,https://api.github.com/repos/simonw/sqlite-utils/issues/207,743701697,MDEyOklzc3VlQ29tbWVudDc0MzcwMTY5Nw==,9599,simonw,2020-12-12T04:39:51Z,2020-12-12T04:39:51Z,OWNER,"CLI could be:

    sqlite-utils analyze-tables

To analyze all tables or:

    sqlite-utils analyze-tables table1 table2

To analyze specific tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763283616,sqlite-utils analyze-tables command,
https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701599,https://api.github.com/repos/simonw/sqlite-utils/issues/207,743701599,MDEyOklzc3VlQ29tbWVudDc0MzcwMTU5OQ==,9599,simonw,2020-12-12T04:38:52Z,2020-12-12T04:39:07Z,OWNER,I'll add a `table.analyze_column(column)` method which is used by the CLI tool - with a note that this is an unstable interface which may change in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763283616,sqlite-utils analyze-tables command,
https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701422,https://api.github.com/repos/simonw/sqlite-utils/issues/207,743701422,MDEyOklzc3VlQ29tbWVudDc0MzcwMTQyMg==,9599,simonw,2020-12-12T04:37:14Z,2020-12-12T04:38:25Z,OWNER,"Prototype:
```python
from collections import namedtuple

ColumnDetails = namedtuple(""ColumnDetails"", (""column"", ""num_null"", ""num_blank"", ""num_distinct"", ""most_common"", ""least_common""))

def analyze_column(db, table, column, values=10):
    num_null = db.execute(""select count(*) from [{}] where [{}] is null"".format(table, column)).fetchone()[0]
    num_blank = db.execute(""select count(*) from [{}] where [{}] = ''"".format(table, column)).fetchone()[0]
    num_distinct = db.execute(""select count(distinct [{}]) from [{}]"".format(column, table)).fetchone()[0]
    most_common = None
    least_common = None
    if num_distinct != 1:
        most_common = [(r[0], r[1]) for r in db.execute(
            ""select [{}], count(*) from [{}] group by [{}] order by count(*) desc limit "".format(column, table, column, values)
        ).fetchall()]
        if num_distinct <= values:
            # No need to run the query if it will just return the results in revers order
            least_common = most_common[::-1]
        else:
            least_common = [(r[0], r[1]) for r in db.execute(
                ""select [{}], count(*) from [{}] group by [{}] order by count(*) limit {}"".format(column, table, column, values)
            ).fetchall()]
    return ColumnDetails(column, num_null, num_blank, num_distinct, most_common, least_common)


def analyze_table(db, table):
    for column in db[table].columns:
        details = analyze_column(db, table, column.name)
        print(details)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763283616,sqlite-utils analyze-tables command,
https://github.com/simonw/sqlite-utils/issues/205#issuecomment-742737794,https://api.github.com/repos/simonw/sqlite-utils/issues/205,742737794,MDEyOklzc3VlQ29tbWVudDc0MjczNzc5NA==,9599,simonw,2020-12-10T19:18:22Z,2020-12-10T19:18:22Z,OWNER,"Yup, it looks like you're using a window function that was added in SQLite 3.25.0: https://www.sqlite.org/changes.html#version_3_25_0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760960559,"sqlite3.OperationalError: near ""("": syntax error",
https://github.com/simonw/datasette/issues/1091#issuecomment-741992106,https://api.github.com/repos/simonw/datasette/issues/1091,741992106,MDEyOklzc3VlQ29tbWVudDc0MTk5MjEwNg==,9599,simonw,2020-12-09T19:19:54Z,2020-12-09T20:27:45Z,OWNER,"Could you try removing the `ProxyPassReverse /datasette http://0.0.0.0:8001` line?

My hunch is that `ProxyPassReverse` is rewriting some of the links in the HTML (or maybe in the HTTP headers) in a way that breaks things.

Normally you would need `ProxyPassReverse` to compensate for the underlying application being unable to rewrite its links - but Datasette's `base_url` setting causes Datasette to rewrite all of the links for you, so `ProxyPassReverse` should be unneccessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1134#issuecomment-742024588,https://api.github.com/repos/simonw/datasette/issues/1134,742024588,MDEyOklzc3VlQ29tbWVudDc0MjAyNDU4OA==,9599,simonw,2020-12-09T20:19:59Z,2020-12-09T20:20:33Z,OWNER,https://byraadsarkivet.aarhus.dk/db/cases?_searchmode=raw&_search=sundhedsfrem%2A is an absolutely beautiful example of a themed Datasette! Very excited to show this to people.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760312579,"""_searchmode=raw"" throws an index out of range error when combined with ""_search_COLUMN""",
https://github.com/simonw/datasette/issues/1134#issuecomment-742023775,https://api.github.com/repos/simonw/datasette/issues/1134,742023775,MDEyOklzc3VlQ29tbWVudDc0MjAyMzc3NQ==,9599,simonw,2020-12-09T20:18:23Z,2020-12-09T20:18:23Z,OWNER,A fix for this should be available if you upgrade to 0.52.5,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760312579,"""_searchmode=raw"" throws an index out of range error when combined with ""_search_COLUMN""",
https://github.com/simonw/datasette/issues/1091#issuecomment-742023541,https://api.github.com/repos/simonw/datasette/issues/1091,742023541,MDEyOklzc3VlQ29tbWVudDc0MjAyMzU0MQ==,9599,simonw,2020-12-09T20:17:54Z,2020-12-09T20:17:54Z,OWNER,OK that is really weird. I'll have another go at replicating this locally.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1136#issuecomment-742023111,https://api.github.com/repos/simonw/datasette/issues/1136,742023111,MDEyOklzc3VlQ29tbWVudDc0MjAyMzExMQ==,9599,simonw,2020-12-09T20:17:02Z,2020-12-09T20:17:02Z,OWNER,Documentation for this procedure is now here: https://docs.datasette.io/en/latest/contributing.html#releasing-bug-fixes-from-a-branch,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742022222,https://api.github.com/repos/simonw/datasette/issues/1136,742022222,MDEyOklzc3VlQ29tbWVudDc0MjAyMjIyMg==,9599,simonw,2020-12-09T20:15:24Z,2020-12-09T20:15:51Z,OWNER,Used this procedure for the first time for 0.52.5 - deploy run here: https://github.com/simonw/datasette/actions/runs/411465648 - PyPI release here: https://pypi.org/project/datasette/0.52.5/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742017622,https://api.github.com/repos/simonw/datasette/issues/1136,742017622,MDEyOklzc3VlQ29tbWVudDc0MjAxNzYyMg==,9599,simonw,2020-12-09T20:06:47Z,2020-12-09T20:06:47Z,OWNER,"Then I can ship the release directly from that branch, creating the tag as part of the release process:

<img width=""1057"" alt=""New_release_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/101681442-ffc93500-3a16-11eb-8801-3a0e357a2bc1.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742014881,https://api.github.com/repos/simonw/datasette/issues/1136,742014881,MDEyOklzc3VlQ29tbWVudDc0MjAxNDg4MQ==,9599,simonw,2020-12-09T20:01:27Z,2020-12-09T20:01:27Z,OWNER,"I'll write the release notes in the branch, then cherry-pick them over to `main`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742014366,https://api.github.com/repos/simonw/datasette/issues/1136,742014366,MDEyOklzc3VlQ29tbWVudDc0MjAxNDM2Ng==,9599,simonw,2020-12-09T20:00:35Z,2020-12-09T20:00:35Z,OWNER,"Actually I'll start from 0.52.4 and then cherry-pick the fixes.

    git branch 0.52.x 0.52.4
    git checkout 0.52.x
    git cherry-pick COMMIT","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1134#issuecomment-742012324,https://api.github.com/repos/simonw/datasette/issues/1134,742012324,MDEyOklzc3VlQ29tbWVudDc0MjAxMjMyNA==,9599,simonw,2020-12-09T19:57:05Z,2020-12-09T19:57:05Z,OWNER,Thanks for the bug report!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760312579,"""_searchmode=raw"" throws an index out of range error when combined with ""_search_COLUMN""",
https://github.com/simonw/datasette/issues/1136#issuecomment-742009294,https://api.github.com/repos/simonw/datasette/issues/1136,742009294,MDEyOklzc3VlQ29tbWVudDc0MjAwOTI5NA==,9599,simonw,2020-12-09T19:51:18Z,2020-12-09T19:51:18Z,OWNER,"Likewise, Read The Docs publishes as stable the docs from the latest tagged release, so I would expect that to work fine as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742009101,https://api.github.com/repos/simonw/datasette/issues/1136,742009101,MDEyOklzc3VlQ29tbWVudDc0MjAwOTEwMQ==,9599,simonw,2020-12-09T19:50:53Z,2020-12-09T19:50:53Z,OWNER,"My concern is if this will  break anything about CI. I don't think it will - the code that deploys the latest `main` to https://latest.datasette.io/ should be unaffected, and the checkout code in `publish.yml` should check out the correct code based on the tag used for that release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742008087,https://api.github.com/repos/simonw/datasette/issues/1136,742008087,MDEyOklzc3VlQ29tbWVudDc0MjAwODA4Nw==,9599,simonw,2020-12-09T19:48:56Z,2020-12-09T19:48:56Z,OWNER,I think I'm going to create a branch called `0.52.x` that starts with `8ae0f9f7f0d644b0161165a1084f53acd2786f7c` and then tag the release from there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1133#issuecomment-740850920,https://api.github.com/repos/simonw/datasette/issues/1133,740850920,MDEyOklzc3VlQ29tbWVudDc0MDg1MDkyMA==,9599,simonw,2020-12-08T18:55:59Z,2020-12-08T18:55:59Z,OWNER,Inspiration was this script: https://gist.github.com/simonw/f6e3cd29fde5d15ea9cd746c942046ba - which pipes output through `tail -n +2` to strip off the headers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",759695780,Option to omit header row in CSV export,
https://github.com/simonw/datasette/issues/1133#issuecomment-740850057,https://api.github.com/repos/simonw/datasette/issues/1133,740850057,MDEyOklzc3VlQ29tbWVudDc0MDg1MDA1Nw==,9599,simonw,2020-12-08T18:55:29Z,2020-12-08T18:55:29Z,OWNER,Can work on this as part of #1062.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",759695780,Option to omit header row in CSV export,
https://github.com/simonw/sqlite-utils/pull/204#issuecomment-740796067,https://api.github.com/repos/simonw/sqlite-utils/issues/204,740796067,MDEyOklzc3VlQ29tbWVudDc0MDc5NjA2Nw==,9599,simonw,2020-12-08T17:49:22Z,2020-12-08T17:49:22Z,OWNER,"Great catch, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752888228,use jsonify_if_need for sql updates,
https://github.com/simonw/datasette/issues/815#issuecomment-740385032,https://api.github.com/repos/simonw/datasette/issues/815,740385032,MDEyOklzc3VlQ29tbWVudDc0MDM4NTAzMg==,9599,simonw,2020-12-08T05:26:09Z,2020-12-08T05:26:16Z,OWNER,"Sure! It's a bit of a fiddle one - I've not found an approach that I like, but I also haven't thought about it since June. I'd love to see what you come up with!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/1132#issuecomment-740228858,https://api.github.com/repos/simonw/datasette/issues/1132,740228858,MDEyOklzc3VlQ29tbWVudDc0MDIyODg1OA==,9599,simonw,2020-12-07T22:50:36Z,2020-12-07T22:50:36Z,OWNER,"Documented here: https://docs.datasette.io/en/latest/json_api.html#column-filter-arguments

Demo: https://latest.datasette.io/fixtures/facetable?tags__arraynotcontains=tag2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",758899581,New filter: array does not contain,
https://github.com/simonw/datasette/issues/1131#issuecomment-739414118,https://api.github.com/repos/simonw/datasette/issues/1131,739414118,MDEyOklzc3VlQ29tbWVudDczOTQxNDExOA==,9599,simonw,2020-12-05T20:48:33Z,2020-12-05T20:48:33Z,OWNER,"Oddly enough, I tried fixing this with `sys.stderr.write(""{}\n"".format(e))` - but my Click `CLIRunner` tests failed because `result.stderr` was an empty string. Adding `sys.stderr.flush()` to the code that output errors fixed that issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/simonw/datasette/issues/398#issuecomment-739357330,https://api.github.com/repos/simonw/datasette/issues/398,739357330,MDEyOklzc3VlQ29tbWVudDczOTM1NzMzMA==,9599,simonw,2020-12-05T19:36:27Z,2020-12-05T19:36:27Z,OWNER,This was fixed in #749 by 88ac538b41a4753c3de9b509c3a0e13077f66182,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",398011658,Ensure downloading a 100+MB SQLite database file works,
https://github.com/simonw/datasette/pull/1128#issuecomment-739355855,https://api.github.com/repos/simonw/datasette/issues/1128,739355855,MDEyOklzc3VlQ29tbWVudDczOTM1NTg1NQ==,9599,simonw,2020-12-05T19:34:57Z,2020-12-05T19:34:57Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756867924,Fix startup error on windows,
https://github.com/simonw/datasette/issues/1131#issuecomment-739083673,https://api.github.com/repos/simonw/datasette/issues/1131,739083673,MDEyOklzc3VlQ29tbWVudDczOTA4MzY3Mw==,9599,simonw,2020-12-05T00:02:10Z,2020-12-05T00:02:10Z,OWNER,"https://clig.dev/#the-basics

> **Send messaging to stderr**. Log messages, errors, and so on should all be sent to stderr. This means that when commands are piped together, these messages are displayed to the user and not fed into the next command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/simonw/datasette/issues/1131#issuecomment-739083472,https://api.github.com/repos/simonw/datasette/issues/1131,739083472,MDEyOklzc3VlQ29tbWVudDczOTA4MzQ3Mg==,9599,simonw,2020-12-05T00:01:12Z,2020-12-05T00:01:12Z,OWNER,Here's why: https://github.com/simonw/datasette/blob/37f87b5e52e7f8ddd1c4ffcf368bd7a62a406a6d/datasette/database.py#L158-L163,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/dogsheep/dogsheep-photos/pull/29#issuecomment-739058820,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29,739058820,MDEyOklzc3VlQ29tbWVudDczOTA1ODgyMA==,9599,simonw,2020-12-04T22:32:35Z,2020-12-04T22:32:35Z,MEMBER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638375985,Fixed bug in SQL query for photo scores,
https://github.com/simonw/datasette/issues/188#issuecomment-738905376,https://api.github.com/repos/simonw/datasette/issues/188,738905376,MDEyOklzc3VlQ29tbWVudDczODkwNTM3Ng==,9599,simonw,2020-12-04T17:18:34Z,2020-12-04T17:18:34Z,OWNER,This is likely to be covered by plugin hooks: #860 for the metadata and after investigating in #1042 it looks like the existing `prepare_jinja2_environment` hook may already be enough to load templates from the database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",309047460,Ability to bundle metadata and templates inside the SQLite file,
https://github.com/simonw/datasette/issues/111#issuecomment-738904347,https://api.github.com/repos/simonw/datasette/issues/111,738904347,MDEyOklzc3VlQ29tbWVudDczODkwNDM0Nw==,9599,simonw,2020-12-04T17:16:56Z,2020-12-04T17:16:56Z,OWNER,This is STILL a good idea.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/datasette/pull/1130#issuecomment-738897582,https://api.github.com/repos/simonw/datasette/issues/1130,738897582,MDEyOklzc3VlQ29tbWVudDczODg5NzU4Mg==,9599,simonw,2020-12-04T17:03:30Z,2020-12-04T17:03:30Z,OWNER,"I deployed this to https://datasette-issue-1129.vercel.app/ (using `datasette publish vercel fixtures.db --branch 8d4c69c6fb0ef741a19070f5172017ea3522e83c --about_url https://github.com/simonw/datasette/issues/1129 --about datasette/issues/1129 --project datasette-issue-1129`) - weirdly, on Mobile Safari the footer appears just below the visible window:

![RPReplay_Final1607100726](https://user-images.githubusercontent.com/9599/101191950-336b2000-360f-11eb-8f14-ed83bd86515c.gif)

I've seen other problems with fixed footers on Mobile Safari too: at Eventbrite this was a really nasty problem for us to figure out: https://www.eventbrite.com/engineering/mobile-safari-why/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756876238,Fix footer not sticking to bottom in short pages,
https://github.com/simonw/datasette/issues/1125#issuecomment-738554392,https://api.github.com/repos/simonw/datasette/issues/1125,738554392,MDEyOklzc3VlQ29tbWVudDczODU1NDM5Mg==,9599,simonw,2020-12-04T04:16:57Z,2020-12-04T04:16:57Z,OWNER,"https://latest.datasette.io/-/versions now shows this:
```json
{
    ""python"": {
        ""version"": ""3.8.6"",
        ""full"": ""3.8.6 (default, Nov 18 2020, 13:49:49) \n[GCC 8.3.0]""
    },
    ""datasette"": {
        ""version"": ""0.52.3"",
        ""note"": ""49d8fc056844d5a537d6cfd96dab0dd5686fe718""
    },
    ""asgi"": ""3.0"",
    ""uvicorn"": ""0.12.3"",
    ""sqlite"": {
        ""version"": ""3.33.0"",
        ""fts_versions"": [
            ""FTS5"",
            ""FTS4"",
            ""FTS3""
        ],
        ""extensions"": {
            ""json1"": null
        },
        ""compile_options"": []
    },
    ""pysqlite3"": ""0.4.4""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738551280,https://api.github.com/repos/simonw/datasette/issues/1125,738551280,MDEyOklzc3VlQ29tbWVudDczODU1MTI4MA==,9599,simonw,2020-12-04T04:03:54Z,2020-12-04T04:03:54Z,OWNER,"I'm going to check `pkg_resources.get_distribution(""pysqlite3-binary"").version` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738550588,https://api.github.com/repos/simonw/datasette/issues/1125,738550588,MDEyOklzc3VlQ29tbWVudDczODU1MDU4OA==,9599,simonw,2020-12-04T04:01:10Z,2020-12-04T04:01:10Z,OWNER,"Urgh, figuring out the version of `pysqlite3` is WAY harder than I expected. The `getversion` module looks like the smartest attempt at solving this problem generally, but I'd like to avoid adding another dependency just for this: https://github.com/smarie/python-getversion","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738548693,https://api.github.com/repos/simonw/datasette/issues/1125,738548693,MDEyOklzc3VlQ29tbWVudDczODU0ODY5Mw==,9599,simonw,2020-12-04T03:52:51Z,2020-12-04T03:52:51Z,OWNER,"That didn't work - https://latest.datasette.io/-/versions isn't showing the package.

I bet that's because I'm actually installing `pysqlite3-binary` here: https://github.com/simonw/datasette/blob/e2fea36540e952d8d72c1bd0af7144b85b7a4671/.github/workflows/deploy-latest.yml#L57","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1126#issuecomment-738548393,https://api.github.com/repos/simonw/datasette/issues/1126,738548393,MDEyOklzc3VlQ29tbWVudDczODU0ODM5Mw==,9599,simonw,2020-12-04T03:51:38Z,2020-12-04T03:51:38Z,OWNER,That worked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756761963,Switch to google-github-actions/setup-gcloud for demo deploy,
https://github.com/simonw/datasette/issues/1125#issuecomment-738347171,https://api.github.com/repos/simonw/datasette/issues/1125,738347171,MDEyOklzc3VlQ29tbWVudDczODM0NzE3MQ==,9599,simonw,2020-12-03T22:04:52Z,2020-12-03T22:04:52Z,OWNER,"```
pkg_resources.get_distribution(""pysqlite3"").version
Out[14]: '0.4.4'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1124#issuecomment-738215686,https://api.github.com/repos/simonw/datasette/issues/1124,738215686,MDEyOklzc3VlQ29tbWVudDczODIxNTY4Ng==,9599,simonw,2020-12-03T18:50:48Z,2020-12-03T21:42:02Z,OWNER,I'm going to punt on writing a unit test for this (not sure how I'd simulate those symlinks) - I'll manually test it and push out a dot release instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738215487,https://api.github.com/repos/simonw/datasette/issues/1124,738215487,MDEyOklzc3VlQ29tbWVudDczODIxNTQ4Nw==,9599,simonw,2020-12-03T18:50:26Z,2020-12-03T21:41:25Z,OWNER,"This fix works - calling `.resolve()` on the `root_path` before the comparison to ensure symlinks are resolved:
```python
        # Ensure full_path is within root_path to avoid weird ""../"" tricks
        try:
            print(""full_path={}, root_path={}"".format(full_path, root_path))
            full_path.relative_to(root_path.resolve())
        except ValueError as e:
            print(""  ValueError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738213342,https://api.github.com/repos/simonw/datasette/issues/1124,738213342,MDEyOklzc3VlQ29tbWVudDczODIxMzM0Mg==,9599,simonw,2020-12-03T18:46:22Z,2020-12-03T21:40:51Z,OWNER,"I replaced that function with this code:
```python
def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
    async def inner_static(request, send):
        path = request.scope[""url_route""][""kwargs""][""path""]
        print(""path ="", path)
        try:
            full_path = (Path(root_path) / path).resolve().absolute()
        except FileNotFoundError as e:
            print(""FileNotFoundError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
        if full_path.is_dir():
            await asgi_send_html(send, ""403: Directory listing is not allowed"", 403)
            return
        # Ensure full_path is within root_path to avoid weird ""../"" tricks
        try:
            print(""full_path={}, root_path={}"".format(full_path, root_path))
            full_path.relative_to(root_path)
        except ValueError as e:
            print(""  ValueError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
        try:
            await asgi_send_file(send, full_path, chunk_size=chunk_size)
        except FileNotFoundError:
            await asgi_send_html(send, ""404"", 404)
            return

    return inner_static
```
Edited using `vi /home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/utils/asgi.py`

The output shows me what the bug is:
```
$ datasette --get /-/static/app.css --pdb
app_root =  /home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages
path = app.css
full_path=/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css, root_path=/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static
  ValueError: '/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css' does not start with '/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static'
404
```
`  ValueError: '/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css' does not start with '/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static'`

One is `../lib/python3.7/..` and the other is `../lib64/python3.7/..` - there's clearly some kind of symlink in play here which I'm not taking into account.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738313399,https://api.github.com/repos/simonw/datasette/issues/1124,738313399,MDEyOklzc3VlQ29tbWVudDczODMxMzM5OQ==,9599,simonw,2020-12-03T21:10:54Z,2020-12-03T21:10:54Z,OWNER,Confirmed that installing a fresh copy of Datasette 0.52.3 on that server works correctly as expected.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738224865,https://api.github.com/repos/simonw/datasette/issues/1124,738224865,MDEyOklzc3VlQ29tbWVudDczODIyNDg2NQ==,9599,simonw,2020-12-03T19:01:52Z,2020-12-03T19:01:52Z,OWNER,"https://github.com/simonw/datasette/runs/1494631261
```
/home/runner/work/datasette/datasette/tests/test_html.py:81: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File ""/home/runner/work/datasette/datasette/datasette/app.py"", line 1039, in route_path
    response = await view(request, send)
  File ""/home/runner/work/datasette/datasette/datasette/utils/asgi.py"", line 297, in inner_static
    full_path.relative_to(root_path.resolve())
AttributeError: 'str' object has no attribute 'resolve'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738220067,https://api.github.com/repos/simonw/datasette/issues/1124,738220067,MDEyOklzc3VlQ29tbWVudDczODIyMDA2Nw==,9599,simonw,2020-12-03T18:58:17Z,2020-12-03T18:58:17Z,OWNER,"I tested this by running:

    pipx uninstall datasette
    pipx install 'https://github.com/simonw/datasette/archive/6b4c55efea3e9d34d92cbe5f0066553ad9b14071.zip'

To replace that version of Datasette (in the correct virtual environment) with this patch. It worked!

```
[ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/static/app.css
/* Reset and Page Setup ==================================================== */
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738211776,https://api.github.com/repos/simonw/datasette/issues/1124,738211776,MDEyOklzc3VlQ29tbWVudDczODIxMTc3Ng==,9599,simonw,2020-12-03T18:43:21Z,2020-12-03T18:43:21Z,OWNER,I'm suspicious of this code here:https://github.com/simonw/datasette/blob/e048791a9a2686f47d81a2c8aa88aa1966d82521/datasette/utils/asgi.py#L284-L307,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738211152,https://api.github.com/repos/simonw/datasette/issues/1124,738211152,MDEyOklzc3VlQ29tbWVudDczODIxMTE1Mg==,9599,simonw,2020-12-03T18:42:12Z,2020-12-03T18:42:12Z,OWNER,"Added a line to print out `app_root` from https://github.com/simonw/datasette/blob/e048791a9a2686f47d81a2c8aa88aa1966d82521/datasette/app.py#L848-L853
```
app_root =  /home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738209642,https://api.github.com/repos/simonw/datasette/issues/1124,738209642,MDEyOklzc3VlQ29tbWVudDczODIwOTY0Mg==,9599,simonw,2020-12-03T18:39:19Z,2020-12-03T18:39:19Z,OWNER,"The CSS files are in the expected location:
```
[ec2-user@ip-172-31-30-7 ~]$ find /home/ec2-user/.local/pipx/venvs/datasette | grep css
/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css
/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/codemirror-5.57.0.min.css
```
Wow it's running an ANCIENT version of SQLite:
```
[ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/versions.json
{""python"": {""version"": ""3.7.9"", ""full"": ""3.7.9 (default, Aug 27 2020, 21:58:41) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)]""}, ""datasette"": {""version"": ""0.52.2""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.12.3"", ""sqlite"": {""version"": ""3.7.17"", ""fts_versions"": [""FTS4"", ""FTS3""], ""extensions"": {}, ""compile_options"": [""DISABLE_DIRSYNC"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_FTS3"", ""ENABLE_RTREE"", ""ENABLE_UNLOCK_NOTIFY"", ""SECURE_DELETE"", ""TEMP_STORE=1"", ""THREADSAFE=1""]}}
```
http://www.sqlite.org/releaselog/3_7_17.html - SQLite Release 3.7.17 On 2013-05-20","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1121#issuecomment-737591281,https://api.github.com/repos/simonw/datasette/issues/1121,737591281,MDEyOklzc3VlQ29tbWVudDczNzU5MTI4MQ==,9599,simonw,2020-12-03T01:03:18Z,2020-12-03T01:03:18Z,OWNER,"Demo: https://latest.datasette.io/fixtures?_bot=1

<img width=""729"" alt=""fixtures"" src=""https://user-images.githubusercontent.com/9599/100950004-43c0b500-34c0-11eb-918c-aa959376461f.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",754178780,Table actions cog is misaligned,
https://github.com/simonw/datasette/issues/1100#issuecomment-737589314,https://api.github.com/repos/simonw/datasette/issues/1100,737589314,MDEyOklzc3VlQ29tbWVudDczNzU4OTMxNA==,9599,simonw,2020-12-03T00:57:35Z,2020-12-03T00:57:35Z,OWNER,"Fixed in the demo:
```
% curl -XOPTIONS https://latest.datasette.io/fixtures.json
ok%
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",747702144,Error on OPTIONS request to database,
https://github.com/simonw/datasette/issues/1123#issuecomment-737586248,https://api.github.com/repos/simonw/datasette/issues/1123,737586248,MDEyOklzc3VlQ29tbWVudDczNzU4NjI0OA==,9599,simonw,2020-12-03T00:47:37Z,2020-12-03T00:47:37Z,OWNER,"Affected tests:
```
FAILED tests/test_plugins.py::test_hook_table_actions[facetable] - AssertionE...
FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - Assertio...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",755721275,"Table actions hook are order dependent, should not be",
https://github.com/simonw/datasette/issues/1100#issuecomment-737581719,https://api.github.com/repos/simonw/datasette/issues/1100,737581719,MDEyOklzc3VlQ29tbWVudDczNzU4MTcxOQ==,9599,simonw,2020-12-03T00:35:23Z,2020-12-03T00:35:23Z,OWNER,"Replicated this against the live demo as well:
```
/tmp % curl -XOPTIONS https://latest.datasette.io/fixtures.json
{""ok"": false, ""error"": ""object Response can't be used in 'await' expression"", ""status"": 500, ""title"": null}%                                                                                                                                  /tmp % 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",747702144,Error on OPTIONS request to database,
https://github.com/simonw/datasette/pull/1122#issuecomment-737580813,https://api.github.com/repos/simonw/datasette/issues/1122,737580813,MDEyOklzc3VlQ29tbWVudDczNzU4MDgxMw==,9599,simonw,2020-12-03T00:33:09Z,2020-12-03T00:33:09Z,OWNER,"This is a very neat fix, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",754179035,Fix misaligned table actions cog,
https://github.com/simonw/datasette/issues/749#issuecomment-737580084,https://api.github.com/repos/simonw/datasette/issues/749,737580084,MDEyOklzc3VlQ29tbWVudDczNzU4MDA4NA==,9599,simonw,2020-12-03T00:31:14Z,2020-12-03T00:31:14Z,OWNER,"This works!

```
/tmp % wget 'https://covid-19.datasettes.com/covid.db'
--2020-12-02 16:28:02--  https://covid-19.datasettes.com/covid.db
Resolving covid-19.datasettes.com (covid-19.datasettes.com)... 172.217.5.83
Connecting to covid-19.datasettes.com (covid-19.datasettes.com)|172.217.5.83|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘covid.db’

covid.db                [              <=>   ] 306.42M  3.27MB/s    in 98s     

2020-12-02 16:29:40 (3.13 MB/s) - ‘covid.db’ saved [321306624]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610829227,Cloud Run fails to serve database files larger than 32MB,
https://github.com/simonw/datasette/issues/749#issuecomment-737563699,https://api.github.com/repos/simonw/datasette/issues/749,737563699,MDEyOklzc3VlQ29tbWVudDczNzU2MzY5OQ==,9599,simonw,2020-12-02T23:45:42Z,2020-12-02T23:45:42Z,OWNER,"I asked about this on Twitter - https://twitter.com/steren/status/1334281184965140483

> You simply need to send the `Transfer-Encoding: chunked` header.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610829227,Cloud Run fails to serve database files larger than 32MB,
https://github.com/simonw/datasette/issues/942#issuecomment-737463116,https://api.github.com/repos/simonw/datasette/issues/942,737463116,MDEyOklzc3VlQ29tbWVudDczNzQ2MzExNg==,9599,simonw,2020-12-02T20:02:10Z,2020-12-02T20:03:01Z,OWNER,"My idea is that if you installed my proposed plugin you wouldn't need `metadata.json` at all - your metadata would instead live in a table in the connected SQLite database files - either one table per database (so the metadata can live in the same place as the data) or maybe also in a dedicated separate database file, for if you want to add metadata to an otherwise read-only database.

The plugin would then provide a UI for editing that metadata - maybe by configuring some writable canned queries or maybe something more custom than that. Or you could edit the metadata by manually editing the SQLite database file (or loading data into it using a tool like [yaml-to-sqlite](https://github.com/simonw/yaml-to-sqlite)).","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-737402392,https://api.github.com/repos/simonw/datasette/issues/942,737402392,MDEyOklzc3VlQ29tbWVudDczNzQwMjM5Mg==,9599,simonw,2020-12-02T18:08:55Z,2020-12-02T18:08:55Z,OWNER,"SQLite does let you add comments in your CREATE TABLE statements:
```sql
CREATE TABLE something (
    id integer primary key, -- integer primary key
    created text -- created date as ISO datetime
);
```
But the only mechanism for reading those back is to retrieve that `CREATE TABLE` block of SQL from the `sqlite_master` table and run a parser against it.

I've so far resisted adding a SQL syntax parser to Datasette for complexity reasons - though I'm increasingly thinking I'll need to do it at some point.

I think I'll leave this to plugins. I'm definitely going to build a plugin that lets you store metadata for tables and columns in a SQLite database table, which will then support interactively editing metadata through a UI.

A plugin which extracts column comments from the SQLite CREATE TABLE comments would be feasible too, if I design the plugin hooks well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1119#issuecomment-736142201,https://api.github.com/repos/simonw/datasette/issues/1119,736142201,MDEyOklzc3VlQ29tbWVudDczNjE0MjIwMQ==,9599,simonw,2020-12-01T00:41:14Z,2020-12-01T00:41:14Z,OWNER,"On my laptop:

<img width=""1123"" alt=""fixtures__generated_columns__1_row"" src=""https://user-images.githubusercontent.com/9599/100681839-8734eb00-3329-11eb-8aea-813bdc18efa1.png"">

https://latest.datasette.io/-/versions is running SQLite 3.27.2 at the moment so it won't show that table until it gets to 3.31.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753876808,"Include generated columns in fixtures.db, if SQLite version supports it",
https://github.com/simonw/datasette/issues/1116#issuecomment-736030599,https://api.github.com/repos/simonw/datasette/issues/1116,736030599,MDEyOklzc3VlQ29tbWVudDczNjAzMDU5OQ==,9599,simonw,2020-11-30T20:41:41Z,2020-11-30T20:41:41Z,OWNER,"Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

> ### 2018-12-01 (3.26.0)
> 
> -  Added [PRAGMA table_xinfo](https://www.sqlite.org/pragma.html#pragma_table_xinfo) that works just like [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that it also shows [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables.

CI is running 3.22.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/pull/1117#issuecomment-736029337,https://api.github.com/repos/simonw/datasette/issues/1117,736029337,MDEyOklzc3VlQ29tbWVudDczNjAyOTMzNw==,9599,simonw,2020-11-30T20:39:06Z,2020-11-30T20:39:06Z,OWNER,"Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

> ### 2018-12-01 (3.26.0)
> 
> -  Added [PRAGMA table_xinfo](https://www.sqlite.org/pragma.html#pragma_table_xinfo) that works just like [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that it also shows [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables.

CI is running 3.22.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736028726,https://api.github.com/repos/simonw/datasette/issues/1117,736028726,MDEyOklzc3VlQ29tbWVudDczNjAyODcyNg==,9599,simonw,2020-11-30T20:37:50Z,2020-11-30T20:37:50Z,OWNER,"This kind of problem is why I have a `tmate` workflow:

<img width=""1086"" alt=""Actions_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/100660845-72dff680-3307-11eb-89dc-211d226f68dd.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736023089,https://api.github.com/repos/simonw/datasette/issues/1117,736023089,MDEyOklzc3VlQ29tbWVudDczNjAyMzA4OQ==,9599,simonw,2020-11-30T20:26:27Z,2020-11-30T20:26:27Z,OWNER,"On my laptop:
```
platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
SQLite: 3.33.0
```
In CI they are all SQLite: 3.22.0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736018609,https://api.github.com/repos/simonw/datasette/issues/1117,736018609,MDEyOklzc3VlQ29tbWVudDczNjAxODYwOQ==,9599,simonw,2020-11-30T20:17:31Z,2020-11-30T20:17:31Z,OWNER,I need to replicate these failures on my laptop. My hunch is that this is down to the version of SQLite available to Python.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/issues/1116#issuecomment-736015487,https://api.github.com/repos/simonw/datasette/issues/1116,736015487,MDEyOklzc3VlQ29tbWVudDczNjAxNTQ4Nw==,9599,simonw,2020-11-30T20:11:07Z,2020-11-30T20:11:07Z,OWNER,Working on this in a pull request: https://github.com/simonw/datasette/pull/1117,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736014372,https://api.github.com/repos/simonw/datasette/issues/1116,736014372,MDEyOklzc3VlQ29tbWVudDczNjAxNDM3Mg==,9599,simonw,2020-11-30T20:08:48Z,2020-11-30T20:08:48Z,OWNER,"Ouch, the tests pass on my laptop but failed in CI: https://github.com/simonw/datasette/actions/runs/392367997

Lots of failures look like this:
```
ERROR: conn=<sqlite3.Connection object at 0x7f44f0494030>, sql = 'select rowid,  from facetable order by rowid limit 51', params = {}: near ""from"": syntax error
```
Note the `select rowid, from...` - so it looks like invalid SQL queries are being constructed maybe due to mis-detecting columns somehow.

I wonder why it didn't fail on my laptop?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736010720,https://api.github.com/repos/simonw/datasette/issues/1116,736010720,MDEyOklzc3VlQ29tbWVudDczNjAxMDcyMA==,9599,simonw,2020-11-30T20:01:53Z,2020-11-30T20:01:53Z,OWNER,"I'm OK exposing hidden columns, unless someone comes up with a pressing reason not to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736004383,https://api.github.com/repos/simonw/datasette/issues/1116,736004383,MDEyOklzc3VlQ29tbWVudDczNjAwNDM4Mw==,9599,simonw,2020-11-30T19:51:51Z,2020-11-30T19:51:51Z,OWNER,"This change will also have an impact on how hidden virtual FTS tables are displayed, since apparently those have some hidden columns: https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27searchable_fts%27%29

| cid | name | type | notnull | dflt_value | pk | hidden |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | text1 |   | 0 |   | 0 | 0 |
| 1 | text2 |   | 0 |   | 0 | 0 |
| 2 | name with . and spaces |   | 0 |   | 0 | 0 |
| 3 | searchable_fts |   | 0 |   | 0 | 1 |
| 4 | docid |   | 0 |   | 0 | 1 |
| 5 | __langid |   | 0 |   | 0 | 1 |","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-735995695,https://api.github.com/repos/simonw/datasette/issues/1116,735995695,MDEyOklzc3VlQ29tbWVudDczNTk5NTY5NQ==,9599,simonw,2020-11-30T19:34:15Z,2020-11-30T19:34:15Z,OWNER,"Generated column support was added in SQLite 3.31.0, so any unit tests I write for this should use skipIf to only run on that version or later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,