html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/1077#issuecomment-720654925,https://api.github.com/repos/simonw/datasette/issues/1077,720654925,MDEyOklzc3VlQ29tbWVudDcyMDY1NDkyNQ==,9599,simonw,2020-11-02T18:43:25Z,2020-11-02T18:43:25Z,OWNER,Demo: https://latest.datasette.io/fixtures?_bot=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1077#issuecomment-720637322,https://api.github.com/repos/simonw/datasette/issues/1077,720637322,MDEyOklzc3VlQ29tbWVudDcyMDYzNzMyMg==,9599,simonw,2020-11-02T18:09:17Z,2020-11-02T18:09:17Z,OWNER,Here's the `table_actions` implementation: 2f7731e9e5ff9b324beb5039fbe2be55d704a184,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1079#issuecomment-720110298,https://api.github.com/repos/simonw/datasette/issues/1079,720110298,MDEyOklzc3VlQ29tbWVudDcyMDExMDI5OA==,9599,simonw,2020-11-01T15:58:22Z,2020-11-01T15:58:22Z,OWNER,Might try a drop shadow on that menu too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733999615,Handle long breadcrumbs better with new menu,
https://github.com/simonw/datasette/issues/782#issuecomment-720028476,https://api.github.com/repos/simonw/datasette/issues/782,720028476,MDEyOklzc3VlQ29tbWVudDcyMDAyODQ3Ng==,9599,simonw,2020-11-01T05:00:05Z,2020-11-01T05:00:05Z,OWNER,This should be the key focus for Datasette 0.52.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/949#issuecomment-720021029,https://api.github.com/repos/simonw/datasette/issues/949,720021029,MDEyOklzc3VlQ29tbWVudDcyMDAyMTAyOQ==,9599,simonw,2020-11-01T03:29:48Z,2020-11-01T03:29:48Z,OWNER,I'm not going to do any more work on this - SQL isn't an auto-complete friendly enough language.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints,
https://github.com/simonw/datasette/issues/1077#issuecomment-720003026,https://api.github.com/repos/simonw/datasette/issues/1077,720003026,MDEyOklzc3VlQ29tbWVudDcyMDAwMzAyNg==,9599,simonw,2020-10-31T23:48:21Z,2020-10-31T23:50:07Z,OWNER,Needed by https://github.com/simonw/datasette-backup/issues/6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook,
https://github.com/simonw/datasette/issues/1047#issuecomment-719994676,https://api.github.com/repos/simonw/datasette/issues/1047,719994676,MDEyOklzc3VlQ29tbWVudDcxOTk5NDY3Ng==,9599,simonw,2020-10-31T22:11:25Z,2020-10-31T22:11:25Z,OWNER,https://docs.datasette.io/en/latest/binary_data.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895233,A new section in the docs about how Datasette handles BLOB columns,
https://github.com/simonw/datasette/issues/1027#issuecomment-719988113,https://api.github.com/repos/simonw/datasette/issues/1027,719988113,MDEyOklzc3VlQ29tbWVudDcxOTk4ODExMw==,9599,simonw,2020-10-31T21:03:37Z,2020-10-31T21:03:37Z,OWNER,"On my Mac, I run:

    datasette . -p 8009 --config base_url:/datasette-prefix/

Then I edited `/usr/local/etc/httpd/httpd.conf` and add this section:
```
LoadModule proxy_module lib/httpd/modules/mod_proxy.so
LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so
ProxyPass          /datasette-prefix/    http://localhost:8009/datasette-prefix/
```
I ran Apache in the foreground like so:

    apachectl -X

Now hitting http://localhost:8081/datasette-prefix/fixtures/compound_three_primary_keys worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-719986922,https://api.github.com/repos/simonw/datasette/issues/1023,719986922,MDEyOklzc3VlQ29tbWVudDcxOTk4NjkyMg==,9599,simonw,2020-10-31T20:51:01Z,2020-10-31T20:51:01Z,OWNER,This should all be working correctly now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/838#issuecomment-719986904,https://api.github.com/repos/simonw/datasette/issues/838,719986904,MDEyOklzc3VlQ29tbWVudDcxOTk4NjkwNA==,9599,simonw,2020-10-31T20:50:41Z,2020-10-31T20:50:41Z,OWNER,"OK, this should be working now. You can use the `datasette.urls.static_plugins()` method to generate the correct URLs in the `extra_css_urls` plugin hook: https://docs.datasette.io/en/latest/internals.html#datasette-urls","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/1041#issuecomment-719986800,https://api.github.com/repos/simonw/datasette/issues/1041,719986800,MDEyOklzc3VlQ29tbWVudDcxOTk4NjgwMA==,9599,simonw,2020-10-31T20:49:28Z,2020-10-31T20:49:28Z,OWNER,Implemented in a4ca26a2659d21779adf625183061d8879954c15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1072#issuecomment-719986698,https://api.github.com/repos/simonw/datasette/issues/1072,719986698,MDEyOklzc3VlQ29tbWVudDcxOTk4NjY5OA==,9599,simonw,2020-10-31T20:48:17Z,2020-10-31T20:48:17Z,OWNER,Here's the `datasette-edit-templates` plugin WIP I had before removing the hook: https://github.com/simonw/datasette-edit-templates/tree/82855c2612b84bc09c48fca885f831633a0d1552,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983750,https://api.github.com/repos/simonw/datasette/issues/1075,719983750,MDEyOklzc3VlQ29tbWVudDcxOTk4Mzc1MA==,9599,simonw,2020-10-31T20:22:29Z,2020-10-31T20:22:29Z,OWNER,I bet this is because I'm mucking around with one of those `__` methods. I'll try just doing the non-underscore methods instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983565,https://api.github.com/repos/simonw/datasette/issues/1075,719983565,MDEyOklzc3VlQ29tbWVudDcxOTk4MzU2NQ==,9599,simonw,2020-10-31T20:21:03Z,2020-10-31T20:21:03Z,OWNER,"Here's the output of `dir(str)`:

`['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983484,https://api.github.com/repos/simonw/datasette/issues/1075,719983484,MDEyOklzc3VlQ29tbWVudDcxOTk4MzQ4NA==,9599,simonw,2020-10-31T20:20:28Z,2020-10-31T20:20:28Z,OWNER,"It looks like this is specific to the way `PrefixedUrlString` is built.
```
(Pdb) class Weird(str): pass
(Pdb) isinstance(Weird('bob'), collections.abc.Awaitable)
False
```
So subclassing strings doesn't trigger this bug, but something about `PrefixedUrlString` causes the problem.

Here's the current `PrefixedUrlString` implementation:

https://github.com/simonw/datasette/blob/bf18b9ba175a7b25fb8b765847397dd6efb8bb7b/datasette/utils/__init__.py#L1015-L1035","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719983240,https://api.github.com/repos/simonw/datasette/issues/1075,719983240,MDEyOklzc3VlQ29tbWVudDcxOTk4MzI0MA==,9599,simonw,2020-10-31T20:18:49Z,2020-10-31T20:18:49Z,OWNER,"Here's the core problem:
```
(Pdb) isinstance('bob', collections.abc.Awaitable)
False
(Pdb) isinstance(PrefixedUrlString('bob'), collections.abc.Awaitable)
*** TypeError: issubclass() arg 1 must be a class
```
For some reason `isinstance()` does not like being handed an instance of PrefixedUrlString.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719981173,https://api.github.com/repos/simonw/datasette/issues/1075,719981173,MDEyOklzc3VlQ29tbWVudDcxOTk4MTE3Mw==,9599,simonw,2020-10-31T20:02:30Z,2020-10-31T20:03:45Z,OWNER,"I wonder how Jinja's `Markup()` class works? It uses https://pypi.org/project/MarkupSafe/

It's a subclass of `str`, defined here: https://github.com/pallets/markupsafe/blob/master/src/markupsafe/__init__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1075#issuecomment-719980742,https://api.github.com/repos/simonw/datasette/issues/1075,719980742,MDEyOklzc3VlQ29tbWVudDcxOTk4MDc0Mg==,9599,simonw,2020-10-31T19:58:57Z,2020-10-31T19:58:57Z,OWNER,"Sample traceback:
```
    <link rel=""stylesheet"" href=""{{ urls.static('app.css') }}?{{ app_css_hash }}"">
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/jinja2/asyncsupport.py:173: in auto_await
    if inspect.isawaitable(value):
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/inspect.py:226: in isawaitable
    isinstance(object, collections.abc.Awaitable))
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/abc.py:139: in __instancecheck__
    return _abc_instancecheck(cls, instance)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'collections.abc.Awaitable'>
subclass = <bound method PrefixedUrlString.__getattribute__.<locals>.method of '/-/static/app.css'>

    def __subclasscheck__(cls, subclass):
        """"""Override for issubclass(subclass, cls).""""""
>       return _abc_subclasscheck(cls, subclass)
E       TypeError: issubclass() arg 1 must be a class
```
This is within Jinja. It looks like Jinja really doesn't like methods that return non-string objects like `PrefixedUrlString`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything,
https://github.com/simonw/datasette/issues/1074#issuecomment-719977864,https://api.github.com/repos/simonw/datasette/issues/1074,719977864,MDEyOklzc3VlQ29tbWVudDcxOTk3Nzg2NA==,9599,simonw,2020-10-31T19:35:01Z,2020-10-31T19:35:01Z,OWNER,"These plugins were not designed to be actually hosted online, so they do some nasty things like linking to the made-up `plugin-example.com` domain. I should fix that.

<img width=""853"" alt=""Datasette_Fixtures__fixtures"" src=""https://user-images.githubusercontent.com/9599/97788348-7b60d800-1b75-11eb-8519-a58e52108841.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1067#issuecomment-719966176,https://api.github.com/repos/simonw/datasette/issues/1067,719966176,MDEyOklzc3VlQ29tbWVudDcxOTk2NjE3Ng==,9599,simonw,2020-10-31T17:51:31Z,2020-10-31T17:51:31Z,OWNER,"Demo:

- https://latest.datasette.io/fixtures/facetable?_bot=1
- https://latest.datasette.io/fixtures/simple_view?_bot=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1074#issuecomment-719965426,https://api.github.com/repos/simonw/datasette/issues/1074,719965426,MDEyOklzc3VlQ29tbWVudDcxOTk2NTQyNg==,9599,simonw,2020-10-31T17:45:00Z,2020-10-31T17:45:00Z,OWNER,"This is working. Go to https://latest.datasette.io/login-as-root and click the button, then visit this page to see extra content added by plugins: https://latest.datasette.io/fixtures/compound_three_primary_keys

![latest-plugins](https://user-images.githubusercontent.com/9599/97786052-19e53d00-1b66-11eb-8268-b452e08965e3.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1074#issuecomment-719963074,https://api.github.com/repos/simonw/datasette/issues/1074,719963074,MDEyOklzc3VlQ29tbWVudDcxOTk2MzA3NA==,9599,simonw,2020-10-31T17:23:48Z,2020-10-31T17:23:48Z,OWNER,"Needs a way to login as root, seeing as several plugins only show extra content if the user is logged in as root.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures,
https://github.com/simonw/datasette/issues/1067#issuecomment-719961701,https://api.github.com/repos/simonw/datasette/issues/1067,719961701,MDEyOklzc3VlQ29tbWVudDcxOTk2MTcwMQ==,9599,simonw,2020-10-31T17:11:59Z,2020-10-31T17:11:59Z,OWNER,It bothers me that these aren't visible in any public demos. Maybe `latest.datasette.io` should include the `my_plugins.py` and `my_plugins2.py` plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1026#issuecomment-719959754,https://api.github.com/repos/simonw/datasette/issues/1026,719959754,MDEyOklzc3VlQ29tbWVudDcxOTk1OTc1NA==,9599,simonw,2020-10-31T16:56:35Z,2020-10-31T16:56:35Z,OWNER,#1041 can also benefit from the string subclass that shows that `base_url` has been added.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1067#issuecomment-719959419,https://api.github.com/repos/simonw/datasette/issues/1067,719959419,MDEyOklzc3VlQ29tbWVudDcxOTk1OTQxOQ==,9599,simonw,2020-10-31T16:53:42Z,2020-10-31T16:53:42Z,OWNER,For the 0.51 release I'm going to add tests that show this works on view pages. I won't implement it for query pages.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719956184,https://api.github.com/repos/simonw/datasette/issues/1067,719956184,MDEyOklzc3VlQ29tbWVudDcxOTk1NjE4NA==,9599,simonw,2020-10-31T16:26:09Z,2020-10-31T16:26:09Z,OWNER,"Should the hook provide an indication that it's running on a different type of page? I think yes for queries. Not sure about views  - they behave very much like tables, and the plugin can always introspect to see if something is a view if it needs to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1070#issuecomment-719955724,https://api.github.com/repos/simonw/datasette/issues/1070,719955724,MDEyOklzc3VlQ29tbWVudDcxOTk1NTcyNA==,9599,simonw,2020-10-31T16:22:45Z,2020-10-31T16:22:45Z,OWNER,I've removed this plugin hook in #1073.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733390884,load_template() example in documentation showing loading from a database,
https://github.com/simonw/datasette/issues/1072#issuecomment-719955491,https://api.github.com/repos/simonw/datasette/issues/1072,719955491,MDEyOklzc3VlQ29tbWVudDcxOTk1NTQ5MQ==,9599,simonw,2020-10-31T16:20:58Z,2020-10-31T16:20:58Z,OWNER,"Here's the proof of concept `FunctionLoader` that showed me that this wasn't going to work:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 4b28e71..b076be7 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -21,7 +21,7 @@ from pathlib import Path
 from markupsafe import Markup
 from itsdangerous import URLSafeSerializer
 import jinja2
-from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
+from jinja2 import ChoiceLoader, Environment, FileSystemLoader, FunctionLoader, PrefixLoader
 from jinja2.environment import Template
 from jinja2.exceptions import TemplateNotFound
 import uvicorn
@@ -300,6 +300,7 @@ class Datasette:
         template_paths.append(default_templates)
         template_loader = ChoiceLoader(
             [
+                FunctionLoader(self._load_template_from_plugins),
                 FileSystemLoader(template_paths),
                 # Support {% extends ""default:table.html"" %}:
                 PrefixLoader(
@@ -322,6 +323,17 @@ class Datasette:
         self._root_token = secrets.token_hex(32)
         self.client = DatasetteClient(self)
 
+    def _load_template_from_plugins(self, template):
+        # ""If auto reloading is enabled it’s called to check if the template changed""
+        uptodatefunc = lambda: True
+        source = pm.hook.load_template(
+            template=template,
+            datasette=self,
+        )
+        if source is None:
+            return None
+        return source, template, uptodatefunc
+
     @property
     def urls(self):
         return Urls(self)
@@ -719,35 +731,7 @@ class Datasette:
         else:
             if isinstance(templates, str):
                 templates = [templates]
-
-            # Give plugins first chance at loading the template
-            break_outer = False
-            plugin_template_source = None
-            plugin_template_name = None
-            template_name = None
-            for template_name in templates:
-                if break_outer:
-                    break
-                plugin_template_source = pm.hook.load_template(
-                    template=template_name,
-                    request=request,
-                    datasette=self,
-                )
-                plugin_template_source = await await_me_maybe(plugin_template_source)
-                if plugin_template_source:
-                    break_outer = True
-                    plugin_template_name = template_name
-                    break
-            if plugin_template_source is not None:
-                template = self.jinja_env.from_string(plugin_template_source)
-            else:
-                template = self.jinja_env.select_template(templates)
-            for template_name in templates:
-                from_plugin = template_name == plugin_template_name
-                used = from_plugin or template_name == template.name
-                templates_considered.append(
-                    {""name"": template_name, ""used"": used, ""from_plugin"": from_plugin}
-                )
+            template = self.jinja_env.select_template(templates)
         body_scripts = []
         # pylint: disable=no-member
         for extra_script in pm.hook.extra_body_script(
diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py
index ca84b35..7804def 100644
--- a/datasette/hookspecs.py
+++ b/datasette/hookspecs.py
@@ -50,7 +50,7 @@ def extra_template_vars(
 
 
 @hookspec(firstresult=True)
-def load_template(template, request, datasette):
+def load_template(template, datasette):
     ""Load the specified template, returning the template code as a string""
 
 
diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst
index 3c57b6a..8f2704e 100644
--- a/docs/plugin_hooks.rst
+++ b/docs/plugin_hooks.rst
@@ -273,15 +273,12 @@ Example: `datasette-cluster-map <https://github.com/simonw/datasette-cluster-map
 
 .. _plugin_hook_load_template:
 
-load_template(template, request, datasette)
--------------------------------------------
+load_template(template, datasette)
+----------------------------------
 
 ``template`` - string
     The template that is being rendered, e.g. ``database.html``
 
-``request`` - object or None
-    The current HTTP :ref:`internals_request`. This can be ``None`` if the request object is not available.
-
 ``datasette`` - :ref:`internals_datasette`
     You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``
 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1073#issuecomment-719834200,https://api.github.com/repos/simonw/datasette/issues/1073,719834200,MDEyOklzc3VlQ29tbWVudDcxOTgzNDIwMA==,9599,simonw,2020-10-30T22:52:48Z,2020-10-30T22:52:48Z,OWNER,Should mostly be a case of backing out the changes from this commit: 81dea4b07ab2b6f4eaaf248307d2b588472054a1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733560417,Remove load_template plugin hook,
https://github.com/simonw/datasette/issues/1072#issuecomment-719833744,https://api.github.com/repos/simonw/datasette/issues/1072,719833744,MDEyOklzc3VlQ29tbWVudDcxOTgzMzc0NA==,9599,simonw,2020-10-30T22:50:57Z,2020-10-30T22:50:57Z,OWNER,Yeah I'm going to remove the `load_template` plugin hook and see if it's possible to build the edit templates extension against `prepare_jinja2_environment` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719833070,https://api.github.com/repos/simonw/datasette/issues/1072,719833070,MDEyOklzc3VlQ29tbWVudDcxOTgzMzA3MA==,9599,simonw,2020-10-30T22:48:04Z,2020-10-30T22:48:04Z,OWNER,"https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L310-L318

So yeah that plugin hook can probably modify the list of loaders available to the `Environment`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719832853,https://api.github.com/repos/simonw/datasette/issues/1072,719832853,MDEyOklzc3VlQ29tbWVudDcxOTgzMjg1Mw==,9599,simonw,2020-10-30T22:47:12Z,2020-10-30T22:47:12Z,OWNER,Maybe I should ditch this hook entirely in favour of the existing `prepare_jinja2_environment` hook. Could that add new template loaders?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719832651,https://api.github.com/repos/simonw/datasette/issues/1072,719832651,MDEyOklzc3VlQ29tbWVudDcxOTgzMjY1MQ==,9599,simonw,2020-10-30T22:46:25Z,2020-10-30T22:46:25Z,OWNER,"I tried using a `FunctionLoader` and got this error on startup:
```python
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 989, in __init__
    for filepath in self.ds.jinja_env.list_templates()
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/environment.py"", line 810, in list_templates
    names = self.loader.list_templates()
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py"", line 434, in list_templates
    found.update(loader.list_templates())
  File ""/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py"", line 99, in list_templates
    raise TypeError(""this loader cannot iterate over all templates"")
TypeError: this loader cannot iterate over all templates
```
So if I'm going to define a custom Jinja loader I'll need to teach plugins to answer the ""list templates"" query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719819331,https://api.github.com/repos/simonw/datasette/issues/1072,719819331,MDEyOklzc3VlQ29tbWVudDcxOTgxOTMzMQ==,9599,simonw,2020-10-30T22:00:43Z,2020-10-30T22:00:43Z,OWNER,I'll try getting that to work. If I can't get it to work I'll drop the plugin hook for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719819234,https://api.github.com/repos/simonw/datasette/issues/1072,719819234,MDEyOklzc3VlQ29tbWVudDcxOTgxOTIzNA==,9599,simonw,2020-10-30T22:00:21Z,2020-10-30T22:00:21Z,OWNER,"There might be a way to save this. Async template loading can't be supported, but what if you could define a `load_template()` hook which returned a sync function that returned templates...

Then the `datasette-edit-templates` plugin could reply to `load_template` by loading all DB templates into memory and returning a `load_template` sync function that looked up the values in those already-loaded templates.

It could even maintain an in-memory cache that gets updated when a template is edited.

If I do this, I could remove the ability to return an `async` function from `load_template()` but add that in the future should Jinja implement a mechanism for async template loading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719814279,https://api.github.com/repos/simonw/datasette/issues/1072,719814279,MDEyOklzc3VlQ29tbWVudDcxOTgxNDI3OQ==,9599,simonw,2020-10-30T21:45:33Z,2020-10-30T21:45:33Z,OWNER,Sadly I'm going to bump `load_template` from Datasette 0.51 - I don't think I should block the release on resolving this issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719813970,https://api.github.com/repos/simonw/datasette/issues/1072,719813970,MDEyOklzc3VlQ29tbWVudDcxOTgxMzk3MA==,9599,simonw,2020-10-30T21:44:40Z,2020-10-30T21:44:40Z,OWNER,"I'm pretty sure that `run_in_executor()` workaround won't work. https://github.com/django/asgiref/blob/7becc9daca2628c46af1cb7e46b4c47c1ea27adf/asgiref/sync.py#L83 for example says ""You cannot use AsyncToSync in the same thread as an async event loop"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719813212,https://api.github.com/repos/simonw/datasette/issues/1072,719813212,MDEyOklzc3VlQ29tbWVudDcxOTgxMzIxMg==,9599,simonw,2020-10-30T21:42:35Z,2020-10-30T21:42:35Z,OWNER,Filed a feature request here: https://github.com/pallets/jinja/issues/1304,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719811312,https://api.github.com/repos/simonw/datasette/issues/1072,719811312,MDEyOklzc3VlQ29tbWVudDcxOTgxMTMxMg==,9599,simonw,2020-10-30T21:36:49Z,2020-10-30T21:36:49Z,OWNER,"There's one other option: in `datasette-edit-templates` I could maybe use `asyncio.get_event_loop().run_in_executor(...)` to load the templates asynchronously within the Jinja template loader mechanism.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719810533,https://api.github.com/repos/simonw/datasette/issues/1072,719810533,MDEyOklzc3VlQ29tbWVudDcxOTgxMDUzMw==,9599,simonw,2020-10-30T21:34:38Z,2020-10-30T21:34:38Z,OWNER,"... no wait, my comments above assume that I'm just building the `datasette-edit-templates` plugin. Does this work as a general solution for all of Datasette? I don't think it does.

This may mean I need to delay the whole feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719810023,https://api.github.com/repos/simonw/datasette/issues/1072,719810023,MDEyOklzc3VlQ29tbWVudDcxOTgxMDAyMw==,9599,simonw,2020-10-30T21:33:06Z,2020-10-30T21:33:06Z,OWNER,"The ideal solution is for Jinja to offer `async` template loading. I'll file a feature request, then I'll implement the second option above (async load all templates from the DB before each render).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719809780,https://api.github.com/repos/simonw/datasette/issues/1072,719809780,MDEyOklzc3VlQ29tbWVudDcxOTgwOTc4MA==,9599,simonw,2020-10-30T21:32:28Z,2020-10-30T21:32:28Z,OWNER,"Here's an alternative that would definitely work and would be a lot simpler, at the cost of a fair amount of RAM:

1. Before rendering the template, load ALL of the most-recent-versions of the templates that are stored in the DB. Use those to populate a `DictLoader`.
2. Render the template.

This does mean loading template bodies that we won't use. Provided an instance has less than 100 templates I imagine this will work just fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719809259,https://api.github.com/repos/simonw/datasette/issues/1072,719809259,MDEyOklzc3VlQ29tbWVudDcxOTgwOTI1OQ==,9599,simonw,2020-10-30T21:31:10Z,2020-10-30T21:31:10Z,OWNER,"How can we tell what template Jinja will need to render?

One approach that could work:

1. Set up a dummy template loader which records the name of the template that was requested
2. Load the template
3. Now we know the list of templates that were requested. Async load those
4. The dummy template loader can now return the ones we have loaded. Load the template again.
5. Did it request any more templates? If so, load those, and repeat.
6. Keep on with this loop until a template load (which might even have to be a render) fails to request any templates that we have not yet loaded.
7. Render the template.

This is GROSS. It feels like a huge waste of CPU, and it could lead to very weird behaviour if any template variables have side effects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719807502,https://api.github.com/repos/simonw/datasette/issues/1072,719807502,MDEyOklzc3VlQ29tbWVudDcxOTgwNzUwMg==,9599,simonw,2020-10-30T21:26:49Z,2020-10-30T21:26:49Z,OWNER,"It looks like Jinja does not have a mechanism for asynchronous template loading - the loader API is synchronous.

One option may be to figure out which templates are needed (including inherited templates and includes) before rendering the template. Then async load those templates from the database into a `DictLoader`, then pass that `DictLoader` to Jinja.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719803880,https://api.github.com/repos/simonw/datasette/issues/1072,719803880,MDEyOklzc3VlQ29tbWVudDcxOTgwMzg4MA==,9599,simonw,2020-10-30T21:17:11Z,2020-10-30T21:17:11Z,OWNER,"Example from the Jinja docs: https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.BaseLoader

```python
from jinja2 import BaseLoader, TemplateNotFound
from os.path import join, exists, getmtime

class MyLoader(BaseLoader):

    def __init__(self, path):
        self.path = path

    def get_source(self, environment, template):
        path = join(self.path, template)
        if not exists(path):
            raise TemplateNotFound(template)
        mtime = getmtime(path)
        with file(path) as f:
            source = f.read().decode('utf-8')
        return source, path, lambda: mtime == getmtime(path)
```
Also available: `jinja2.FunctionLoader(load_func)` which lets me pass it a function like this one:
```
>>> def load_template(name):
...     if name == 'index.html':
...         return '...'
...
>>> loader = FunctionLoader(load_template)
```

Just one catch: I need to be able to load templates asynchronously, because they live in the database. Let's hope Jinja has a mechanism for that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719785005,https://api.github.com/repos/simonw/datasette/issues/1072,719785005,MDEyOklzc3VlQ29tbWVudDcxOTc4NTAwNQ==,9599,simonw,2020-10-30T20:36:22Z,2020-10-30T20:36:22Z,OWNER,"It should be easy enough to show a comment that says which original template names were considered, but I may not be able to show which one was actually used (or which ones came from plugins).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1072#issuecomment-719784606,https://api.github.com/repos/simonw/datasette/issues/1072,719784606,MDEyOklzc3VlQ29tbWVudDcxOTc4NDYwNg==,9599,simonw,2020-10-30T20:35:33Z,2020-10-30T20:35:33Z,OWNER,"To fix this I think I need to move the `load_template` implementation into a Jinja template loader.

I'm not sure I'll be able to keep the `Templates considered` comment working though:

https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L745-L750","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends,
https://github.com/simonw/datasette/issues/1071#issuecomment-719777499,https://api.github.com/repos/simonw/datasette/issues/1071,719777499,MDEyOklzc3VlQ29tbWVudDcxOTc3NzQ5OQ==,9599,simonw,2020-10-30T20:20:01Z,2020-10-30T20:20:01Z,OWNER,"Fixed: https://latest.datasette.io/-/messages

![demo-message](https://user-images.githubusercontent.com/9599/97753199-9c142980-1ab2-11eb-9be9-c0be41acc68e.gif)


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733485423,Messages should be displayed full width,
https://github.com/simonw/datasette/pull/1069#issuecomment-719672967,https://api.github.com/repos/simonw/datasette/issues/1069,719672967,MDEyOklzc3VlQ29tbWVudDcxOTY3Mjk2Nw==,9599,simonw,2020-10-30T16:58:01Z,2020-10-30T16:58:01Z,OWNER,"OK, new hook specification is:
```python
@hookspec(firstresult=True)
def load_template(template, request, datasette):
    ""Load the specified template, returning the template code as a string""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/pull/1069#issuecomment-719670714,https://api.github.com/repos/simonw/datasette/issues/1069,719670714,MDEyOklzc3VlQ29tbWVudDcxOTY3MDcxNA==,9599,simonw,2020-10-30T16:53:56Z,2020-10-30T16:53:56Z,OWNER,"I'm having second thoughts about the design of the plugin hook.

Consider the following:
```python
            plugin_template_source = pm.hook.load_template(
                template=template_name,
                database=context.get(""database""),
                table=context.get(""table""),
                columns=context.get(""columns""),
                view_name=self.name,
                request=request,
                datasette=self.ds,
            )
```
It's a bit gross that `database`, `table` and `columns` are pulled out of the context like that. This doesn't make sense for pages that are rendered by plugins, for example.

So maybe for the first release of this plugin hook I should cut it down to just seeing `template`, `request` and `datasette`. I can add the table/view/etc stuff back in later if it turns out to be necessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/pull/1069#issuecomment-719666912,https://api.github.com/repos/simonw/datasette/issues/1069,719666912,MDEyOklzc3VlQ29tbWVudDcxOTY2NjkxMg==,9599,simonw,2020-10-30T16:47:44Z,2020-10-30T16:47:44Z,OWNER,"Bringing over a comment from #1042:

> I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class.
> 
> This code is the problem:
> 
> https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133
> 
> I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/pull/1069#issuecomment-719664530,https://api.github.com/repos/simonw/datasette/issues/1069,719664530,MDEyOklzc3VlQ29tbWVudDcxOTY2NDUzMA==,9599,simonw,2020-10-30T16:43:40Z,2020-10-30T16:43:40Z,OWNER,I should include an example in the documentation that shows loading templates from a database table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/pull/1069#issuecomment-719640430,https://api.github.com/repos/simonw/datasette/issues/1069,719640430,MDEyOklzc3VlQ29tbWVudDcxOTY0MDQzMA==,9599,simonw,2020-10-30T16:01:13Z,2020-10-30T16:01:13Z,OWNER,Next steps: build a demonstration plugin against this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook,
https://github.com/simonw/datasette/issues/1068#issuecomment-719630745,https://api.github.com/repos/simonw/datasette/issues/1068,719630745,MDEyOklzc3VlQ29tbWVudDcxOTYzMDc0NQ==,9599,simonw,2020-10-30T15:44:13Z,2020-10-30T15:44:13Z,OWNER,Documentation: https://docs.datasette.io/en/latest/authentication.html#debug-menu,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719332460,https://api.github.com/repos/simonw/datasette/issues/1068,719332460,MDEyOklzc3VlQ29tbWVudDcxOTMzMjQ2MA==,9599,simonw,2020-10-30T07:13:10Z,2020-10-30T07:13:10Z,OWNER,I mainly want this so I can add that debug menu to my Dogsheep.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719331236,https://api.github.com/repos/simonw/datasette/issues/1068,719331236,MDEyOklzc3VlQ29tbWVudDcxOTMzMTIzNg==,9599,simonw,2020-10-30T07:11:58Z,2020-10-30T07:11:58Z,OWNER,Document the new permission here: https://docs.datasette.io/en/stable/authentication.html#built-in-permissions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719329219,https://api.github.com/repos/simonw/datasette/issues/1068,719329219,MDEyOklzc3VlQ29tbWVudDcxOTMyOTIxOQ==,9599,simonw,2020-10-30T07:09:59Z,2020-10-30T07:09:59Z,OWNER,Permission idea: `debug-menu`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1068#issuecomment-719328661,https://api.github.com/repos/simonw/datasette/issues/1068,719328661,MDEyOklzc3VlQ29tbWVudDcxOTMyODY2MQ==,9599,simonw,2020-10-30T07:09:30Z,2020-10-30T07:09:30Z,OWNER,Then this can make it available to root: https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_permissions.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission ,
https://github.com/simonw/datasette/issues/1067#issuecomment-719322666,https://api.github.com/repos/simonw/datasette/issues/1067,719322666,MDEyOklzc3VlQ29tbWVudDcxOTMyMjY2Ng==,9599,simonw,2020-10-30T07:04:02Z,2020-10-30T07:04:02Z,OWNER,"Maybe rename it to `actions_menu` and have it work for database, view, table and query pages using different arguments on each.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/1067#issuecomment-719320948,https://api.github.com/repos/simonw/datasette/issues/1067,719320948,MDEyOklzc3VlQ29tbWVudDcxOTMyMDk0OA==,9599,simonw,2020-10-30T07:02:37Z,2020-10-30T07:02:37Z,OWNER,"Yes, this should be possible - no point restricting what plugin authors can do with the feature. Will need to add some extra arguments to the plugin hook for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages",
https://github.com/simonw/datasette/issues/690#issuecomment-719195346,https://api.github.com/repos/simonw/datasette/issues/690,719195346,MDEyOklzc3VlQ29tbWVudDcxOTE5NTM0Ng==,9599,simonw,2020-10-30T05:20:42Z,2020-10-30T05:20:42Z,OWNER,"I've now added two new plugin hooks: [menu_links()](https://docs.datasette.io/en/latest/plugin_hooks.html#menu-links-datasette-actor) and [table_actions()](https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table).

I'm going to close this issue. Further work (on column actions and and database actions) can happen in separate tickets, but I won't include them in Datasette 0.51 since they're much less interesting than table and instance actions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/1066#issuecomment-719194756,https://api.github.com/repos/simonw/datasette/issues/1066,719194756,MDEyOklzc3VlQ29tbWVudDcxOTE5NDc1Ng==,9599,simonw,2020-10-30T05:18:35Z,2020-10-30T05:18:35Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/issues/1066#issuecomment-719194619,https://api.github.com/repos/simonw/datasette/issues/1066,719194619,MDEyOklzc3VlQ29tbWVudDcxOTE5NDYxOQ==,9599,simonw,2020-10-30T05:18:04Z,2020-10-30T05:18:04Z,OWNER,"The cog only appears if at least one table action has been registered by a plugin. It looks like this:

![table-actions](https://user-images.githubusercontent.com/9599/97662535-9bd54900-1a34-11eb-8e0f-56d159c8834e.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/issues/1066#issuecomment-719154646,https://api.github.com/repos/simonw/datasette/issues/1066,719154646,MDEyOklzc3VlQ29tbWVudDcxOTE1NDY0Ng==,9599,simonw,2020-10-30T03:48:15Z,2020-10-30T03:48:15Z,OWNER,This will use a very similar implementation to the navigation menu in #1064 - similar plugin hook and I'll use a `<details><summary>` to implement it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719117185,https://api.github.com/repos/simonw/datasette/issues/1064,719117185,MDEyOklzc3VlQ29tbWVudDcxOTExNzE4NQ==,9599,simonw,2020-10-30T01:35:17Z,2020-10-30T01:35:17Z,OWNER,"I'm going to go with a list of `{""label"": ..., ""href"": ...}` as the first iteration of this. The logout link will not be returned as part of the plugin output. A default plugin will provide the debug tools if the user is logged in as root.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719111597,https://api.github.com/repos/simonw/datasette/issues/1064,719111597,MDEyOklzc3VlQ29tbWVudDcxOTExMTU5Nw==,9599,simonw,2020-10-30T01:15:05Z,2020-10-30T01:15:05Z,OWNER,I'm torn on this one. I think I have a very slight preference for plugins returning structured objects as opposed to HTML. Less likely to regret that choice in the future?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719111373,https://api.github.com/repos/simonw/datasette/issues/1064,719111373,MDEyOklzc3VlQ29tbWVudDcxOTExMTM3Mw==,9599,simonw,2020-10-30T01:14:13Z,2020-10-30T01:14:13Z,OWNER,"Plugins returning HTML makes more sense for some of the other areas that plugins will be able to inject content - e.g. injecting content on the table or row page above the table.

If I'm going to have that as a pattern though it may make sense to use HTML here, since that will be consistent with other places that plugins can inject additional content.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719110808,https://api.github.com/repos/simonw/datasette/issues/1064,719110808,MDEyOklzc3VlQ29tbWVudDcxOTExMDgwOA==,9599,simonw,2020-10-30T01:12:09Z,2020-10-30T01:12:19Z,OWNER,Or... plugins could return HTML - maybe optionally using helper functions to generate common HTML such that plugins which use the helpers can have their HTML modified in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719110582,https://api.github.com/repos/simonw/datasette/issues/1064,719110582,MDEyOklzc3VlQ29tbWVudDcxOTExMDU4Mg==,9599,simonw,2020-10-30T01:11:13Z,2020-10-30T01:11:13Z,OWNER,"Should plugins be able to add forms like the logout form here, or should they be restricted to adding navigation links?

I can't think of a reason a plugin would need to add a form. The logout form is a special case to protect against logout-csrf attacks.

So I think plugins get to return a list of dictionaries, each with a `label` and an `href`:

```python
return [{
  ""label"": ""Upload CSVs"",
  ""href"": datasette.urls.path(""/-/upload-csvs"")
}]
```
But... is there an argument for returning headings, to divide up the menu?

I think so. I also like the idea that a default plugin checks for the `root` user and outputs links to the different debugging tools - maybe those should be wrapped in a section heading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719109770,https://api.github.com/repos/simonw/datasette/issues/1064,719109770,MDEyOklzc3VlQ29tbWVudDcxOTEwOTc3MA==,9599,simonw,2020-10-30T01:08:14Z,2020-10-30T01:08:14Z,OWNER,"How should the plugin hook work?

Here's the first version of the HTML:
```html
        <div class=""nav-menu-inner"">
            <ul>
                <li><a href=""{{ urls.instance() }}"">Home</a></li>
                <li><a href=""{{ urls.path('/-/plugins') }}"">Installed plugins</a></li>
                <li><a href=""{{ urls.path('/-/versions') }}"">Software versions</a></li>
                <li><a href=""{{ urls.path('/-/metadata') }}"">Metadata</a></li>
                {% if show_logout %}
                <form action=""{{ urls.logout() }}"" method=""post"">
                    <input type=""hidden"" name=""csrftoken"" value=""{{ csrftoken() }}"">
                    <button class=""button-as-link"">Log out</button>
                </form>{% endif %}
            </ul>
        </div>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719106174,https://api.github.com/repos/simonw/datasette/issues/1064,719106174,MDEyOklzc3VlQ29tbWVudDcxOTEwNjE3NA==,9599,simonw,2020-10-30T00:55:12Z,2020-10-30T00:55:12Z,OWNER,"So what should go in this menu?

If the user is logged in as root, I'll link to the various debug pages.

If they're not logged in at all I don't think the menu should appear.

If they are logged in as anyone, it should display to give them access to the ""log out"" button.

Plugins can add links to it. If those plugins add links, the menu will display.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719105641,https://api.github.com/repos/simonw/datasette/issues/1064,719105641,MDEyOklzc3VlQ29tbWVudDcxOTEwNTY0MQ==,9599,simonw,2020-10-30T00:53:00Z,2020-10-30T00:53:00Z,OWNER,Tips for making this accessible: https://css-tricks.com/accessible-svgs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719104883,https://api.github.com/repos/simonw/datasette/issues/1064,719104883,MDEyOklzc3VlQ29tbWVudDcxOTEwNDg4Mw==,9599,simonw,2020-10-30T00:50:01Z,2020-10-30T00:52:29Z,OWNER,"Here's what the prototype looks like so far:

![menu](https://user-images.githubusercontent.com/9599/97647443-7eda4f00-1a0f-11eb-8d78-b703b7a13616.gif)

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1064#issuecomment-719105197,https://api.github.com/repos/simonw/datasette/issues/1064,719105197,MDEyOklzc3VlQ29tbWVudDcxOTEwNTE5Nw==,9599,simonw,2020-10-30T00:51:16Z,2020-10-30T00:51:16Z,OWNER,"I used a `<details><summary>` for this: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L16-L36

I added a bit of JavaScript so that clicking outside the menu would close it: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L59-L74","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook,
https://github.com/simonw/datasette/issues/1034#issuecomment-719094027,https://api.github.com/repos/simonw/datasette/issues/1034,719094027,MDEyOklzc3VlQ29tbWVudDcxOTA5NDAyNw==,9599,simonw,2020-10-30T00:11:17Z,2020-10-30T00:11:17Z,OWNER,"Demos:

https://latest.datasette.io/fixtures/binary_data.csv?_size=max

```csv
rowid,data
1,http://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data
2,http://latest.datasette.io/fixtures/binary_data/2.blob?_blob_column=data
3,
```

https://latest.datasette.io/fixtures.csv?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max

```csv
rowid,data
1,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d
2,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=b835b0483cedb86130b9a2c280880bf5fadc5318ddf8c18d0df5204d40df1724
3,
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1063#issuecomment-719066706,https://api.github.com/repos/simonw/datasette/issues/1063,719066706,MDEyOklzc3VlQ29tbWVudDcxOTA2NjcwNg==,9599,simonw,2020-10-29T22:46:28Z,2020-10-29T22:46:28Z,OWNER,I'm not going to do the base64 thing unless someone asks for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/issues/1051#issuecomment-719053669,https://api.github.com/repos/simonw/datasette/issues/1051,719053669,MDEyOklzc3VlQ29tbWVudDcxOTA1MzY2OQ==,9599,simonw,2020-10-29T22:12:16Z,2020-10-29T22:12:16Z,OWNER,"https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+101 now looks like this:

<img width=""816"" alt=""fixtures__select_rowid__data_from_binary_data_order_by_rowid_limit_101_and_understanding_Google_auths_-_for_your_information"" src=""https://user-images.githubusercontent.com/9599/97638086-15e7dc80-19f9-11eb-90a9-686e012c4db9.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1034#issuecomment-719050754,https://api.github.com/repos/simonw/datasette/issues/1034,719050754,MDEyOklzc3VlQ29tbWVudDcxOTA1MDc1NA==,9599,simonw,2020-10-29T22:04:52Z,2020-10-29T22:04:52Z,OWNER,I'm going to link to. the new `.blob` representation using the new `?_blob_hash=xxx` argument to ensure that the content served is the expected binary blob.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1063#issuecomment-719050390,https://api.github.com/repos/simonw/datasette/issues/1063,719050390,MDEyOklzc3VlQ29tbWVudDcxOTA1MDM5MA==,9599,simonw,2020-10-29T22:04:00Z,2020-10-29T22:04:00Z,OWNER,This will close #1034.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/pull/1061#issuecomment-719042601,https://api.github.com/repos/simonw/datasette/issues/1061,719042601,MDEyOklzc3VlQ29tbWVudDcxOTA0MjYwMQ==,9599,simonw,2020-10-29T21:45:35Z,2020-10-29T21:50:42Z,OWNER,Moving the CSV work to a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer,
https://github.com/simonw/datasette/issues/1063#issuecomment-719043108,https://api.github.com/repos/simonw/datasette/issues/1063,719043108,MDEyOklzc3VlQ29tbWVudDcxOTA0MzEwOA==,9599,simonw,2020-10-29T21:46:48Z,2020-10-29T21:46:48Z,OWNER,Remove this `xfail` and `import pytest`: https://github.com/simonw/datasette/blob/503a5b7b4080a26ef9ceb1ecd1a4a6f4ef4ffc59/tests/test_csv.py#L83-L96,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads,
https://github.com/simonw/datasette/pull/1061#issuecomment-719035336,https://api.github.com/repos/simonw/datasette/issues/1061,719035336,MDEyOklzc3VlQ29tbWVudDcxOTAzNTMzNg==,9599,simonw,2020-10-29T21:29:29Z,2020-10-29T21:29:29Z,OWNER,Those display_rows have already been processed by the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/database.py#L320-L346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer,
https://github.com/simonw/datasette/pull/1061#issuecomment-719033013,https://api.github.com/repos/simonw/datasette/issues/1061,719033013,MDEyOklzc3VlQ29tbWVudDcxOTAzMzAxMw==,9599,simonw,2020-10-29T21:27:14Z,2020-10-29T21:27:14Z,OWNER,"Next challenge: link to `.blob` downloads from https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data

This will be a bit tricky. Here's how that template works at the moment: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/templates/query.html#L69-L77","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer,
https://github.com/simonw/datasette/issues/1062#issuecomment-719031901,https://api.github.com/repos/simonw/datasette/issues/1062,719031901,MDEyOklzc3VlQ29tbWVudDcxOTAzMTkwMQ==,9599,simonw,2020-10-29T21:25:54Z,2020-10-29T21:25:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/base.py#L258-L345,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1050#issuecomment-719021514,https://api.github.com/repos/simonw/datasette/issues/1050,719021514,MDEyOklzc3VlQ29tbWVudDcxOTAyMTUxNA==,9599,simonw,2020-10-29T21:05:08Z,2020-10-29T21:05:08Z,OWNER,"Idea: what if Datasette had a custom SQLite function that could be used to generate URLs to the row-level BLOB download for a value? Then custom SQL query authors could use that function to link to the relevant content.

This could be expanded to exposing other `datasette.urls` functionality as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-719001701,https://api.github.com/repos/simonw/datasette/issues/1050,719001701,MDEyOklzc3VlQ29tbWVudDcxOTAwMTcwMQ==,9599,simonw,2020-10-29T20:26:44Z,2020-10-29T20:26:44Z,OWNER,I'll do the rest of the work on this in the pull request #1061.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718989895,https://api.github.com/repos/simonw/datasette/issues/1050,718989895,MDEyOklzc3VlQ29tbWVudDcxODk4OTg5NQ==,9599,simonw,2020-10-29T20:04:15Z,2020-10-29T20:04:15Z,OWNER,I'll use `hashlib.sha256` for these hashes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718987852,https://api.github.com/repos/simonw/datasette/issues/1050,718987852,MDEyOklzc3VlQ29tbWVudDcxODk4Nzg1Mg==,9599,simonw,2020-10-29T20:00:32Z,2020-10-29T20:00:32Z,OWNER,"The reason I like the `?_blob_hash=` solution is that it feels really misleading to provide a link to ""download this binary"" which could conceivably download some other data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718980944,https://api.github.com/repos/simonw/datasette/issues/1050,718980944,MDEyOklzc3VlQ29tbWVudDcxODk4MDk0NA==,9599,simonw,2020-10-29T19:46:19Z,2020-10-29T19:46:19Z,OWNER,"Had an idea in https://github.com/simonw/datasette/issues/1051#issuecomment-718980659

> OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once.
> 
> For an arbitrary SQL query, how about if I link to this?
> 
> `/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666`
> 
> Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data?
> 
> If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated).
> 
> I think this might be the best solution to the problem.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1051#issuecomment-718980659,https://api.github.com/repos/simonw/datasette/issues/1051,718980659,MDEyOklzc3VlQ29tbWVudDcxODk4MDY1OQ==,9599,simonw,2020-10-29T19:45:42Z,2020-10-29T19:45:42Z,OWNER,"OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once. 

For an arbitrary SQL query, how about if I link to this?

`/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666`

Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data?

If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated).

I think this might be the best solution to the problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1053#issuecomment-718976679,https://api.github.com/repos/simonw/datasette/issues/1053,718976679,MDEyOklzc3VlQ29tbWVudDcxODk3NjY3OQ==,9599,simonw,2020-10-29T19:37:57Z,2020-10-29T19:37:57Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#designing-urls-for-your-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729604838,Document recommendations for plugin authors to design URLs,
https://github.com/simonw/datasette/issues/1050#issuecomment-718346019,https://api.github.com/repos/simonw/datasette/issues/1050,718346019,MDEyOklzc3VlQ29tbWVudDcxODM0NjAxOQ==,9599,simonw,2020-10-29T04:05:07Z,2020-10-29T04:05:07Z,OWNER,"Yes, confirmed - this is a bug where if the `BLOB` column contains a `null` you get a nasty exception if you try to download it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-718342036,https://api.github.com/repos/simonw/datasette/issues/1050,718342036,MDEyOklzc3VlQ29tbWVudDcxODM0MjAzNg==,9599,simonw,2020-10-29T03:49:57Z,2020-10-29T03:49:57Z,OWNER,"@thadk from that error it looks like the problem may have been that you had a BLOB column containing a `null` value? If so that's definitely a bug, I'll fix that.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/pull/1049#issuecomment-718340847,https://api.github.com/repos/simonw/datasette/issues/1049,718340847,MDEyOklzc3VlQ29tbWVudDcxODM0MDg0Nw==,9599,simonw,2020-10-29T03:45:47Z,2020-10-29T03:48:26Z,OWNER,"[thebe](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html) is the first time I've seen a library that requires you to set up some global JavaScript configuration before loading the script itself.

I'm hesitant to add an extra template block just to cover that one case since it's such a rare pattern. But it's important that `thebelab` can be used with Datasette.

Would this pattern work for you instead?

```html+jinja
{% block extra_head %}
<script type=""text/x-thebe-config"">
  {
    requestKernel: true,
    binderOptions: {
      repo: ""binder-examples/requirements"",
    },
  }
</script>
<script src=""https://unpkg.com/thebelab@latest/lib/index.js""></script>
{% endblock %}
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders,
https://github.com/simonw/datasette/pull/1049#issuecomment-718341542,https://api.github.com/repos/simonw/datasette/issues/1049,718341542,MDEyOklzc3VlQ29tbWVudDcxODM0MTU0Mg==,9599,simonw,2020-10-29T03:48:12Z,2020-10-29T03:48:12Z,OWNER,"You could use Datasette's new `{{ urls.static_plugins(...) }}` template option - see https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls - to generate a link to code that was bundled with the plugin:

```html+jinja
{% block extra_head %}
<script type=""text/x-thebe-config"">
  {
    requestKernel: true,
    binderOptions: {
      repo: ""binder-examples/requirements"",
    },
  }
</script>
<script src=""{{ urls.static_plugins(""datasette-thebelab-plugin"", ""thebelab-index.js"")""></script>
{% endblock %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders,
https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718170295,https://api.github.com/repos/simonw/sqlite-utils/issues/191,718170295,MDEyOklzc3VlQ29tbWVudDcxODE3MDI5NQ==,9599,simonw,2020-10-28T19:50:16Z,2020-10-28T19:50:16Z,OWNER,"I think I made a mistake when I designed the initial decorator. I should have had it work like this:

```python
@db.register_function()
def reverse_string(s):
    return """".join(reversed(list(s)))
```
As this leaves open the option to add new parameters in the future.

To avoid breaking backwards compatibility I'll use the hack that detects the argument this time, but in the future I'll try to remember to always design decorators to be called like `@decorator()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731740458,Idea: @db.register_function(deterministic=True),
https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718168730,https://api.github.com/repos/simonw/sqlite-utils/issues/191,718168730,MDEyOklzc3VlQ29tbWVudDcxODE2ODczMA==,9599,simonw,2020-10-28T19:47:20Z,2020-10-28T19:47:20Z,OWNER,"https://stackoverflow.com/a/3931903 looks useful:
```python
def trace(*args):
    def _trace(func):
        def wrapper(*args, **kwargs):
            print enter_string
            func(*args, **kwargs)
            print exit_string
        return wrapper
    if len(args) == 1 and callable(args[0]):
        # No arguments, this is the decorator
        # Set default values for the arguments
        enter_string = 'entering'
        exit_string = 'exiting'
        return _trace(args[0])
    else:
        # This is just returning the decorator
        enter_string, exit_string = args
        return _trace
```
Can improve that code with `functools.wraps`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731740458,Idea: @db.register_function(deterministic=True),
https://github.com/simonw/datasette/pull/1059#issuecomment-718078447,https://api.github.com/repos/simonw/datasette/issues/1059,718078447,MDEyOklzc3VlQ29tbWVudDcxODA3ODQ0Nw==,9599,simonw,2020-10-28T17:07:59Z,2020-10-28T17:08:14Z,OWNER,"> #### 0.6.0 (2020-10-27)
> 
> -   aiofiles is now tested on ppc64le.
> -   Added name and mode properties to async file objects. [#82](https://github.com/Tinche/aiofiles/pull/82)
> -   Fixed a DeprecationWarning internally. [#75](https://github.com/Tinche/aiofiles/pull/75)
> -   Python 3.9 support and tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731445447,"Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7",
https://github.com/simonw/datasette/issues/1057#issuecomment-717531272,https://api.github.com/repos/simonw/datasette/issues/1057,717531272,MDEyOklzc3VlQ29tbWVudDcxNzUzMTI3Mg==,9599,simonw,2020-10-27T20:51:09Z,2020-10-27T20:51:09Z,OWNER,"That works!

<img width=""1370"" alt=""Banners_and_Alerts_and_SQLite___Mike_Bostock___Observable"" src=""https://user-images.githubusercontent.com/9599/97360439-717c6380-185b-11eb-9996-e0064e4b22d4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730797787,--cors should enable /fixtures.db CORS access,
https://github.com/simonw/datasette/issues/1058#issuecomment-717527606,https://api.github.com/repos/simonw/datasette/issues/1058,717527606,MDEyOklzc3VlQ29tbWVudDcxNzUyNzYwNg==,9599,simonw,2020-10-27T20:44:06Z,2020-10-27T20:44:06Z,OWNER,Example: https://github.com/simonw/datasette/blob/5a1519796037105bc20bcf2f91a76e022926c204/datasette/views/database.py#L26-L32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730802994,Database download should implement cascading permissions,
https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717361487,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717361487,MDEyOklzc3VlQ29tbWVudDcxNzM2MTQ4Nw==,9599,simonw,2020-10-27T16:24:04Z,2020-10-27T16:24:04Z,OWNER,"This is great, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records,
https://github.com/simonw/datasette/issues/1054#issuecomment-717051707,https://api.github.com/repos/simonw/datasette/issues/1054,717051707,MDEyOklzc3VlQ29tbWVudDcxNzA1MTcwNw==,9599,simonw,2020-10-27T07:41:21Z,2020-10-27T07:41:21Z,OWNER,Essentially it's this problem: https://github.com/python-versioneer/python-versioneer/issues/140,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py,
https://github.com/simonw/datasette/issues/1054#issuecomment-717050585,https://api.github.com/repos/simonw/datasette/issues/1054,717050585,MDEyOklzc3VlQ29tbWVudDcxNzA1MDU4NQ==,9599,simonw,2020-10-27T07:38:50Z,2020-10-27T07:38:50Z,OWNER,"Maybe imitate how Django does this, e.g. https://github.com/django/django/commit/6b9b2af7352908d40ca4d31bdb1b80c013cab29a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py,
https://github.com/simonw/sqlite-utils/pull/189#issuecomment-716756103,https://api.github.com/repos/simonw/sqlite-utils/issues/189,716756103,MDEyOklzc3VlQ29tbWVudDcxNjc1NjEwMw==,9599,simonw,2020-10-26T18:56:19Z,2020-10-26T18:56:19Z,OWNER,"This is a great fix, thanks! If you add a unit test somewhere in here I'll merge the PR:  https://github.com/simonw/sqlite-utils/blob/main/tests/test_m2m.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records,
https://github.com/simonw/datasette/issues/1051#issuecomment-716681602,https://api.github.com/repos/simonw/datasette/issues/1051,716681602,MDEyOklzc3VlQ29tbWVudDcxNjY4MTYwMg==,9599,simonw,2020-10-26T16:51:58Z,2020-10-26T16:51:58Z,OWNER,"I still need to improve the current binary display on the query page though, where it outputs a Python `b'...'` literal.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-716681167,https://api.github.com/repos/simonw/datasette/issues/1051,716681167,MDEyOklzc3VlQ29tbWVudDcxNjY4MTE2Nw==,9599,simonw,2020-10-26T16:51:15Z,2020-10-26T16:51:15Z,OWNER,"Crazy idea: generate a signed URL containing a base64 of the gzip of the binary content (to try and reduce size).

No: this will blow through URL limits in various hosting providers and possibly even browsers. It could be made to work a little bit more reliably with some extra JavaScript that turns it into a download on the browser-side, but that would be hideously complicated.

Also the signed bit doesn't prevent people from generating SQL queries that generate nasty binary blobs for download.

I'm beginning to think that restricting this feature to just table view, not query view, is a better idea. Query view can still get at the binary using JSON and base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/976#issuecomment-716305890,https://api.github.com/repos/simonw/datasette/issues/976,716305890,MDEyOklzc3VlQ29tbWVudDcxNjMwNTg5MA==,9599,simonw,2020-10-26T05:07:10Z,2020-10-26T05:07:10Z,OWNER,"I used the new `datasette.urls` methods to handle escaping table names.

https://github.com/simonw/datasette/blob/f5dbe61a4568c0915ec6be820095c2960cf0857c/datasette/utils/__init__.py#L996-L1008","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708289783,Idea: -o could open to a more convenient location,
https://github.com/simonw/datasette/issues/1052#issuecomment-716265360,https://api.github.com/repos/simonw/datasette/issues/1052,716265360,MDEyOklzc3VlQ29tbWVudDcxNjI2NTM2MA==,9599,simonw,2020-10-26T02:17:58Z,2020-10-26T02:17:58Z,OWNER,"The default z-index values for Leaflet are defined here: https://github.com/Leaflet/Leaflet/blob/b346bb8bf7bb80899baa1f4fc1536bae58e7e3e6/dist/leaflet.css#L81-L91

```css
.leaflet-pane         { z-index: 400; }

.leaflet-tile-pane    { z-index: 200; }
.leaflet-overlay-pane { z-index: 400; }
.leaflet-shadow-pane  { z-index: 500; }
.leaflet-marker-pane  { z-index: 600; }
.leaflet-tooltip-pane   { z-index: 650; }
.leaflet-popup-pane   { z-index: 700; }

.leaflet-map-pane canvas { z-index: 100; }
.leaflet-map-pane svg    { z-index: 200; }
```
So a `z-index` of 1000 on the menu should fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729183332,Column action menu overlapped by Leaflet maps,
https://github.com/simonw/datasette/issues/1051#issuecomment-716204271,https://api.github.com/repos/simonw/datasette/issues/1051,716204271,MDEyOklzc3VlQ29tbWVudDcxNjIwNDI3MQ==,9599,simonw,2020-10-25T20:08:04Z,2020-10-25T20:08:04Z,OWNER,"This is bad though, because if I want to provide binary data in CSV as requested in #1034 I need some way of providing that data.

Which suggests to me that the base64 option is the only one that can make sense for arbitrary SQL queries represented as CSV. Download links won't work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1051#issuecomment-716204090,https://api.github.com/repos/simonw/datasette/issues/1051,716204090,MDEyOklzc3VlQ29tbWVudDcxNjIwNDA5MA==,9599,simonw,2020-10-25T20:06:42Z,2020-10-25T20:06:42Z,OWNER,"Providing a binary download link here is actually extremely difficult.

The problem is that the SQL query itself represents data that can change from one moment to the next. It's no good showing a ""Binary: 55 bytes"" message that links to that same SQL query but with a `.blob` extension and arguments to select the particular result, because the data may change in a way that causes that query to return a different row - at which point the download link will give you the wrong data, not the 55 bytes you asked for.

So providing a download link risks being misleading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page,
https://github.com/simonw/datasette/issues/1050#issuecomment-716174203,https://api.github.com/repos/simonw/datasette/issues/1050,716174203,MDEyOklzc3VlQ29tbWVudDcxNjE3NDIwMw==,9599,simonw,2020-10-25T16:27:39Z,2020-10-25T16:53:27Z,OWNER,"Idea: `.blob` output rendererer, where you tell it which column you want using `?_blob_column=x`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1050#issuecomment-716175236,https://api.github.com/repos/simonw/datasette/issues/1050,716175236,MDEyOklzc3VlQ29tbWVudDcxNjE3NTIzNg==,9599,simonw,2020-10-25T16:35:20Z,2020-10-25T16:35:20Z,OWNER,"This is clearly a better solution than the one I  implemented in #1040 - I don't have to add a new route, I don't have to implement permission checks, it reuses mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078777,https://api.github.com/repos/simonw/datasette/issues/1034,716078777,MDEyOklzc3VlQ29tbWVudDcxNjA3ODc3Nw==,9599,simonw,2020-10-25T01:25:11Z,2020-10-25T01:25:11Z,OWNER,"SQLite actually has APIs that could help here: https://www.sqlite.org/c3ref/column_database_name.html - for any given SQL query they identify the origin/table/column that is the source of each resulting column.

Those aren't exposed in the Python `sqlite3` module though, so using them could be extremely tricky.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078605,https://api.github.com/repos/simonw/datasette/issues/1034,716078605,MDEyOklzc3VlQ29tbWVudDcxNjA3ODYwNQ==,9599,simonw,2020-10-25T01:22:22Z,2020-10-25T01:22:22Z,OWNER,For arbitrary CSV the only solution I can think of is to embed the base64 value.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078512,https://api.github.com/repos/simonw/datasette/issues/1034,716078512,MDEyOklzc3VlQ29tbWVudDcxNjA3ODUxMg==,9599,simonw,2020-10-25T01:21:11Z,2020-10-25T01:21:11Z,OWNER,"What should happen for CSV export of arbitrary SQL queries, where there's no obvious BLOB to link to?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716078420,https://api.github.com/repos/simonw/datasette/issues/1034,716078420,MDEyOklzc3VlQ29tbWVudDcxNjA3ODQyMA==,9599,simonw,2020-10-25T01:20:00Z,2020-10-25T01:20:00Z,OWNER,That documentation: https://docs.datasette.io/en/latest/internals.html#absolute-url-request-path,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077541,https://api.github.com/repos/simonw/datasette/issues/1034,716077541,MDEyOklzc3VlQ29tbWVudDcxNjA3NzU0MQ==,9599,simonw,2020-10-25T01:09:38Z,2020-10-25T01:10:04Z,OWNER,I should turn `datasette.absolute_url(...)` into a documented internal API on https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077508,https://api.github.com/repos/simonw/datasette/issues/1034,716077508,MDEyOklzc3VlQ29tbWVudDcxNjA3NzUwOA==,9599,simonw,2020-10-25T01:09:17Z,2020-10-25T01:09:17Z,OWNER,Here's how those absolute `next_url` values are generated: https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/views/table.py#L774-L776,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-716077436,https://api.github.com/repos/simonw/datasette/issues/1034,716077436,MDEyOklzc3VlQ29tbWVudDcxNjA3NzQzNg==,9599,simonw,2020-10-25T01:08:35Z,2020-10-25T01:08:42Z,OWNER,"This is actually a bit tricky to implement, for a few reasons:

- Need to generate a full URL, including the `https://host/` bit. I've done this for `next_url` in the JSON output before, thankfully.
- This only makes sense for CSV output for tables. If it's the CSV output of an arbitrary query there's no `/db/table/-/blob/pk/column.blob` page for me to link to.
- Need to generate those `/.../-/blob/...` URLs for the data that is being output as CSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713277810,https://api.github.com/repos/simonw/datasette/issues/1034,713277810,MDEyOklzc3VlQ29tbWVudDcxMzI3NzgxMA==,9599,simonw,2020-10-21T03:40:50Z,2020-10-25T01:01:23Z,OWNER,Blocked awaiting #1036 (update: now unblocked),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1046#issuecomment-716071507,https://api.github.com/repos/simonw/datasette/issues/1046,716071507,MDEyOklzc3VlQ29tbWVudDcxNjA3MTUwNw==,9599,simonw,2020-10-25T00:06:47Z,2020-10-25T00:06:47Z,OWNER,"I used https://primer.style/octicons/download-16 instead.

<img width=""734"" alt=""favicons__favicons__71_rows"" src=""https://user-images.githubusercontent.com/9599/97096021-44417280-161b-11eb-9bc4-88def6afaca4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895193,Link to blob downloads in the right places,
https://github.com/simonw/datasette/issues/1046#issuecomment-716066342,https://api.github.com/repos/simonw/datasette/issues/1046,716066342,MDEyOklzc3VlQ29tbWVudDcxNjA2NjM0Mg==,9599,simonw,2020-10-24T23:02:07Z,2020-10-24T23:02:25Z,OWNER,"A download icon would be nice for the links in the table display. I like this one https://primer.style/octicons/download-24

```svg
<svg xmlns=""http://www.w3.org/2000/svg"" viewBox=""0 0 24 24"" width=""24"" height=""24"">
  <path d=""M4.97 11.03a.75.75 0 111.06-1.06L11 14.94V2.75a.75.75 0 011.5 0v12.19l4.97-4.97a.75.75 0 111.06 1.06l-6.25 6.25a.75.75 0 01-1.06 0l-6.25-6.25zm-.22 9.47a.75.75 0 000 1.5h14.5a.75.75 0 000-1.5H4.75z""></path>
</svg>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895193,Link to blob downloads in the right places,
https://github.com/simonw/datasette/issues/1033#issuecomment-716048564,https://api.github.com/repos/simonw/datasette/issues/1033,716048564,MDEyOklzc3VlQ29tbWVudDcxNjA0ODU2NA==,9599,simonw,2020-10-24T20:08:31Z,2020-10-24T20:08:31Z,OWNER,Documentation here: https://docs.datasette.io/en/latest/internals.html#datasette-urls,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/575#issuecomment-716048199,https://api.github.com/repos/simonw/datasette/issues/575,716048199,MDEyOklzc3VlQ29tbWVudDcxNjA0ODE5OQ==,9599,simonw,2020-10-24T20:05:44Z,2020-10-24T20:05:44Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#static-assets,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497162288,Plugin documentation should cover how to bundle static/templates in setup.py,
https://github.com/simonw/datasette/issues/1042#issuecomment-715643763,https://api.github.com/repos/simonw/datasette/issues/1042,715643763,MDEyOklzc3VlQ29tbWVudDcxNTY0Mzc2Mw==,9599,simonw,2020-10-24T00:34:31Z,2020-10-24T00:34:52Z,OWNER,I'm going to rename that to template variable from `select_templates` to `templates_considered` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715643646,https://api.github.com/repos/simonw/datasette/issues/1042,715643646,MDEyOklzc3VlQ29tbWVudDcxNTY0MzY0Ng==,9599,simonw,2020-10-24T00:33:46Z,2020-10-24T00:33:46Z,OWNER,"I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class.

This code is the problem:
https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133

I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1045#issuecomment-715641183,https://api.github.com/repos/simonw/datasette/issues/1045,715641183,MDEyOklzc3VlQ29tbWVudDcxNTY0MTE4Mw==,9599,simonw,2020-10-24T00:19:29Z,2020-10-24T00:19:29Z,OWNER,"It turns out it already does that:

https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L710-L720

But the documentation doesn't reflect that:

> `template` - string
> 
> > The template file to be rendered, e.g. `my_plugin.html`. Datasette will search for this file first in the `--template-dir=` location, if it was specified - then in the plugin's bundled templates and finally in Datasette's set of default templates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728600048,"Document that datasette.render_template(template, ...) also accepts a list of templates",
https://github.com/simonw/datasette/issues/1042#issuecomment-715618333,https://api.github.com/repos/simonw/datasette/issues/1042,715618333,MDEyOklzc3VlQ29tbWVudDcxNTYxODMzMw==,9599,simonw,2020-10-23T22:33:24Z,2020-10-23T22:33:24Z,OWNER,"It wouldn't be a disaster if template-loading plugins were unable to hook into the `/{slug1}/{slug2}.html` custom page mechanism, since plugins can define their own pages already using `register_routes()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715618077,https://api.github.com/repos/simonw/datasette/issues/1042,715618077,MDEyOklzc3VlQ29tbWVudDcxNTYxODA3Nw==,9599,simonw,2020-10-23T22:32:24Z,2020-10-23T22:32:24Z,OWNER,"Another option: the first version of the plugin hook could accept only the template filename. Subsequent releases could add more arguments, since Pluggy allows new arguments to be added without breaking backwards compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715617830,https://api.github.com/repos/simonw/datasette/issues/1042,715617830,MDEyOklzc3VlQ29tbWVudDcxNTYxNzgzMA==,9599,simonw,2020-10-23T22:31:26Z,2020-10-23T22:31:26Z,OWNER,"So maybe this should be a `register_template_loader` mechanism that returns a Jinja loader after all? That would mean that only the template filename could be used as the input to the plugin, which doesn't seem as useful as emulating the `extra_template_vars()` interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715617405,https://api.github.com/repos/simonw/datasette/issues/1042,715617405,MDEyOklzc3VlQ29tbWVudDcxNTYxNzQwNQ==,9599,simonw,2020-10-23T22:29:53Z,2020-10-23T22:29:53Z,OWNER,"Also consider that `DatasetteRouter` uses `.list_templates()` to gather together `{slug}.html` style templates for the custom page templates mechanism: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L949-L967

For that to work with the new plugin hook, custom template providing plugins will need a way to provide a list of templates that they know about.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715616757,https://api.github.com/repos/simonw/datasette/issues/1042,715616757,MDEyOklzc3VlQ29tbWVudDcxNTYxNjc1Nw==,9599,simonw,2020-10-23T22:27:28Z,2020-10-23T22:27:28Z,OWNER,"Almost all of the core template loading happens in the `BaseView.render` method: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/views/base.py#L114-L133

The one exception is the 404 handling code here: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L1034-L1042","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715614971,https://api.github.com/repos/simonw/datasette/issues/1042,715614971,MDEyOklzc3VlQ29tbWVudDcxNTYxNDk3MQ==,9599,simonw,2020-10-23T22:20:14Z,2020-10-23T22:23:51Z,OWNER,"Alternative plugin hook idea:
```python
@hookspec
def load_template(template, database, table, columns, view_name, request, datasette):
    ""Load the specified template, returning the template code as a string""
```
Imitating the existing `extra_template_vars` family of hooks: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/pull/1040#issuecomment-715587715,https://api.github.com/repos/simonw/datasette/issues/1040,715587715,MDEyOklzc3VlQ29tbWVudDcxNTU4NzcxNQ==,9599,simonw,2020-10-23T21:01:07Z,2020-10-23T21:03:10Z,OWNER,"A download icon would be nice for the links in the table display.

I like this one https://primer.style/octicons/download-24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726910999,/db/table/-/blob/pk/column.blob download URL,
https://github.com/simonw/datasette/pull/1043#issuecomment-715586711,https://api.github.com/repos/simonw/datasette/issues/1043,715586711,MDEyOklzc3VlQ29tbWVudDcxNTU4NjcxMQ==,9599,simonw,2020-10-23T20:58:26Z,2020-10-23T20:58:26Z,OWNER,I misunderstood - `asgi-csrf` already has an sdist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist,
https://github.com/simonw/datasette/pull/1043#issuecomment-715585140,https://api.github.com/repos/simonw/datasette/issues/1043,715585140,MDEyOklzc3VlQ29tbWVudDcxNTU4NTE0MA==,9599,simonw,2020-10-23T20:54:29Z,2020-10-23T20:54:29Z,OWNER,Thanks. I'll push a source release of `asgi-csrf`.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist,
https://github.com/simonw/datasette/pull/1044#issuecomment-715584579,https://api.github.com/repos/simonw/datasette/issues/1044,715584579,MDEyOklzc3VlQ29tbWVudDcxNTU4NDU3OQ==,9599,simonw,2020-10-23T20:53:01Z,2020-10-23T20:53:01Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727916744,Add minimum supported python,
https://github.com/simonw/datasette/issues/745#issuecomment-715556545,https://api.github.com/repos/simonw/datasette/issues/745,715556545,MDEyOklzc3VlQ29tbWVudDcxNTU1NjU0NQ==,9599,simonw,2020-10-23T19:47:10Z,2020-10-23T19:47:10Z,OWNER,Dupe of #647 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608613033,Extract the hash-URL mechanism out into a plugin,
https://github.com/simonw/datasette/issues/1042#issuecomment-715497419,https://api.github.com/repos/simonw/datasette/issues/1042,715497419,MDEyOklzc3VlQ29tbWVudDcxNTQ5NzQxOQ==,9599,simonw,2020-10-23T18:12:40Z,2020-10-23T18:12:40Z,OWNER,Maybe the template loader can optionally return some extra context to pass to the template. That could be used to solve the templates considered comment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715496859,https://api.github.com/repos/simonw/datasette/issues/1042,715496859,MDEyOklzc3VlQ29tbWVudDcxNTQ5Njg1OQ==,9599,simonw,2020-10-23T18:11:27Z,2020-10-23T18:11:27Z,OWNER,"When loading a template the filename is required, but you can optionally also send a set of extra arguments which the template loader can take into consideration.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-715490532,https://api.github.com/repos/simonw/datasette/issues/1042,715490532,MDEyOklzc3VlQ29tbWVudDcxNTQ5MDUzMg==,9599,simonw,2020-10-23T17:57:34Z,2020-10-23T17:57:34Z,OWNER,"A better version of this hook would be passed the database, table and query name depending on what was being rendered.

This would require some re-thinking of how core templates are loaded, especially since I would want the templates considered comment to continue working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868867,https://api.github.com/repos/simonw/datasette/issues/1042,714868867,MDEyOklzc3VlQ29tbWVudDcxNDg2ODg2Nw==,9599,simonw,2020-10-23T02:31:17Z,2020-10-23T02:31:17Z,OWNER,I'll build this in conjunction with a plugin that supports editing templates stored in SQLite.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868624,https://api.github.com/repos/simonw/datasette/issues/1042,714868624,MDEyOklzc3VlQ29tbWVudDcxNDg2ODYyNA==,9599,simonw,2020-10-23T02:30:27Z,2020-10-23T02:30:37Z,OWNER,Maybe `register_template_loader(datasette)` which returns an object which is added in at the beginning of the list passed to `ChoiceLoader` here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/datasette/issues/1042#issuecomment-714868207,https://api.github.com/repos/simonw/datasette/issues/1042,714868207,MDEyOklzc3VlQ29tbWVudDcxNDg2ODIwNw==,9599,simonw,2020-10-23T02:29:12Z,2020-10-23T02:29:12Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d0cc6f4c32e1f89238ddec782086b3122f445bd4/datasette/app.py#L288-L311,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-714758139,https://api.github.com/repos/simonw/sqlite-utils/issues/173,714758139,MDEyOklzc3VlQ29tbWVudDcxNDc1ODEzOQ==,9599,simonw,2020-10-22T20:57:56Z,2020-10-22T20:57:56Z,OWNER,"I could use `ijson` to provide a progress bar for JSON arrays too. I'd prefer to keep that as an optional dependency though, since `sqlite-utils` is a library dependency for many other projects and it would be using `ijson` purely for the CLI component.

Here's how to iterate through a list of objects being read from a file:

```python
import json
parser = ijson.items(open(
    ""/tmp/list.json""
), ""item"")
for object in parser:
   # ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/datasette/issues/1041#issuecomment-714683801,https://api.github.com/repos/simonw/datasette/issues/1041,714683801,MDEyOklzc3VlQ29tbWVudDcxNDY4MzgwMQ==,9599,simonw,2020-10-22T18:37:47Z,2020-10-22T18:37:47Z,OWNER,"I think I'll do this by looking for URLs that start with `/` - since it's also possible to have full `https://...` URLs in that setting.
```json
{
    ""extra_css_urls"": [
        ""/static/styles.css""
    ],
    ""extra_js_urls"": [
        ""/static/app.js""
    ]
}
```
I need to think about the `extra_css_urls` and `extra_js_urls` plugin hooks too: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-css-urls-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1041#issuecomment-714682825,https://api.github.com/repos/simonw/datasette/issues/1041,714682825,MDEyOklzc3VlQ29tbWVudDcxNDY4MjgyNQ==,9599,simonw,2020-10-22T18:36:10Z,2020-10-22T18:36:10Z,OWNER,I'll need to update these docs once there's a solution for this in place: https://docs.datasette.io/en/latest/custom_templates.html#serving-static-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1041#issuecomment-714682288,https://api.github.com/repos/simonw/datasette/issues/1041,714682288,MDEyOklzc3VlQ29tbWVudDcxNDY4MjI4OA==,9599,simonw,2020-10-22T18:35:15Z,2020-10-22T18:35:15Z,OWNER,"@psychemedia said: https://github.com/simonw/datasette/issues/1033#issuecomment-714657366

> How does `/-/static` relate to [current guidance docs around `static`](https://docs.datasette.io/en/latest/custom_templates.html?highlight=static#serving-static-files) regarding the `--static option` and metadata formulations such as `""extra_js_urls"": [ ""/static/app.js""]` (I've not managed to get this to work in a Jupyter server proxied set up; the [datasette / jupyter server proxy repo](https://github.com/simonw/jupyterserverproxy-datasette-demo) may provide a useful test example, eg via MyBinder, for folk to crib from?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting,
https://github.com/simonw/datasette/issues/1033#issuecomment-714681365,https://api.github.com/repos/simonw/datasette/issues/1033,714681365,MDEyOklzc3VlQ29tbWVudDcxNDY4MTM2NQ==,9599,simonw,2020-10-22T18:33:48Z,2020-10-22T18:33:48Z,OWNER,"That's a good question - I hadn't considered that. I'm going to open a new issue to have `extra_js_urls` respect the `base_url` setting, since the static files will be served from a different location.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/sqlite-utils/issues/171#issuecomment-714208848,https://api.github.com/repos/simonw/sqlite-utils/issues/171,714208848,MDEyOklzc3VlQ29tbWVudDcxNDIwODg0OA==,9599,simonw,2020-10-22T04:07:14Z,2020-10-22T04:07:14Z,OWNER,"I made the `--load-extension` command much more widely supported in #137 - which should be useful for anyone who wants to use this extension.

It's a bit too obscure for me to want to add direct Python library support relating to that extension though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707407567,Idea: transitive closure tables for tree structures,
https://github.com/simonw/datasette/pull/1031#issuecomment-714206875,https://api.github.com/repos/simonw/datasette/issues/1031,714206875,MDEyOklzc3VlQ29tbWVudDcxNDIwNjg3NQ==,9599,simonw,2020-10-22T04:01:19Z,2020-10-22T04:01:19Z,OWNER,I don't fully understand the bug you're fixing here. Could you provide a bit more explanation?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed,
https://github.com/simonw/datasette/pull/1040#issuecomment-714206533,https://api.github.com/repos/simonw/datasette/issues/1040,714206533,MDEyOklzc3VlQ29tbWVudDcxNDIwNjUzMw==,9599,simonw,2020-10-22T04:00:25Z,2020-10-22T04:00:25Z,OWNER,I've decided not to offer a configuration option to turn this off. I'll reconsider if someone asks for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726910999,/db/table/-/blob/pk/column.blob download URL,
https://github.com/simonw/datasette/issues/998#issuecomment-714205783,https://api.github.com/repos/simonw/datasette/issues/998,714205783,MDEyOklzc3VlQ29tbWVudDcxNDIwNTc4Mw==,9599,simonw,2020-10-22T03:58:13Z,2020-10-22T03:58:13Z,OWNER,This is now live here: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/998#issuecomment-714117534,https://api.github.com/repos/simonw/datasette/issues/998,714117534,MDEyOklzc3VlQ29tbWVudDcxNDExNzUzNA==,9599,simonw,2020-10-22T01:12:06Z,2020-10-22T01:12:06Z,OWNER,"Demo:
![table-scroll](https://user-images.githubusercontent.com/9599/96806421-e74e7e00-13c8-11eb-95fe-44d01e4c2eb3.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/998#issuecomment-714092002,https://api.github.com/repos/simonw/datasette/issues/998,714092002,MDEyOklzc3VlQ29tbWVudDcxNDA5MjAwMg==,9599,simonw,2020-10-22T00:55:10Z,2020-10-22T00:55:10Z,OWNER,This isn't blocked on #987 - it just means that `datasette-cluster-map` will need to learn to look for `.table-wrapper` first and fall back on the table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/998#issuecomment-714090965,https://api.github.com/repos/simonw/datasette/issues/998,714090965,MDEyOklzc3VlQ29tbWVudDcxNDA5MDk2NQ==,9599,simonw,2020-10-22T00:54:30Z,2020-10-22T00:54:30Z,OWNER,"Easiest fix for the column action menu positioning - hide them when the user scrolls the containing div:
```javascript
document.querySelector('.table-wrapper').addEventListener(
  'scroll', () => document.querySelector('.dropdown-menu').style.display = 'none'
);                                            
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/pull/1038#issuecomment-713920461,https://api.github.com/repos/simonw/datasette/issues/1038,713920461,MDEyOklzc3VlQ29tbWVudDcxMzkyMDQ2MQ==,9599,simonw,2020-10-21T22:43:51Z,2020-10-21T22:43:51Z,OWNER,Thanks for spotting this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726154220,DOC: Fix syntax error,
https://github.com/simonw/datasette/issues/1036#issuecomment-713899530,https://api.github.com/repos/simonw/datasette/issues/1036,713899530,MDEyOklzc3VlQ29tbWVudDcxMzg5OTUzMA==,9599,simonw,2020-10-21T21:55:00Z,2020-10-21T21:55:00Z,OWNER,"This code needs these permission checks:
https://github.com/simonw/datasette/blob/bf82b3d6a605c9ddadd5fb739249dfe6defaf635/datasette/views/table.py#L911-L913","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713821656,https://api.github.com/repos/simonw/datasette/issues/1036,713821656,MDEyOklzc3VlQ29tbWVudDcxMzgyMTY1Ng==,9599,simonw,2020-10-21T19:22:45Z,2020-10-21T19:41:48Z,OWNER,"So for https://latest.datasette.io/fixtures/binary_data the BLOB download URLs would be:

`https://latest.datasette.io/fixtures/-/blob/binary_data/1/data.blob` - that last bit after the primary key is to indicate the `data` column

With these headers:

- `Content-Disposition: attachment; filename=""binary_data-1-data.blob""`
- `X-Content-Type-Options: nosniff`
- `Content-Type: application/binary`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713830842,https://api.github.com/repos/simonw/datasette/issues/1036,713830842,MDEyOklzc3VlQ29tbWVudDcxMzgzMDg0Mg==,9599,simonw,2020-10-21T19:41:20Z,2020-10-21T19:41:20Z,OWNER,Another useful demo database: https://datasette-render-images-demo.datasette.io/favicons/favicons - see https://datasette-render-images-demo.datasette.io/favicons/favicons.csv,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713829629,https://api.github.com/repos/simonw/datasette/issues/1036,713829629,MDEyOklzc3VlQ29tbWVudDcxMzgyOTYyOQ==,9599,simonw,2020-10-21T19:38:43Z,2020-10-21T19:38:43Z,OWNER,"Should this work just for BLOB columns, or should it work for other columns too?

For the moment I'm going to restrict it to BLOBs, since data from other columns is available through the UI whereas BLOB columns are not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713818817,https://api.github.com/repos/simonw/datasette/issues/1036,713818817,MDEyOklzc3VlQ29tbWVudDcxMzgxODgxNw==,9599,simonw,2020-10-21T19:17:01Z,2020-10-21T19:17:01Z,OWNER,Actually I like `.blob`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713818178,https://api.github.com/repos/simonw/datasette/issues/1036,713818178,MDEyOklzc3VlQ29tbWVudDcxMzgxODE3OA==,9599,simonw,2020-10-21T19:15:38Z,2020-10-21T19:16:34Z,OWNER,"What should the suggested filename be?

I think something that includes the table name, primary key and the name of the column would work.

How about a file extension? I guess `.binary`, then let the user rename it? Or `.raw`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1039#issuecomment-713754844,https://api.github.com/repos/simonw/datasette/issues/1039,713754844,MDEyOklzc3VlQ29tbWVudDcxMzc1NDg0NA==,9599,simonw,2020-10-21T17:58:27Z,2020-10-21T17:58:27Z,OWNER,"Now live: https://latest.datasette.io/fixtures/roadside_attraction_characteristics

![anim](https://user-images.githubusercontent.com/9599/96759016-55288480-138c-11eb-8ba0-d8e0f6dd8b1f.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726687572,Add an animation to the column actions menu,
https://github.com/simonw/datasette/issues/1036#issuecomment-713278349,https://api.github.com/repos/simonw/datasette/issues/1036,713278349,MDEyOklzc3VlQ29tbWVudDcxMzI3ODM0OQ==,9599,simonw,2020-10-21T03:42:29Z,2020-10-21T03:42:29Z,OWNER,Possible URL for this: `/db/table/-/blob/primary-keys` - this would use the `/db/table/-/` namespace proposed in #296.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/998#issuecomment-713269155,https://api.github.com/repos/simonw/datasette/issues/998,713269155,MDEyOklzc3VlQ29tbWVudDcxMzI2OTE1NQ==,9599,simonw,2020-10-21T03:17:07Z,2020-10-21T03:17:07Z,OWNER,"This may require updates to the column action menu JavaScript too, since it was not built with scrolling sideways in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/1037#issuecomment-713268905,https://api.github.com/repos/simonw/datasette/issues/1037,713268905,MDEyOklzc3VlQ29tbWVudDcxMzI2ODkwNQ==,9599,simonw,2020-10-21T03:16:36Z,2020-10-21T03:16:36Z,OWNER,Dupe of #998.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1037#issuecomment-713268498,https://api.github.com/repos/simonw/datasette/issues/1037,713268498,MDEyOklzc3VlQ29tbWVudDcxMzI2ODQ5OA==,9599,simonw,2020-10-21T03:15:44Z,2020-10-21T03:15:44Z,OWNER,"This may require updates to the column action menu JavaScript too, since it was not built with scrolling sideways in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1037#issuecomment-713267989,https://api.github.com/repos/simonw/datasette/issues/1037,713267989,MDEyOklzc3VlQ29tbWVudDcxMzI2Nzk4OQ==,9599,simonw,2020-10-21T03:14:34Z,2020-10-21T03:14:34Z,OWNER,"This is particularly relevant to the `datasette-cluster-map` plugin - the map is much nicer to use if the table itself can be scrolled.

That plugin also makes this harder to build, because the plugin inserts the map as the direct predecessor of the `<table>` element and hence breaks if you try to wrap that in a `<div>`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables,
https://github.com/simonw/datasette/issues/1036#issuecomment-713226726,https://api.github.com/repos/simonw/datasette/issues/1036,713226726,MDEyOklzc3VlQ29tbWVudDcxMzIyNjcyNg==,9599,simonw,2020-10-21T01:04:25Z,2020-10-21T01:04:25Z,OWNER,"Extra security idea: a `blob_download_host` setting which can be used to indicate a host that should be used for downloads - for example `datasettestatic.com`. If this setting is populated then binary downloads are served from paths on that host only, and no other Datasette URLs from that host will be served.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/262#issuecomment-713208667,https://api.github.com/repos/simonw/datasette/issues/262,713208667,MDEyOklzc3VlQ29tbWVudDcxMzIwODY2Nw==,9599,simonw,2020-10-21T00:03:18Z,2020-10-21T00:03:18Z,OWNER,"I think I should prioritize the facets component of this, since that could have significant performance wins while also supporting `datasette-graphql`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/262#issuecomment-713200782,https://api.github.com/repos/simonw/datasette/issues/262,713200782,MDEyOklzc3VlQ29tbWVudDcxMzIwMDc4Mg==,9599,simonw,2020-10-20T23:41:30Z,2020-10-20T23:41:30Z,OWNER,This is now blocking https://github.com/simonw/datasette-graphql/issues/61 because that issue needs a way to turn off suggested facets when retrieving the results of a table query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/1034#issuecomment-713191819,https://api.github.com/repos/simonw/datasette/issues/1034,713191819,MDEyOklzc3VlQ29tbWVudDcxMzE5MTgxOQ==,9599,simonw,2020-10-20T23:12:58Z,2020-10-20T23:12:58Z,OWNER,"Enzo has a great solution here: https://twitter.com/enzo_mdd/status/1318685442976436226

> Or maybe an option for a url. This keeps the CSV small but allows scripts to download binary data as needed.

In #1036 I'm planning on adding a way for users to access BLOB data. I can include that URL in the CSV output.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1036#issuecomment-713186189,https://api.github.com/repos/simonw/datasette/issues/1036,713186189,MDEyOklzc3VlQ29tbWVudDcxMzE4NjE4OQ==,9599,simonw,2020-10-20T22:56:33Z,2020-10-20T22:56:33Z,OWNER,I think this plus the binary-CSV stuff in #1034 will justify a dedicated section of the documentation to talk about how Datasette handles binary BLOB columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713185871,https://api.github.com/repos/simonw/datasette/issues/1036,713185871,MDEyOklzc3VlQ29tbWVudDcxMzE4NTg3MQ==,9599,simonw,2020-10-20T22:55:36Z,2020-10-20T22:55:36Z,OWNER,I can also use a `Content-Disposition` header to force a download. I'm reasonably confident that the combination of `Content-Disposition` and `X-Content-Type-Options: nosniff` and `application/binary` will let me allow users to download the contents of arbitrary BLOB columns without any XSS risk.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713185173,https://api.github.com/repos/simonw/datasette/issues/1036,713185173,MDEyOklzc3VlQ29tbWVudDcxMzE4NTE3Mw==,9599,simonw,2020-10-20T22:53:41Z,2020-10-20T22:53:41Z,OWNER,"https://security.stackexchange.com/questions/12896/does-x-content-type-options-really-prevent-content-sniffing-attacks says:

> In Tangled Web Michal Zalewski says:
> 
> > Refrain from using Content-Type: application/octet-stream and use application/binary instead, especially for unknown document types. Refrain from returning Content-Type: text/plain.
> > 
> > For example, any code-hosting platform must exercise caution when returning executables or source archives as application/octet-stream, because there is a risk they may be misinterpreted as HTML and displayed inline.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713184374,https://api.github.com/repos/simonw/datasette/issues/1036,713184374,MDEyOklzc3VlQ29tbWVudDcxMzE4NDM3NA==,9599,simonw,2020-10-20T22:51:22Z,2020-10-20T22:51:22Z,OWNER,"From https://hackerone.com/reports/126197:

> archive.uber.com mirrors pypi. When downloading `.tar.gz` files from archive.uber.com, the MIME type is `application/octet-stream`. Injecting `<html><script>alert(0)</script>` into the start of the `.tar.gz` causes an XSS in Internet Explorer due to MIME sniffing.

So you do have to be careful not to open accidental XSS holes with `application/octet-stream` thanks to (presumably older) versions of IE.

From that thread it looks like the solution is to add a `X-Content-Type-Options: nosniff` header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-713183306,https://api.github.com/repos/simonw/datasette/issues/1036,713183306,MDEyOklzc3VlQ29tbWVudDcxMzE4MzMwNg==,9599,simonw,2020-10-20T22:48:10Z,2020-10-20T22:48:10Z,OWNER,Twitter thread: https://twitter.com/dancow/status/1318681053347840005,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1034#issuecomment-713176082,https://api.github.com/repos/simonw/datasette/issues/1034,713176082,MDEyOklzc3VlQ29tbWVudDcxMzE3NjA4Mg==,9599,simonw,2020-10-20T22:27:33Z,2020-10-20T22:27:33Z,OWNER,"This feels good to me - it's consistent with how other features in Datasette work, and it means users who need the binary data in CSV (for whatever reason) can get it if they want to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713175741,https://api.github.com/repos/simonw/datasette/issues/1034,713175741,MDEyOklzc3VlQ29tbWVudDcxMzE3NTc0MQ==,9599,simonw,2020-10-20T22:26:45Z,2020-10-20T22:26:45Z,OWNER,"> New idea: since binary in CSV doesn't make sense anyway, emulate Datasette's HTML UI default and output this:
> 
>     id,title,data
>     1,Some title,<Binary data: 14 bytes>
>     2,Other title,<Binary data: 57 bytes>
> 
> Then allow users to add ?_base64=1 to the URL to get base64 instead
> https://twitter.com/simonw/status/1318679950635888641","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713174690,https://api.github.com/repos/simonw/datasette/issues/1034,713174690,MDEyOklzc3VlQ29tbWVudDcxMzE3NDY5MA==,9599,simonw,2020-10-20T22:23:50Z,2020-10-20T22:23:50Z,OWNER,Or... default to `<Binary data: 7 bytes>` and support a `?_base64=1` option which outputs in base64 instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713174341,https://api.github.com/repos/simonw/datasette/issues/1034,713174341,MDEyOklzc3VlQ29tbWVudDcxMzE3NDM0MQ==,9599,simonw,2020-10-20T22:22:53Z,2020-10-20T22:23:14Z,OWNER,"An even easier option: do what the Datasette UI does and output `<Binary data: 7 bytes>` for that CSV cell, as seen on https://latest.datasette.io/fixtures/binary_data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-713172901,https://api.github.com/repos/simonw/datasette/issues/1034,713172901,MDEyOklzc3VlQ29tbWVudDcxMzE3MjkwMQ==,9599,simonw,2020-10-20T22:19:10Z,2020-10-20T22:20:28Z,OWNER,"I could go with the same format as `datasette-render-binary` but using `0x00` as the format for the hex bytes.

    0x15 0x1C 0x02 0xC7 JFIF 0x00 0x01

Problem with this is that it's ambiguous: if the ASCII characters `0x15` occur in the text they will be indistinguishable from those hex bytes.

But since representing binary data in CSV fundamentally doesn't make sense I'm not sure if that really matters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/741#issuecomment-713171742,https://api.github.com/repos/simonw/datasette/issues/741,713171742,MDEyOklzc3VlQ29tbWVudDcxMzE3MTc0Mg==,9599,simonw,2020-10-20T22:16:25Z,2020-10-20T22:16:25Z,OWNER,See also #992 which will rename `--config` to `--setting`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/262#issuecomment-713170979,https://api.github.com/repos/simonw/datasette/issues/262,713170979,MDEyOklzc3VlQ29tbWVudDcxMzE3MDk3OQ==,9599,simonw,2020-10-20T22:14:37Z,2020-10-20T22:14:37Z,OWNER,"I think it's worth having a plugin hook for this - it can be same hook that is used internally. Maybe `register_extra` - it lets you return one or more `extra` implementations, each with a name and an async function that gets called.

Things like suggested facets will become `register_extra` hooks. Maybe actual facets too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/262#issuecomment-713170284,https://api.github.com/repos/simonw/datasette/issues/262,713170284,MDEyOklzc3VlQ29tbWVudDcxMzE3MDI4NA==,9599,simonw,2020-10-20T22:13:01Z,2020-10-20T22:13:01Z,OWNER,In the documentation for `?_extra=` I think I'll emphasize the comma-separated version of it. Also: there will be `?_extra=` values which act as aliases for collection combinations - e.g. `?_extra=full` will toggle everything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/782#issuecomment-712986115,https://api.github.com/repos/simonw/datasette/issues/782,712986115,MDEyOklzc3VlQ29tbWVudDcxMjk4NjExNQ==,9599,simonw,2020-10-20T16:28:46Z,2020-10-20T16:29:51Z,OWNER,"I think this all comes down to how the `?_extras=` mechanism works (see #262), as first hinted at in a30c5b220c15360d575e94b0e67f3255e120b916 (see commit message) when I added this long-forgotten undocumented feature: https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables

Extras need to be able to execute additional SQL, since that would solve the problem we have now where the expensive ""suggested facets"" code runs on all `.json` output even when its results are not being shown.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1035#issuecomment-712976314,https://api.github.com/repos/simonw/datasette/issues/1035,712976314,MDEyOklzc3VlQ29tbWVudDcxMjk3NjMxNA==,9599,simonw,2020-10-20T16:21:42Z,2020-10-20T16:21:42Z,OWNER,"Makes me question if `datasette.urls` should grow functionality equivalent to the other path and querystring manipulation methods in `datasette.utils`:

https://github.com/simonw/datasette/blob/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc/datasette/utils/__init__.py#L216-L279","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1035#issuecomment-712965574,https://api.github.com/repos/simonw/datasette/issues/1035,712965574,MDEyOklzc3VlQ29tbWVudDcxMjk2NTU3NA==,9599,simonw,2020-10-20T16:13:57Z,2020-10-20T16:13:57Z,OWNER,"That `renderers[key] = path_with_format(` is in a base class which can be used for both arbitrary queries, canned queries and the table view. I think that's OK, but it means that the `format=""json""` argument on `datasette.urls.table()` won't be used by Datasette internally, it will just be available for plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1035#issuecomment-712963959,https://api.github.com/repos/simonw/datasette/issues/1035,712963959,MDEyOklzc3VlQ29tbWVudDcxMjk2Mzk1OQ==,9599,simonw,2020-10-20T16:11:25Z,2020-10-20T16:11:25Z,OWNER,"Relevant code:

https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/utils/__init__.py#L681-L696

Only used here:

https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/views/base.py#L498-L502","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument",
https://github.com/simonw/datasette/issues/1026#issuecomment-712962517,https://api.github.com/repos/simonw/datasette/issues/1026,712962517,MDEyOklzc3VlQ29tbWVudDcxMjk2MjUxNw==,9599,simonw,2020-10-20T16:09:12Z,2020-10-20T16:09:12Z,OWNER,"That `datasette.urls.table(""db"", ""table"") + "".json""` example is bad because if the table name contains a `.` it should be `?_format=json` instead.

Maybe `.table()` should have a `format=""json""` option that knows how to do this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712959034,https://api.github.com/repos/simonw/datasette/issues/1026,712959034,MDEyOklzc3VlQ29tbWVudDcxMjk1OTAzNA==,9599,simonw,2020-10-20T16:03:33Z,2020-10-20T16:03:55Z,OWNER,"Reconsidering this: I think the `.get()` etc methods should automatically add the `base_url` prefix for you, since these APIs are only intended to make internal calls.

The clincher on this is when I went to add a section to the `datasette.client` documentation recommending you use `datasette.urls.path()` for every call to them that you make.

But there's a problem: to handle table name escaping users are likely to want to use `datasette.urls.table()` anyway, like this:

    response = await datasette.client.get(datasette.urls.table(""db"", ""table"") + "".json"")

This risks adding the `base_url` prefix twice. Maybe the `.table()` method could return a string-like object that is marked as already having the `base_url` prefix added, so the `client.get()` method knows not to add it again.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-712607608,https://api.github.com/repos/simonw/datasette/issues/1023,712607608,MDEyOklzc3VlQ29tbWVudDcxMjYwNzYwOA==,9599,simonw,2020-10-20T05:47:42Z,2020-10-20T05:47:42Z,OWNER,Requested alpha testers in https://github.com/simonw/datasette/issues/838#issuecomment-712604364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-712607227,https://api.github.com/repos/simonw/datasette/issues/1026,712607227,MDEyOklzc3VlQ29tbWVudDcxMjYwNzIyNw==,9599,simonw,2020-10-20T05:46:44Z,2020-10-20T05:46:44Z,OWNER,We have a solution for this now: `datasette.urls` from #1033 can be used by plugins to assemble the correct URLs to pass to `.get()` and friends.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/1023#issuecomment-712604541,https://api.github.com/repos/simonw/datasette/issues/1023,712604541,MDEyOklzc3VlQ29tbWVudDcxMjYwNDU0MQ==,9599,simonw,2020-10-20T05:39:44Z,2020-10-20T05:39:44Z,OWNER,Here's the alpha with most of this work ready for people to preview: https://github.com/simonw/datasette/releases/tag/0.51a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url,
https://github.com/simonw/datasette/issues/838#issuecomment-712604364,https://api.github.com/repos/simonw/datasette/issues/838,712604364,MDEyOklzc3VlQ29tbWVudDcxMjYwNDM2NA==,9599,simonw,2020-10-20T05:39:15Z,2020-10-20T05:39:15Z,OWNER,"OK, I've made a ton of improvements to how the `base_url` setting works - see tickets linked from #1023. I've just pushed out an alpha release with those changes in it: https://github.com/simonw/datasette/releases/tag/0.51a0

@tsibley @tballison @ChristopherWilks I'd really appreciate your help testing this alpha! 

You can install it with:

    pip install datasette==0.51a0

It should work with just `ProxyPass`, without needing the `ProxyPassReverse` setting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/865#issuecomment-712597762,https://api.github.com/repos/simonw/datasette/issues/865,712597762,MDEyOklzc3VlQ29tbWVudDcxMjU5Nzc2Mg==,9599,simonw,2020-10-20T05:22:59Z,2020-10-20T05:22:59Z,OWNER,"OK, this is definitely working now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/1025#issuecomment-712593790,https://api.github.com/repos/simonw/datasette/issues/1025,712593790,MDEyOklzc3VlQ29tbWVudDcxMjU5Mzc5MA==,9599,simonw,2020-10-20T05:12:36Z,2020-10-20T05:12:36Z,OWNER,"I'm going to leave the cookies code setting cookies to default to the `""/""` top level.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/782#issuecomment-712590398,https://api.github.com/repos/simonw/datasette/issues/782,712590398,MDEyOklzc3VlQ29tbWVudDcxMjU5MDM5OA==,9599,simonw,2020-10-20T05:03:46Z,2020-10-20T05:04:09Z,OWNER,"OK, https://latest-with-plugins.datasette.io/ is running that now - e.g. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions.json-preview or https://latest-with-plugins.datasette.io/fixtures/compound_three_primary_keys.json-preview

```json
{
  ""rows"": [
    {
      ""pk"": 1,
      ""name"": ""The Mystery Spot"",
      ""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
      ""latitude"": 37.0167,
      ""longitude"": -122.0024
    },
    {
      ""pk"": 2,
      ""name"": ""Winchester Mystery House"",
      ""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
      ""latitude"": 37.3184,
      ""longitude"": -121.9511
    },
    {
      ""pk"": 3,
      ""name"": ""Burlingame Museum of PEZ Memorabilia"",
      ""address"": ""214 California Drive, Burlingame, CA 94010"",
      ""latitude"": 37.5793,
      ""longitude"": -122.3442
    },
    {
      ""pk"": 4,
      ""name"": ""Bigfoot Discovery Museum"",
      ""address"": ""5497 Highway 9, Felton, CA 95018"",
      ""latitude"": 37.0414,
      ""longitude"": -122.0725
    }
  ],
  ""total"": 4,
  ""next_url"": null
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-712585921,https://api.github.com/repos/simonw/datasette/issues/782,712585921,MDEyOklzc3VlQ29tbWVudDcxMjU4NTkyMQ==,9599,simonw,2020-10-20T04:48:01Z,2020-10-20T04:48:01Z,OWNER,I'll update `datasette-json-preview` with that now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-712585687,https://api.github.com/repos/simonw/datasette/issues/782,712585687,MDEyOklzc3VlQ29tbWVudDcxMjU4NTY4Nw==,9599,simonw,2020-10-20T04:47:02Z,2020-10-20T04:47:12Z,OWNER,"Great point about CORS, I hadn't considered that.

I think I'm going to keep the `Link:` header (added in #1014) because I quite enjoy using it with GitHub and WordPress, but I'm not going to have it be the default way of doing pagination. For the default shape I'm now leaning towards this:

```json
{
  ""total"": 36,
  ""rows"": [{""id"": 1, ""name"": ""Cleo""}],
  ""next_url"": ""https://latest-with-plugins.datasette.io/fixtures/facetable.json?_next=5""
}
```

So three keys: `total`, `rows` and `next_url`. Then extra keys can be added using `?_extra=` with various named bundles.","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1034#issuecomment-712582699,https://api.github.com/repos/simonw/datasette/issues/1034,712582699,MDEyOklzc3VlQ29tbWVudDcxMjU4MjY5OQ==,9599,simonw,2020-10-20T04:36:04Z,2020-10-20T04:36:14Z,OWNER,Asked for ideas on Twitter: https://twitter.com/simonw/status/1318409558805467136,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-712581994,https://api.github.com/repos/simonw/datasette/issues/1034,712581994,MDEyOklzc3VlQ29tbWVudDcxMjU4MTk5NA==,9599,simonw,2020-10-20T04:33:28Z,2020-10-20T04:33:28Z,OWNER,"The [datasette-render-binary](https://github.com/simonw/datasette-render-binary) plugin does this, which I really like - but without the different coloured fonts I'm not sure how readable it would be as just plain text:

![image](https://user-images.githubusercontent.com/9599/96540435-9c125f00-1252-11eb-85aa-5fc8d0e63728.png)

Really the goal here is to find the most human-friendly option, so that people looking at the output have a vague idea what's going on. That's why I'm not leaping at the chance to use base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1034#issuecomment-712580976,https://api.github.com/repos/simonw/datasette/issues/1034,712580976,MDEyOklzc3VlQ29tbWVudDcxMjU4MDk3Ng==,9599,simonw,2020-10-20T04:29:23Z,2020-10-20T04:29:23Z,OWNER,Most obvious option is base64. Any other potential solutions I'm missing?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output,
https://github.com/simonw/datasette/issues/1025#issuecomment-712579674,https://api.github.com/repos/simonw/datasette/issues/1025,712579674,MDEyOklzc3VlQ29tbWVudDcxMjU3OTY3NA==,9599,simonw,2020-10-20T04:24:10Z,2020-10-20T04:24:10Z,OWNER,"Changed my mind, `error.html` needs access to `urls` in order to link to its CSS file. Passing it after all (it already got passed `ds.config(""base_url"")` so `ds` was available previously).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-712481127,https://api.github.com/repos/simonw/datasette/issues/1025,712481127,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTEyNw==,9599,simonw,2020-10-19T22:40:37Z,2020-10-20T01:21:36Z,OWNER,Was blocked on #904 - now unblocked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1033#issuecomment-712529413,https://api.github.com/repos/simonw/datasette/issues/1033,712529413,MDEyOklzc3VlQ29tbWVudDcxMjUyOTQxMw==,9599,simonw,2020-10-20T01:21:12Z,2020-10-20T01:21:12Z,OWNER,Also refs #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method,
https://github.com/simonw/datasette/issues/1025#issuecomment-712525557,https://api.github.com/repos/simonw/datasette/issues/1025,712525557,MDEyOklzc3VlQ29tbWVudDcxMjUyNTU1Nw==,9599,simonw,2020-10-20T01:07:02Z,2020-10-20T01:07:02Z,OWNER,"I fixed the `queries.html` one. I'm not going to fix these two:
```
datasette/templates/error.html:        <a href=""/"">home</a>
datasette/templates/patterns.html:        <a href=""/"">home</a> /
```
Because the `error.html` one does not get passed a context (which makes sense since an error has occurred) and the pattern portfolio doesn't need to link to anywhere in particular.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/904#issuecomment-712524699,https://api.github.com/repos/simonw/datasette/issues/904,712524699,MDEyOklzc3VlQ29tbWVudDcxMjUyNDY5OQ==,9599,simonw,2020-10-20T01:04:12Z,2020-10-20T01:04:12Z,OWNER,Documentation is https://docs.datasette.io/en/latest/writing_plugins.html#building-urls-within-plugins and https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712483066,https://api.github.com/repos/simonw/datasette/issues/904,712483066,MDEyOklzc3VlQ29tbWVudDcxMjQ4MzA2Ng==,9599,simonw,2020-10-19T22:46:48Z,2020-10-19T22:46:48Z,OWNER,"OK, I'm committing to `datasette.urls.table()` and friends, which will be available to (and documented for use in) plugins and will be exposed to templates as `{{ urls.table(...) }}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/1020#issuecomment-712482504,https://api.github.com/repos/simonw/datasette/issues/1020,712482504,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjUwNA==,9599,simonw,2020-10-19T22:45:01Z,2020-10-19T22:45:01Z,OWNER,"I'm having trouble coming up with the syntax for this. Here's one option:
```python
response = await datasette.client.get(path, request=request)
```
But this feels confusing to me. We're not using the `request=` argument as a request - we're using it as a source of some default request values (the cookies and incoming headers, but not the path).

We're essentially combining that request with the other arguments passed to `.get()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712482015,https://api.github.com/repos/simonw/datasette/issues/1020,712482015,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjAxNQ==,9599,simonw,2020-10-19T22:43:24Z,2020-10-19T22:43:24Z,OWNER,"... unless I want to support authentication mechanisms that work based on incoming IP address instead, in which case there's an argument for copying more over from the incoming request.

Tailscale is a good example of a system where authentication based on IP address can actually work well, so this is worth doing. Also, there might be authentication mechanisms which work by setting a custom header on the incoming request (not to mention the `Authorization` header).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-712481568,https://api.github.com/repos/simonw/datasette/issues/1020,712481568,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTU2OA==,9599,simonw,2020-10-19T22:41:59Z,2020-10-19T22:41:59Z,OWNER,"It turns out this works just fine:
```python
response = await datasette.client.get(path, cookies=request.cookies)
```
So I don't need a mechanism for this. I'm going to add this to the documentation instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1028#issuecomment-712480866,https://api.github.com/repos/simonw/datasette/issues/1028,712480866,MDEyOklzc3VlQ29tbWVudDcxMjQ4MDg2Ng==,9599,simonw,2020-10-19T22:39:51Z,2020-10-19T22:39:51Z,OWNER,Documentation: https://docs.datasette.io/en/latest/spatialite.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723803777,--load-extension=spatialite shortcut,
https://github.com/simonw/datasette/issues/1032#issuecomment-712367285,https://api.github.com/repos/simonw/datasette/issues/1032,712367285,MDEyOklzc3VlQ29tbWVudDcxMjM2NzI4NQ==,9599,simonw,2020-10-19T18:39:32Z,2020-10-19T18:39:32Z,OWNER,https://github.com/digital-land/brownfield-land-collection/blob/a09ddf9960a6af59e72dc02448f7b645e59bf227/bin/harmonise.py#L217-L247 is a beautiful example of this problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712365439,https://api.github.com/repos/simonw/datasette/issues/1032,712365439,MDEyOklzc3VlQ29tbWVudDcxMjM2NTQzOQ==,9599,simonw,2020-10-19T18:35:50Z,2020-10-19T18:37:57Z,OWNER,"Maybe I don't need to add `dateutil` as a dependency here?

`pendulum` and `arrow` both depend on it so it's pretty deeply embedded in the Python date ecosystem. Adding it as a dependency seems reasonable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712365236,https://api.github.com/repos/simonw/datasette/issues/1032,712365236,MDEyOklzc3VlQ29tbWVudDcxMjM2NTIzNg==,9599,simonw,2020-10-19T18:35:25Z,2020-10-19T18:35:25Z,OWNER,"Since I'm dealing with tables of data I usually have a whole column of examples, so heuristics that check for numbers-greater-than-12 could actually work well for many cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712364532,https://api.github.com/repos/simonw/datasette/issues/1032,712364532,MDEyOklzc3VlQ29tbWVudDcxMjM2NDUzMg==,9599,simonw,2020-10-19T18:34:10Z,2020-10-19T18:34:10Z,OWNER,Lots of great example date data in https://biglocal.datasettes.com/COVID_WARN_Notices,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712364317,https://api.github.com/repos/simonw/datasette/issues/1032,712364317,MDEyOklzc3VlQ29tbWVudDcxMjM2NDMxNw==,9599,simonw,2020-10-19T18:33:42Z,2020-10-19T18:33:42Z,OWNER,"Related challenge: timezones. I think I'll punt on those for the moment and just concentrate on dates, but I should keep them in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712363825,https://api.github.com/repos/simonw/datasette/issues/1032,712363825,MDEyOklzc3VlQ29tbWVudDcxMjM2MzgyNQ==,9599,simonw,2020-10-19T18:32:43Z,2020-10-19T18:32:43Z,OWNER,"Horrible thought: I bet there is data out there that uses more than one date format in the same table! So this needs to be exposed as a visible per-column setting.

Column action menu can help here, although it's not yet the full solution because it isn't yet visible on mobile.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/1032#issuecomment-712363344,https://api.github.com/repos/simonw/datasette/issues/1032,712363344,MDEyOklzc3VlQ29tbWVudDcxMjM2MzM0NA==,9599,simonw,2020-10-19T18:31:46Z,2020-10-19T18:31:46Z,OWNER,"As always with dates, the challenge is America. `mm/dd/yy` is indistinguishable from the more sensible `dd/mm/yy`. It's crucially important to visibly tell people which format you assumed, otherwise all kinds of weird invisible bugs can manifest.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core,
https://github.com/simonw/datasette/issues/904#issuecomment-712355877,https://api.github.com/repos/simonw/datasette/issues/904,712355877,MDEyOklzc3VlQ29tbWVudDcxMjM1NTg3Nw==,9599,simonw,2020-10-19T18:17:22Z,2020-10-19T18:17:22Z,OWNER,I quite like `datasette.urls.table()` but I'm not sure why.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712355706,https://api.github.com/repos/simonw/datasette/issues/904,712355706,MDEyOklzc3VlQ29tbWVudDcxMjM1NTcwNg==,9599,simonw,2020-10-19T18:17:03Z,2020-10-19T18:17:03Z,OWNER,"Options:
- `datasette.urls.instance()` (or `datasette.urls.root()`), `datasette.urls.table()` etc
- `datasette.url_for.instance()`...
- `datasette.url.instance()`...
- `datasette.root_url()`, `datasette.table_url()`...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712354600,https://api.github.com/repos/simonw/datasette/issues/904,712354600,MDEyOklzc3VlQ29tbWVudDcxMjM1NDYwMA==,9599,simonw,2020-10-19T18:15:03Z,2020-10-19T18:15:39Z,OWNER,"Related: #1026 (How should datasette.client interact with base_url)

Also this comment from https://github.com/simonw/datasette/issues/943#issuecomment-675752436

> One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted.
> 
> If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-712324077,https://api.github.com/repos/simonw/datasette/issues/904,712324077,MDEyOklzc3VlQ29tbWVudDcxMjMyNDA3Nw==,9599,simonw,2020-10-19T17:39:38Z,2020-10-19T17:39:38Z,OWNER,"If I do these methods I think this should be available on the `datasette` object too, as an internal API for plugins to use to construct redirects and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-710487083,https://api.github.com/repos/simonw/datasette/issues/904,710487083,MDEyOklzc3VlQ29tbWVudDcxMDQ4NzA4Mw==,9599,simonw,2020-10-16T19:34:10Z,2020-10-19T17:39:04Z,OWNER,"Alternatively, I could expose a single object that knows how to construct all kinds of URLs. Something like this:
```
{{ urls.instance() }}
{{ urls.database(database_name) }}
{{ urls.table(database_name, table_name) }}
etc
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/1027#issuecomment-712320103,https://api.github.com/repos/simonw/datasette/issues/1027,712320103,MDEyOklzc3VlQ29tbWVudDcxMjMyMDEwMw==,9599,simonw,2020-10-19T17:35:04Z,2020-10-19T17:35:04Z,OWNER,Still need to configure proxying though. https://www.netnea.com/cms/apache-tutorial-9_setting-up-a-reverse-proxy/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/991#issuecomment-712317638,https://api.github.com/repos/simonw/datasette/issues/991,712317638,MDEyOklzc3VlQ29tbWVudDcxMjMxNzYzOA==,9599,simonw,2020-10-19T17:30:56Z,2020-10-19T17:30:56Z,OWNER,https://biglocal.datasettes.com/ is one of my larger Datasettes in terms of number of databases.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/pull/1029#issuecomment-711072903,https://api.github.com/repos/simonw/datasette/issues/1029,711072903,MDEyOklzc3VlQ29tbWVudDcxMTA3MjkwMw==,9599,simonw,2020-10-17T20:04:31Z,2020-10-17T20:04:31Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723837704,fix(docs): broken link,
https://github.com/simonw/datasette/issues/1027#issuecomment-710750038,https://api.github.com/repos/simonw/datasette/issues/1027,710750038,MDEyOklzc3VlQ29tbWVudDcxMDc1MDAzOA==,9599,simonw,2020-10-17T05:00:19Z,2020-10-17T05:00:19Z,OWNER,"Running `apachectl -X` appears to run Apache in the foreground so I can hit Ctrl+C to quit it again, using the config file at `/usr/local/etc/httpd/httpd.conf` and serving the `index.html` file from `/usr/local/var/www` (which confusingly says ""Welcome to nginx"", presumably because I installed nginx via Homebrew earlier).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/991#issuecomment-710694711,https://api.github.com/repos/simonw/datasette/issues/991,710694711,MDEyOklzc3VlQ29tbWVudDcxMDY5NDcxMQ==,9599,simonw,2020-10-16T23:24:03Z,2020-10-16T23:24:03Z,OWNER,I'm really interested in exploring how this page could work with hundreds of database files attached and thousands of total tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/991#issuecomment-710694607,https://api.github.com/repos/simonw/datasette/issues/991,710694607,MDEyOklzc3VlQ29tbWVudDcxMDY5NDYwNw==,9599,simonw,2020-10-16T23:23:34Z,2020-10-16T23:23:34Z,OWNER,Thinking more about pagination and search from #461: if the job of the homepage is to showcase the data that is available in the instance - data that is mostly in tables - maybe it's the tables themselves (and the ability to paginate and search through them) that becomes key to the design of the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/991#issuecomment-710694144,https://api.github.com/repos/simonw/datasette/issues/991,710694144,MDEyOklzc3VlQ29tbWVudDcxMDY5NDE0NA==,9599,simonw,2020-10-16T23:21:41Z,2020-10-16T23:21:41Z,OWNER,Relevant: https://github.com/simonw/datasette/discussions/1021 asks about controlling the order of databases on that page that have been loaded using configuration directory mode.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/461#issuecomment-710693927,https://api.github.com/repos/simonw/datasette/issues/461,710693927,MDEyOklzc3VlQ29tbWVudDcxMDY5MzkyNw==,9599,simonw,2020-10-16T23:20:43Z,2020-10-16T23:20:43Z,OWNER,Related: redesign homepage entirely #991.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",443021509,Paginate + search for databases/tables on the homepage,
https://github.com/simonw/datasette/issues/468#issuecomment-710693818,https://api.github.com/repos/simonw/datasette/issues/468,710693818,MDEyOklzc3VlQ29tbWVudDcxMDY5MzgxOA==,9599,simonw,2020-10-16T23:20:22Z,2020-10-16T23:20:22Z,OWNER,This is a duplicate of #461.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",444746021,Pagination for the database index page,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710461468,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710461468,MDEyOklzc3VlQ29tbWVudDcxMDQ2MTQ2OA==,9599,simonw,2020-10-16T19:18:19Z,2020-10-16T19:18:19Z,OWNER,"Reconsidering: #89 was a feature request that relates to this, so maybe this is worth implementing after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242,https://api.github.com/repos/simonw/sqlite-utils/issues/89,710460242,MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==,9599,simonw,2020-10-16T19:17:27Z,2020-10-16T19:17:50Z,OWNER,"I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works:
```python
fresh_db.table(""tree"", extracts=[Extract(
    columns=(""CommonName"", ""LatinName""),
    table=""Species"",
    fk_column=""species_id"",
    rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature,
https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710456981,https://api.github.com/repos/simonw/sqlite-utils/issues/187,710456981,MDEyOklzc3VlQ29tbWVudDcxMDQ1Njk4MQ==,9599,simonw,2020-10-16T19:15:13Z,2020-10-16T19:15:13Z,OWNER,This is a duplicate of #79.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723460107,Maybe: Utility method / CLI tool for initializing SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710440853,https://api.github.com/repos/simonw/sqlite-utils/issues/187,710440853,MDEyOklzc3VlQ29tbWVudDcxMDQ0MDg1Mw==,9599,simonw,2020-10-16T19:04:19Z,2020-10-16T19:04:19Z,OWNER,I split this off from #136.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723460107,Maybe: Utility method / CLI tool for initializing SpatiaLite,
https://github.com/simonw/sqlite-utils/issues/136#issuecomment-710428802,https://api.github.com/repos/simonw/sqlite-utils/issues/136,710428802,MDEyOklzc3VlQ29tbWVudDcxMDQyODgwMg==,9599,simonw,2020-10-16T18:56:47Z,2020-10-16T18:56:47Z,OWNER,"To keep the code cleaner, I'm tempted to support this instead:

    --load-extension=spatialite

Where `spatialite` is a special shortcut value that triggers a search for that module in known locations.

Users could still load a module in a file called `spatialite` in the current directory using:

    --load-extension=./spatialite

In fact, `--load-extension=spatialite` could handle that case too by always checking for a file called `spatialite` before attempting to search for it in known locations.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642,--load-extension=spatialite shortcut option,
https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710405658,MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==,9599,simonw,2020-10-16T18:42:48Z,2020-10-16T18:42:48Z,OWNER,"Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d

I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142,Feature request: enable extensions loading,
https://github.com/simonw/sqlite-utils/issues/48#issuecomment-710402331,https://api.github.com/repos/simonw/sqlite-utils/issues/48,710402331,MDEyOklzc3VlQ29tbWVudDcxMDQwMjMzMQ==,9599,simonw,2020-10-16T18:41:06Z,2020-10-16T18:41:06Z,OWNER,I could use this demo from JupyterCon 2020 https://gist.github.com/simonw/656c21b5800d5e4624dec9930f00e093,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder",
https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593,https://api.github.com/repos/simonw/sqlite-utils/issues/58,710399593,MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==,9599,simonw,2020-10-16T18:39:31Z,2020-10-16T18:39:31Z,OWNER,I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488293926,Support enabling FTS on views,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710397574,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710397574,MDEyOklzc3VlQ29tbWVudDcxMDM5NzU3NA==,9599,simonw,2020-10-16T18:38:21Z,2020-10-16T18:38:21Z,OWNER,"I'm not going to implement this. I'll leave `extract=...` as it is right now, suitable for quick simple single-column operations on input, but if users want to do something more complicated involving multiple columns they should use the `table.extract()` method after the initial insert instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710395444,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710395444,MDEyOklzc3VlQ29tbWVudDcxMDM5NTQ0NA==,9599,simonw,2020-10-16T18:37:10Z,2020-10-16T18:37:10Z,OWNER,"But this begins to feel too complicated, given that `table.extract()` can already be used to achieve the same thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710393550,MDEyOklzc3VlQ29tbWVudDcxMDM5MzU1MA==,9599,simonw,2020-10-16T18:35:57Z,2020-10-16T18:36:39Z,OWNER,"If I want to support that most complicated example, I think the option to pass a `Extracts()` object to `extracts=` is the best way to do it:
```python
fresh_db.table(""tree"", extracts=[Extract(
    columns=(""CommonName"", ""LatinName""),
    table=""Species"",
    fk_column=""species_id"",
    rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710390915,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710390915,MDEyOklzc3VlQ29tbWVudDcxMDM5MDkxNQ==,9599,simonw,2020-10-16T18:34:26Z,2020-10-16T18:34:50Z,OWNER,"Here's the most complex example of `.extracts()`:
```python
db[""Trees""].extract(
    [""CommonName"", ""LatinName""],
    table=""Species"",
    fk_column=""species_id"",
    rename={""CommonName"": ""name"", ""LatinName"": ""latin""}
)
```
Resulting in:
```sql
CREATE TABLE [Species] (
    [id] INTEGER PRIMARY KEY,
    [name] TEXT,
    [latin] TEXT
)
```
From https://sqlite-utils.readthedocs.io/en/stable/python-api.html#extracting-columns-into-a-separate-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710364942,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710364942,MDEyOklzc3VlQ29tbWVudDcxMDM2NDk0Mg==,9599,simonw,2020-10-16T18:18:48Z,2020-10-16T18:18:48Z,OWNER,"I think there is. It's a nice existing feature, and I don't think adding tuple support to it would be a huge lift.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710363789,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710363789,MDEyOklzc3VlQ29tbWVudDcxMDM2Mzc4OQ==,9599,simonw,2020-10-16T18:18:05Z,2020-10-16T18:18:05Z,OWNER,I wonder if there's value in extending the `extracts=` option at all given the existence of `table.extract()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710359724,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710359724,MDEyOklzc3VlQ29tbWVudDcxMDM1OTcyNA==,9599,simonw,2020-10-16T18:15:31Z,2020-10-16T18:15:31Z,OWNER,"Using a tuple would work:
```python
fresh_db.table(""tree"", extracts=[(""common_name"", ""latin_name"")])
```
Or to define a custom name:
```python
fresh_db.table(""tree"", extracts={(""common_name"", ""latin_name""): ""names""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710346830,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710346830,MDEyOklzc3VlQ29tbWVudDcxMDM0NjgzMA==,9599,simonw,2020-10-16T18:08:52Z,2020-10-16T18:09:21Z,OWNER,"The new `.extract()` method can handle multiple columns:

https://github.com/simonw/sqlite-utils/blob/2c541fac352632e23e40b0d21e3f233f7a744a57/tests/test_extract.py#L70-L87","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710258736,https://api.github.com/repos/simonw/sqlite-utils/issues/182,710258736,MDEyOklzc3VlQ29tbWVudDcxMDI1ODczNg==,9599,simonw,2020-10-16T17:20:41Z,2020-10-16T17:20:41Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-710198162,https://api.github.com/repos/simonw/sqlite-utils/issues/186,710198162,MDEyOklzc3VlQ29tbWVudDcxMDE5ODE2Mg==,9599,simonw,2020-10-16T16:41:00Z,2020-10-16T16:41:00Z,OWNER,"Failing test:
```python
def test_extract_null_values(fresh_db):
    fresh_db[""species""].insert({""id"": 1, ""species"": ""Wolf""}, pk=""id"")
    fresh_db[""individuals""].insert_all(
        [
            {""id"": 10, ""name"": ""Terriana"", ""species"": ""Fox""},
            {""id"": 11, ""name"": ""Spenidorm"", ""species"": None},
            {""id"": 12, ""name"": ""Grantheim"", ""species"": ""Wolf""},
            {""id"": 13, ""name"": ""Turnutopia"", ""species"": None},
            {""id"": 14, ""name"": ""Wargal"", ""species"": ""Wolf""},
        ],
        pk=""id"",
    )
    fresh_db[""individuals""].extract(""species"")
    assert fresh_db[""species""].schema == (
        ""CREATE TABLE [species] (\n""
        ""   [id] INTEGER PRIMARY KEY,\n""
        ""   [species] TEXT\n""
        "")""
    )
    assert fresh_db[""individuals""].schema == (
        'CREATE TABLE ""individuals"" (\n'
        ""   [id] INTEGER PRIMARY KEY,\n""
        ""   [name] TEXT,\n""
        ""   [species_id] INTEGER,\n""
        ""   FOREIGN KEY(species_id) REFERENCES species(id)\n""
        "")""
    )
    assert list(fresh_db[""species""].rows) == [
        {""id"": 1, ""species"": ""Wolf""},
        {""id"": 2, ""species"": ""Fox""},
    ]
    assert list(fresh_db[""individuals""].rows) == [
        {""id"": 10, ""name"": ""Terriana"", ""species_id"": 2},
        {""id"": 11, ""name"": ""Spenidorm"", ""species_id"": None},
        {""id"": 12, ""name"": ""Grantheim"", ""species_id"": 1},
        {""id"": 13, ""name"": ""Turnutopia"", ""species_id"": None},
        {""id"": 14, ""name"": ""Wargal"", ""species_id"": 1},
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710178871,https://api.github.com/repos/simonw/sqlite-utils/issues/182,710178871,MDEyOklzc3VlQ29tbWVudDcxMDE3ODg3MQ==,9599,simonw,2020-10-16T16:27:39Z,2020-10-16T16:28:14Z,OWNER,"The file is opened for me by `click.File()`, which also handles things like `-` for stdin. But i neee to be able to switch the encoding used to read from that based on the `--encoding` option.

I think the way to do that is to open the file in binary mode and then wrap it in a codec reader:
```python
fp = codecs.getreader(encoding)(fp)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706260,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709706260,MDEyOklzc3VlQ29tbWVudDcwOTcwNjI2MA==,9599,simonw,2020-10-16T03:17:02Z,2020-10-16T03:17:17Z,OWNER,Actually I think this should be an option to `.extract()` which controls if nulls are extracted or left alone. Maybe called `extract_nulls=True/False`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706065,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709706065,MDEyOklzc3VlQ29tbWVudDcwOTcwNjA2NQ==,9599,simonw,2020-10-16T03:16:22Z,2020-10-16T03:16:22Z,OWNER,Either way I think I'm going to need to add some SQL which uses `where a = b or (a is null and b is null)`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705885,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709705885,MDEyOklzc3VlQ29tbWVudDcwOTcwNTg4NQ==,9599,simonw,2020-10-16T03:15:39Z,2020-10-16T03:15:39Z,OWNER,The alternative solution here would be that a single `null` value DOES get extracted. To implement this I would need to add some logic that uses `is null` instead of `=`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705624,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709705624,MDEyOklzc3VlQ29tbWVudDcwOTcwNTYyNA==,9599,simonw,2020-10-16T03:14:39Z,2020-10-16T03:14:39Z,OWNER,"How should this work with extractions covering multiple columns?

If there's a single column then it makes sense that a `null` value would not be extracted into the lookup table, but would instead become stay as `null`.

For a multiple column extraction, provided at least one of those columns is not null It should map to a record in the lookup table. Only if ALL of the extracted columns are null should the lookup value stay null.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/datasette/issues/1027#issuecomment-709647525,https://api.github.com/repos/simonw/datasette/issues/1027,709647525,MDEyOklzc3VlQ29tbWVudDcwOTY0NzUyNQ==,9599,simonw,2020-10-15T23:49:51Z,2020-10-15T23:51:39Z,OWNER,"I'll install Apache on macOS to figure this out using https://formulae.brew.sh/formula/httpd

`brew install httpd` output this at the end:

```
==> httpd
DocumentRoot is /usr/local/var/www.

The default ports have been set in /usr/local/etc/httpd/httpd.conf to 8080 and in
/usr/local/etc/httpd/extra/httpd-ssl.conf to 8443 so that httpd can run without sudo.

To have launchd start httpd now and restart at login:
  brew services start httpd
Or, if you don't want/need a background service you can just run:
  apachectl start
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1027#issuecomment-709646865,https://api.github.com/repos/simonw/datasette/issues/1027,709646865,MDEyOklzc3VlQ29tbWVudDcwOTY0Njg2NQ==,9599,simonw,2020-10-15T23:47:08Z,2020-10-15T23:47:08Z,OWNER,It should cover both nginx and Apache. nginx config is here: https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url,
https://github.com/simonw/datasette/issues/1026#issuecomment-709636372,https://api.github.com/repos/simonw/datasette/issues/1026,709636372,MDEyOklzc3VlQ29tbWVudDcwOTYzNjM3Mg==,9599,simonw,2020-10-15T23:09:34Z,2020-10-15T23:09:34Z,OWNER,"I'm inclined to say that internal requests should ignore `base_url` - since that seems like the right thing for plugins that need to access default Datasette APIs.

The one catch here is plugins that might want to proxy the current incoming URL for some reason - where that incoming `request.path` could include the `base_url`.

Actually those should be fine - because it will have been stripped off earlier:

https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/app.py#L963-L968","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url,
https://github.com/simonw/datasette/issues/904#issuecomment-709635276,https://api.github.com/repos/simonw/datasette/issues/904,709635276,MDEyOklzc3VlQ29tbWVudDcwOTYzNTI3Ng==,9599,simonw,2020-10-15T23:05:54Z,2020-10-15T23:05:54Z,OWNER,Could have `instance_url()` take an optional path argument which is then turned into the correct path.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709635021,https://api.github.com/repos/simonw/datasette/issues/904,709635021,MDEyOklzc3VlQ29tbWVudDcwOTYzNTAyMQ==,9599,simonw,2020-10-15T23:05:11Z,2020-10-15T23:05:11Z,OWNER,"I think this should be a family of functions:

- `instance_url()` - the root URL of the instance (usually `/` unless `base_url` is set)
- `database_url(database_name)` - already got this
- `table_url(database_name, table_name)`
- `row_url(database_name, table_name, row)` - not sure about this one. The idea would be for `row` to be correctly turned into a URL by introspecting the primary keys for that table, then pulling those values out of the SQLite `row` object. Might not be necessary though.

I also need a way for plugins to link to e.g. `/-/configure-fts` - or even `/-/configure-fts/database-name/table-name`. What should that look like?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709634261,https://api.github.com/repos/simonw/datasette/issues/904,709634261,MDEyOklzc3VlQ29tbWVudDcwOTYzNDI2MQ==,9599,simonw,2020-10-15T23:02:43Z,2020-10-15T23:02:43Z,OWNER,"Here's the current implementation of `database_url` - on the `BaseView` class, but only because it needs access to a `datasette` instance (to read `base_url`):
https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/datasette/views/base.py#L102-L108","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/904#issuecomment-709633823,https://api.github.com/repos/simonw/datasette/issues/904,709633823,MDEyOklzc3VlQ29tbWVudDcwOTYzMzgyMw==,9599,simonw,2020-10-15T23:01:13Z,2020-10-15T23:01:13Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates",
https://github.com/simonw/datasette/issues/988#issuecomment-709633762,https://api.github.com/repos/simonw/datasette/issues/988,709633762,MDEyOklzc3VlQ29tbWVudDcwOTYzMzc2Mg==,9599,simonw,2020-10-15T23:01:01Z,2020-10-15T23:01:01Z,OWNER,This is a dupe of https://github.com/simonw/datasette/issues/904,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",713209404,Mechanism for plugins to construct URLs that respect base_url,
https://github.com/simonw/datasette/issues/865#issuecomment-709633080,https://api.github.com/repos/simonw/datasette/issues/865,709633080,MDEyOklzc3VlQ29tbWVudDcwOTYzMzA4MA==,9599,simonw,2020-10-15T22:58:51Z,2020-10-15T22:58:51Z,OWNER,"It looks like there are places where Datasette might return a redirect that doesn't take `base_url` into account - I'm planning on fixing those here, after which I think `ProxyPassReverse` should no longer be necessary. https://github.com/simonw/datasette/issues/1025#issuecomment-709632136","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/900#issuecomment-709632765,https://api.github.com/repos/simonw/datasette/issues/900,709632765,MDEyOklzc3VlQ29tbWVudDcwOTYzMjc2NQ==,9599,simonw,2020-10-15T22:57:55Z,2020-10-15T22:57:55Z,OWNER,"I believe this particular bug has been fixed, based on my testing here: https://github.com/simonw/datasette/issues/1024#issuecomment-709622973

Please re-open the ticket if you are still experiencing it.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url,
https://github.com/simonw/datasette/issues/1025#issuecomment-709632314,https://api.github.com/repos/simonw/datasette/issues/1025,709632314,MDEyOklzc3VlQ29tbWVudDcwOTYzMjMxNA==,9599,simonw,2020-10-15T22:56:25Z,2020-10-15T22:56:34Z,OWNER,"That `utils/asgi.py` line is the default path for setting cookies. That should likely take `base_url` into account too:
https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/utils/asgi.py#L331-L342","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-709632136,https://api.github.com/repos/simonw/datasette/issues/1025,709632136,MDEyOklzc3VlQ29tbWVudDcwOTYzMjEzNg==,9599,simonw,2020-10-15T22:55:44Z,2020-10-15T22:55:44Z,OWNER,"It looks like there are also some generated redirect responses that don't take `base_url` into account:
```
datasette % git grep '""/' -- '*.py' ':(exclude)*test_*.py' ':(exclude)datasette/app.py'
datasette/_version.py:        for i in cfg.versionfile_source.split(""/""):
datasette/utils/asgi.py:        path=""/"",
datasette/views/base.py:            should_redirect = ""/{}-{}"".format(name, expected)
datasette/views/base.py:                should_redirect += ""/"" + urllib.parse.quote_plus(kwargs[""table""])
datasette/views/base.py:                should_redirect += ""/"" + kwargs[""pk_path""]
datasette/views/special.py:            response = Response.redirect(""/"")
datasette/views/special.py:            return Response.redirect(""/"")
datasette/views/special.py:        response = Response.redirect(""/"")
datasette/views/special.py:        return Response.redirect(""/"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/1025#issuecomment-709629920,https://api.github.com/repos/simonw/datasette/issues/1025,709629920,MDEyOklzc3VlQ29tbWVudDcwOTYyOTkyMA==,9599,simonw,2020-10-15T22:48:20Z,2020-10-15T22:48:20Z,OWNER,"Also these:
```
datasette % git grep '""/' -- '*.html' ':(exclude)*/patterns.html'
datasette/templates/allow_debug.html:<form action=""/-/allow-debug"" method=""get"">
datasette/templates/base.html:        <form action=""/-/logout"" method=""post"">
datasette/templates/error.html:        <a href=""/"">home</a>
datasette/templates/logout.html:<form action=""/-/logout"" method=""post"">
datasette/templates/messages_debug.html:<form action=""/-/messages"" method=""post"">
datasette/templates/query.html:        <a href=""/"">home</a> /
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url",
https://github.com/simonw/datasette/issues/865#issuecomment-709626786,https://api.github.com/repos/simonw/datasette/issues/865,709626786,MDEyOklzc3VlQ29tbWVudDcwOTYyNjc4Ng==,9599,simonw,2020-10-15T22:38:38Z,2020-10-15T22:38:38Z,OWNER,"I managed to recreate proxying using `nginx` in #1024 - but I could not replicate this bug. I did NOT use `ProxyPassReverse` though. I think that may be what caused the problem.

I'll add a section to the documentation about this shortly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/1024#issuecomment-709625063,https://api.github.com/repos/simonw/datasette/issues/1024,709625063,MDEyOklzc3VlQ29tbWVudDcwOTYyNTA2Mw==,9599,simonw,2020-10-15T22:33:22Z,2020-10-15T22:33:22Z,OWNER,"Of those errors...

`http://localhost:8000/robots.txt` 404 is fine.

`http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22` looks to me like a `wget` parsing bug where it got confused by this JavaScript:

```
window.DATASETTE_CLUSTER_MAP_TILE_LAYER_OPTIONS = {""maxZoom"": 19, ""detectRetina"": true, ""attribution"": ""&copy; <a href=\""https://www.openstreetmap.org/copyright\"">OpenStreetMap</a> contributors""};</script>
```

`http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js` is a real bug. It's a bug in `datasette-cluster-map` but also requires me to solve #988 - mechanism for plugins to construct URLs that obey `base_url`.

I'm not sure why I'm getting a hit to `http://localhost:8000/` since I wouldn't expect to link to `/` anywhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709622973,https://api.github.com/repos/simonw/datasette/issues/1024,709622973,MDEyOklzc3VlQ29tbWVudDcwOTYyMjk3Mw==,9599,simonw,2020-10-15T22:27:31Z,2020-10-15T22:27:31Z,OWNER,"Here's how I tested it:
```
time wget -r 'http://localhost:8000/datasette/' 2>&1 | grep -i -C 5 ""failed\|error"" > /tmp/errors.txt
```
This wrote out any errors (plus context) to the `errors.txt` log - and reported that the full crawl took 33s.

Here's what I got in `errors.txt`:
```

     0K .                                                      71.6M=0s

2020-10-15 15:23:09 (71.6 MB/s) - ‘localhost:8000/datasette/index.html’ saved [1276]

Loading robots.txt; please ignore errors.
--2020-10-15 15:23:09--  http://localhost:8000/robots.txt
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
--
--2020-10-15 15:23:09--  http://localhost:8000/robots.txt
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:09 ERROR 404: Not Found.

--2020-10-15 15:23:09--  http://localhost:8000/datasette/-/static/app.css?b576be
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: 8563 (8.4K) [text/css]
--
--
2020-10-15 15:23:13 (7.90 MB/s) - ‘localhost:8000/datasette/fixtures/primary_key_multiple_columns_explicit_label.json?_shape=object’ saved [58]

--2020-10-15 15:23:13--  http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:13 ERROR 404: Not Found.

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:13 (84.3 MB/s) - ‘localhost:8000/datasette/fixtures/roadside_attractions.json?_shape=object’ saved [619]

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures/%5C%22https://www.openstreetmap.org/copyright%5C%22
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:13 ERROR 404: Not Found.

--2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+text1%2C+text2%2C+%5Bname+with+.+and+spaces%5D+from+searchable+order+by+pk+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:14 (28.6 MB/s) - ‘localhost:8000/datasette/fixtures/searchable_view_configured_by_metadata.json?_shape=array&_nl=on’ saved [180]

--2020-10-15 15:23:14--  http://localhost:8000/
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:14 ERROR 404: Not Found.

--2020-10-15 15:23:14--  http://localhost:8000/datasette/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101&_hide_sql=1
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
--
--
2020-10-15 15:23:21 (64.1 MB/s) - ‘localhost:8000/datasette/fixtures.csv?sql=select+pk,+name,+address,+latitude,+longitude+from+roadside_attractions+order+by+pk+limit+101&_size=max’ saved [403]

--2020-10-15 15:23:21--  http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 404 Not Found
2020-10-15 15:23:21 ERROR 404: Not Found.

--2020-10-15 15:23:21--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+desc+limit+101
Reusing existing connection to localhost:8000.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709600335,https://api.github.com/repos/simonw/datasette/issues/1024,709600335,MDEyOklzc3VlQ29tbWVudDcwOTYwMDMzNQ==,9599,simonw,2020-10-15T21:28:02Z,2020-10-15T22:25:43Z,OWNER,"This is working OK so far:

<img width=""897"" alt=""fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk2"" src=""https://user-images.githubusercontent.com/9599/96187878-96371980-0ef2-11eb-875e-5b1ae9b445fd.png"">

I'll try crawling it with `wget -r` to see if I get any errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,https://api.github.com/repos/simonw/datasette/issues/1024,709598324,MDEyOklzc3VlQ29tbWVudDcwOTU5ODMyNA==,9599,simonw,2020-10-15T21:23:33Z,2020-10-15T21:26:55Z,OWNER,"Combining these two examples, here's the config file I am going to use for this. I'll save this as `nginx.conf`:
```
daemon off;

events {
  worker_connections  1024;
}

http {
  server {
    listen 8000;

    location /datasette {
        proxy_pass              http://127.0.0.1:8001;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}
```
Then start the server with:
```
nginx -p `pwd` -c `pwd`/nginx.conf
```
And start Datasette like this:
```
datasette fixtures.db --config base_url:/datasette/
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709597589,https://api.github.com/repos/simonw/datasette/issues/1024,709597589,MDEyOklzc3VlQ29tbWVudDcwOTU5NzU4OQ==,9599,simonw,2020-10-15T21:21:53Z,2020-10-15T21:23:25Z,OWNER,"Here's a recipe for running nginx against a custom config file: https://gist.github.com/simonw/35f0ebf9c1d6df158759

```
daemon off;

events {
  worker_connections  1024;
}

http {
  access_log /dev/stdout;
  error_log /dev/stderr;

  types {
    text/html                             html htm shtml;
    text/css                              css;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
  }

  server {
    listen 8002;
    index index.html;
    root  app;
  }
}
```
```
nginx -p `pwd` -c `pwd`/nginx.conf
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709595960,https://api.github.com/repos/simonw/datasette/issues/1024,709595960,MDEyOklzc3VlQ29tbWVudDcwOTU5NTk2MA==,9599,simonw,2020-10-15T21:18:14Z,2020-10-15T21:18:14Z,OWNER,Typing `nginx` starts it running as a daemon listening on port `http-alt` aka 8080. It uses the config file from ` /usr/local/etc/nginx/nginx.conf`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709590941,https://api.github.com/repos/simonw/datasette/issues/1024,709590941,MDEyOklzc3VlQ29tbWVudDcwOTU5MDk0MQ==,9599,simonw,2020-10-15T21:07:47Z,2020-10-15T21:07:47Z,OWNER,On macOS I ran `brew install nginx`. I'm going to try running it on port 8000 so I don't have to run it as root.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709590337,https://api.github.com/repos/simonw/datasette/issues/1024,709590337,MDEyOklzc3VlQ29tbWVudDcwOTU5MDMzNw==,9599,simonw,2020-10-15T21:06:24Z,2020-10-15T21:07:19Z,OWNER,"From https://stackoverflow.com/questions/32549684/nginx-proxy-and-remove-proxy-pass-prefix/32550251 it looks like the config I should use is:
```
server {
    listen 80;
    server_name example.com;

    location /datasette/ {
        proxy_pass              http://127.0.0.1:8001;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
        proxy_read_timeout      90;
    }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/1024#issuecomment-709589297,https://api.github.com/repos/simonw/datasette/issues/1024,709589297,MDEyOklzc3VlQ29tbWVudDcwOTU4OTI5Nw==,9599,simonw,2020-10-15T21:04:31Z,2020-10-15T21:04:31Z,OWNER,I think nginx or Apache would be the best tools for this. I'm inclined to try with nginx first since I know it better.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting,
https://github.com/simonw/datasette/issues/838#issuecomment-709588425,https://api.github.com/repos/simonw/datasette/issues/838,709588425,MDEyOklzc3VlQ29tbWVudDcwOTU4ODQyNQ==,9599,simonw,2020-10-15T21:02:37Z,2020-10-15T21:02:37Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/865#issuecomment-709588373,https://api.github.com/repos/simonw/datasette/issues/865,709588373,MDEyOklzc3VlQ29tbWVudDcwOTU4ODM3Mw==,9599,simonw,2020-10-15T21:02:31Z,2020-10-15T21:02:31Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/900#issuecomment-709588322,https://api.github.com/repos/simonw/datasette/issues/900,709588322,MDEyOklzc3VlQ29tbWVudDcwOTU4ODMyMg==,9599,simonw,2020-10-15T21:02:26Z,2020-10-15T21:02:26Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url,
https://github.com/simonw/datasette/issues/988#issuecomment-709588290,https://api.github.com/repos/simonw/datasette/issues/988,709588290,MDEyOklzc3VlQ29tbWVudDcwOTU4ODI5MA==,9599,simonw,2020-10-15T21:02:21Z,2020-10-15T21:02:21Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",713209404,Mechanism for plugins to construct URLs that respect base_url,
https://github.com/simonw/datasette/issues/894#issuecomment-709575818,https://api.github.com/repos/simonw/datasette/issues/894,709575818,MDEyOklzc3VlQ29tbWVudDcwOTU3NTgxOA==,9599,simonw,2020-10-15T20:35:03Z,2020-10-15T20:35:03Z,OWNER,"Prototype so far:
```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index ea11a51..d61f8bd 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -497,17 +497,32 @@ class TableView(RowTableShared):
         if sort and sort_desc:
             raise DatasetteError(""Cannot use _sort and _sort_desc at the same time"")
 
+        def parse_sort(sort):
+            if ""~"" in sort:
+                if sort.endswith(""~default""):
+                    col = sort.rsplit(""~"", 1)[0]
+                    return col, escape_sqlite(col)
+                elif sort.endswith(""~numeric""):
+                    col = sort.rsplit(""~"", 1)[0]
+                    return col, ""cast(nullif({}, '') as real)"".format(escape_sqlite(col))
+                else:
+                    return sort, escape_sqlite(sort)
+            else:
+                return sort, escape_sqlite(sort)
+
         if sort:
-            if sort not in sortable_columns:
-                raise DatasetteError(""Cannot sort table by {}"".format(sort))
+            sort_column, sort_clause = parse_sort(sort)
+            if sort_column not in sortable_columns:
+                raise DatasetteError(""Cannot sort table by {}"".format(sort_column))
 
-            order_by = escape_sqlite(sort)
+            order_by = sort_clause
 
         if sort_desc:
-            if sort_desc not in sortable_columns:
-                raise DatasetteError(""Cannot sort table by {}"".format(sort_desc))
+            sort_column, sort_clause = parse_sort(sort_desc)
+            if sort_column not in sortable_columns:
+                raise DatasetteError(""Cannot sort table by {}"".format(sort_column))
 
-            order_by = ""{} desc"".format(escape_sqlite(sort_desc))
+            order_by = ""{} desc"".format(sort_clause)
 
         from_sql = ""from {table_name} {where}"".format(
             table_name=escape_sqlite(table),
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709572425,https://api.github.com/repos/simonw/datasette/issues/894,709572425,MDEyOklzc3VlQ29tbWVudDcwOTU3MjQyNQ==,9599,simonw,2020-10-15T20:28:18Z,2020-10-15T20:28:18Z,OWNER,"Also need to rethink this template logic that decides if to show a column as sorted or not:
https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/templates/_table.html#L10-L14","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709571143,https://api.github.com/repos/simonw/datasette/issues/894,709571143,MDEyOklzc3VlQ29tbWVudDcwOTU3MTE0Mw==,9599,simonw,2020-10-15T20:25:35Z,2020-10-15T20:25:35Z,OWNER,"`cast(nullif(colname, '') as real)` can fix this - it will treat `''` the same as `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709569951,https://api.github.com/repos/simonw/datasette/issues/894,709569951,MDEyOklzc3VlQ29tbWVudDcwOTU2OTk1MQ==,9599,simonw,2020-10-15T20:23:02Z,2020-10-15T20:23:02Z,OWNER,"Something to watch out for: `""""` empty strings cast to `0.0`:

`select cast(""100"" as real), ""100"", cast(null as real), cast("""" as real)`

cast(""100"" as real) | ""100"" | cast(null as real) | cast("""" as real)
-- | -- | -- | --
100.0 | 100 |   | 0.0

https://latest.datasette.io/fixtures?sql=select+cast%28%22100%22+as+real%29%2C+%22100%22%2C+cast%28null+as+real%29%2C+cast%28%22%22+as+real%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709562940,https://api.github.com/repos/simonw/datasette/issues/894,709562940,MDEyOklzc3VlQ29tbWVudDcwOTU2Mjk0MA==,9599,simonw,2020-10-15T20:08:16Z,2020-10-15T20:08:16Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/views/table.py#L485-L510,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709546976,https://api.github.com/repos/simonw/datasette/issues/894,709546976,MDEyOklzc3VlQ29tbWVudDcwOTU0Njk3Ng==,9599,simonw,2020-10-15T19:35:55Z,2020-10-15T19:36:38Z,OWNER,"Much easier solution: if the suffix is `~numeric` then treat it as the column name sorted numerically. If the suffix is missing OR the suffix is `~default`, sort without casting. Only add the `~default` suffix if the column name itself contains at least one `~` symbol.

Using `~` because it doesn't need to be URL-encoded.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709539257,https://api.github.com/repos/simonw/datasette/issues/894,709539257,MDEyOklzc3VlQ29tbWVudDcwOTUzOTI1Nw==,9599,simonw,2020-10-15T19:19:29Z,2020-10-15T19:34:07Z,OWNER,"Urgh this isn't going to work. `%7E~%7E` gets decoded as `~~~` so I wouldn't be able to tell the difference.

I could use double-percentage-encoding here instead. I feel like there's a simpler solution that I'm missing (and that may well be in use within Datasette already, I'm not doing great thinking this morning).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709534197,https://api.github.com/repos/simonw/datasette/issues/894,709534197,MDEyOklzc3VlQ29tbWVudDcwOTUzNDE5Nw==,9599,simonw,2020-10-15T19:08:53Z,2020-10-15T19:17:55Z,OWNER,"Even better solution: use URL encoding in the parameter details. This is consistent with how `?_next=` tokens work, e.g. `?_next=0.291861560261786%2Ce%2Cj`.

So the format can be:

- `mycolumn`
- `urlencoded-mycolumn$castname`

For most columns this will look like: `?_sort=score$numeric`

For columns with a `$` in their name it will be `?_sort=score%24hasdollar$numeric`

Problem: both `$` and `,` are usually URL encoded anyway. I need a character which isn't encoded by default, so that I can use its encoded form to show it is part of the column name and its un-encoded form to split the cast indicator.

`_` is a candidate here - not encoded by default, but can be encoded as `%5F`.

The other unreserved non-alphanumeric characters are `-`, `.`, `_`, `~`.

Of these, `~` is least likely to show up in a column name. So I'll use that.

- `mycolumn`
- `mycolumn~numeric`
- `mycolumn%7Ewith%7Etildes~numeric`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709532369,https://api.github.com/repos/simonw/datasette/issues/894,709532369,MDEyOklzc3VlQ29tbWVudDcwOTUzMjM2OQ==,9599,simonw,2020-10-15T19:05:07Z,2020-10-15T19:07:35Z,OWNER,"Simpler option: `?_sort=` column values look like this:

- `mycolumn` - for sort by column
- `mycolumn$numeric` - for sort by column after cast to float
- `mycolumn$morename$default` - for the edge case where the column name itself contains a $ symbol","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709531343,https://api.github.com/repos/simonw/datasette/issues/894,709531343,MDEyOklzc3VlQ29tbWVudDcwOTUzMTM0Mw==,9599,simonw,2020-10-15T19:03:12Z,2020-10-15T19:03:12Z,OWNER,"The Sort by `<select>` menu needs a rethink: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/templates/table.html#L80-L87

If it's going to include sort by numeric options it needs a different format - since `?_sort=colname` needs to also support `?_sort=numeric_colname` - but shouldn't clash with a weird column already named `numeric_colname`.

Can I come up with a value syntax for this that is guaranteed not to clash with a weirdly named existing column?

I think so. I could use `{""type"": ""numeric"", ""column"": ""mycolumn""}` as the value - then if a column has a name that is itself valid JSON (weird but possible) the `_sort=` value would be `{""type"": ""default"", ""column"": ""{\""type\"": \""numeric\"", \""name\"": \""column\""}""}`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709525082,https://api.github.com/repos/simonw/datasette/issues/894,709525082,MDEyOklzc3VlQ29tbWVudDcwOTUyNTA4Mg==,9599,simonw,2020-10-15T18:51:59Z,2020-10-15T18:51:59Z,OWNER,This is enough of a design to build a working prototype.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709505147,https://api.github.com/repos/simonw/datasette/issues/894,709505147,MDEyOklzc3VlQ29tbWVudDcwOTUwNTE0Nw==,9599,simonw,2020-10-15T18:18:45Z,2020-10-15T18:50:50Z,OWNER,"This needs querystring parameter design. Some options:

* `?_sort_clause=cast(mycol+as+integer)` and `?_sort_clause_desc=cast(mycol+as+integer)` - allowing any expression. This would need to be disabled if arbitrary SQL was turned off, similar to the restrictions on `?_where=`.
* `?_sort_numeric=mycol` and `?_sort_numeric_desc=mycol` - this would cast to `real` which would work on integer values as well. It could be allowed even when arbitrary SQL was disabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709524123,https://api.github.com/repos/simonw/datasette/issues/894,709524123,MDEyOklzc3VlQ29tbWVudDcwOTUyNDEyMw==,9599,simonw,2020-10-15T18:50:25Z,2020-10-15T18:50:25Z,OWNER,"For the ""Sort by X"" select menu case... I could automatically expand that menu to contain extra options for ""Sort numerically by X"" for each TEXT column in the table. That's a pretty good option.

For the action cog menu, I can add the extra options to the cog menu - and rely on the fact that the title of the page will say ""Sorted numerically by colname descending"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709513483,https://api.github.com/repos/simonw/datasette/issues/894,709513483,MDEyOklzc3VlQ29tbWVudDcwOTUxMzQ4Mw==,9599,simonw,2020-10-15T18:31:56Z,2020-10-15T18:31:56Z,OWNER,"I think the first version of this feature involves implementing `?_sort_numeric=col` and `?_sort_numeric_desc=col` plus the JavaScript to detect if those values should be shown in the column actions menu.

One question: how to reflect that this is happening in the current sort UI. This menu here for example:

<img width=""440"" alt=""fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk3_descending"" src=""https://user-images.githubusercontent.com/9599/96171519-e35ac180-0ed9-11eb-9945-6780054e4ad5.png"">

And this interface: how should it indicate that a text is currently sorted numerically v.s. sorted alphabetically, and allow the user to switch from one to the other?

<img width=""794"" alt=""fixtures__sortable__201_rows_where_sorted_by_sortable_with_nulls_descending"" src=""https://user-images.githubusercontent.com/9599/96171618-04bbad80-0eda-11eb-946c-ed3b17a8c749.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709511399,https://api.github.com/repos/simonw/datasette/issues/894,709511399,MDEyOklzc3VlQ29tbWVudDcwOTUxMTM5OQ==,9599,simonw,2020-10-15T18:28:09Z,2020-10-15T18:28:09Z,OWNER,"The simplest solution would be for Python code to scan all of the visible values on the current page and show the column action for ""sort by this numeric"" based purely on that. I already do that in the JavaScript for ""are there any blank values in the first page?"" here:

https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/static/table.js#L106-L118

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709510422,https://api.github.com/repos/simonw/datasette/issues/894,709510422,MDEyOklzc3VlQ29tbWVudDcwOTUxMDQyMg==,9599,simonw,2020-10-15T18:26:25Z,2020-10-15T18:26:25Z,OWNER,"There's something interesting about figuring out which sort options should be offered in the column actions menu.

Two options:

- Try to detect if a text column is all-integers or all-floats, either by scanning the entire table (if it's small enough) or by scanning the first X rows, where X might be the size of the first page or maybe the first 1,000 or similar.
- Could also let users define this in `metadata.yml` for the table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709509635,https://api.github.com/repos/simonw/datasette/issues/894,709509635,MDEyOklzc3VlQ29tbWVudDcwOTUwOTYzNQ==,9599,simonw,2020-10-15T18:24:57Z,2020-10-15T18:24:57Z,OWNER,"This does feel like a weird plugin hook just because there aren't really THAT many different use-cases that plugins could solve. The ones I can think of are:

- Sort numeric
- Sort by parsed date
- Sort by FTS rank

Could this work if I just allow `_sort_clause=`?

One possible solution for the no-arbitrary-SQL case: users can define sort orders in `metadata.json/yml`. So if you want to enable sort-by-distance without enabling arbitrary SQL you could add something like this:

```yaml
databases:
  mydb:
    tables:
      museums:
        sort_clause: bm25(fts)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709505953,https://api.github.com/repos/simonw/datasette/issues/894,709505953,MDEyOklzc3VlQ29tbWVudDcwOTUwNTk1Mw==,9599,simonw,2020-10-15T18:20:13Z,2020-10-15T18:20:13Z,OWNER,Sorting by date when the column has a junk date format in it is such a column need it should maybe ship in Datasette by default - though I've been trying to avoid adding heavy dependencies like `dateutil` if I can get away with it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-709505608,https://api.github.com/repos/simonw/datasette/issues/894,709505608,MDEyOklzc3VlQ29tbWVudDcwOTUwNTYwOA==,9599,simonw,2020-10-15T18:19:35Z,2020-10-15T18:19:35Z,OWNER,"I could even let plugins define new sort types. Imagine a plugin that enables this:

`?_sort_date_desc=mycol` - where it knows how to handle specific date formats, or even uses `dateutil.parser.parse`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/878#issuecomment-709503359,https://api.github.com/repos/simonw/datasette/issues/878,709503359,MDEyOklzc3VlQ29tbWVudDcwOTUwMzM1OQ==,9599,simonw,2020-10-15T18:15:28Z,2020-10-15T18:15:28Z,OWNER,I think this is blocking #619,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-709502889,https://api.github.com/repos/simonw/datasette/issues/878,709502889,MDEyOklzc3VlQ29tbWVudDcwOTUwMjg4OQ==,9599,simonw,2020-10-15T18:14:34Z,2020-10-15T18:14:34Z,OWNER,"The `BaseView` class does this for Datasette internals at the moment, but I'm not convinced it works as well as it could.

I'd like to turn this into a class that is documented and available to plugins as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/690#issuecomment-709497595,https://api.github.com/repos/simonw/datasette/issues/690,709497595,MDEyOklzc3VlQ29tbWVudDcwOTQ5NzU5NQ==,9599,simonw,2020-10-15T18:04:35Z,2020-10-15T18:12:15Z,OWNER,"For the table actions: attaching it to a cog icon next to the table name could make sense.

<img width=""503"" alt=""data__faa-wildlife-strikes__26_rows_where_where_AIRPORT____OIL_RIG__sorted_by_rowid_and_Fix_table_name_in_spatialite_example_command_by_jsfenfen_·_Pull_Request__1022_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/96168896-45192c80-0ed6-11eb-9d90-6172a8346468.png"">

This is the column action icon at twice the size, color `#666`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/690#issuecomment-709500715,https://api.github.com/repos/simonw/datasette/issues/690,709500715,MDEyOklzc3VlQ29tbWVudDcwOTUwMDcxNQ==,9599,simonw,2020-10-15T18:10:32Z,2020-10-15T18:10:32Z,OWNER,"Row action menus are a question mark for me. Adding them to the table page itself could get really noisy - though maybe they're a vertical ... menu at the end of the row? They could also go on the `row.html` template, though that page is rarely used at the moment.

I'm going to skip row actions for the moment and concentrate on the other four, which I know I have use-cases for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/690#issuecomment-709499944,https://api.github.com/repos/simonw/datasette/issues/690,709499944,MDEyOklzc3VlQ29tbWVudDcwOTQ5OTk0NA==,9599,simonw,2020-10-15T18:09:06Z,2020-10-15T18:09:06Z,OWNER,"I'm going to dedicate this issue to thinking about action menus. #981 added column action menus, others to build could be:

- Table action menus (for things like configure FTS, edit schema)
- Database action menus (import CSV file into this table)
- Instance action menus (manage users, import CSV into this instance)
- Column action menus (already there, plugins could include extract this column)
- Row action menus?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/690#issuecomment-709498425,https://api.github.com/repos/simonw/datasette/issues/690,709498425,MDEyOklzc3VlQ29tbWVudDcwOTQ5ODQyNQ==,9599,simonw,2020-10-15T18:06:08Z,2020-10-15T18:06:08Z,OWNER,And for instance-level actions (linking to `datasette-import-csv` for example) an actions menu anchored against a burger-bar menu icon in the navigation bar.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/pull/1022#issuecomment-708718475,https://api.github.com/repos/simonw/datasette/issues/1022,708718475,MDEyOklzc3VlQ29tbWVudDcwODcxODQ3NQ==,9599,simonw,2020-10-14T23:46:38Z,2020-10-14T23:46:38Z,OWNER,"Good catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721830990,Fix table name in spatialite example command,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-708695761,https://api.github.com/repos/simonw/sqlite-utils/issues/182,708695761,MDEyOklzc3VlQ29tbWVudDcwODY5NTc2MQ==,9599,simonw,2020-10-14T22:33:55Z,2020-10-14T22:35:17Z,OWNER,"For the moment I'm going to add a `--encoding` option and some code that catches `UnicodeDecodeError` and shows an error message that suggests using `--encoding`.

That error message could detect if the `file` command is available and, if it is, suggest running `file filename.txt` to detect the character encoding.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-708695369,https://api.github.com/repos/simonw/sqlite-utils/issues/163,708695369,MDEyOklzc3VlQ29tbWVudDcwODY5NTM2OQ==,9599,simonw,2020-10-14T22:32:39Z,2020-10-14T22:32:39Z,OWNER,`sqlite-utils convert my.db mytable mycolumn` could become the new `sqlite-transform`. The sub-command could take `--code` but could also take `--parsedate` and `--parsedatetime` options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions,
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-708693290,https://api.github.com/repos/simonw/sqlite-utils/issues/163,708693290,MDEyOklzc3VlQ29tbWVudDcwODY5MzI5MA==,9599,simonw,2020-10-14T22:26:50Z,2020-10-14T22:27:48Z,OWNER,"This could even be exposed as a feature of the CLI tool - similar to how `sqlite-transform` works: https://github.com/simonw/sqlite-transform

    $ sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'

Maybe `sqlite-utils` should absorb all of the functionality from `sqlite-transform` - having two separate tools doesn't necessarily make sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-708693133,https://api.github.com/repos/simonw/sqlite-utils/issues/182,708693133,MDEyOklzc3VlQ29tbWVudDcwODY5MzEzMw==,9599,simonw,2020-10-14T22:26:26Z,2020-10-14T22:26:26Z,OWNER,"I could use https://github.com/chardet/chardet to help here, though I'd rather not add it as a dependency (`sqlite-utils` has very few dependencies at the moment). I could add it as an optional dependency though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/datasette/issues/1020#issuecomment-708670392,https://api.github.com/repos/simonw/datasette/issues/1020,708670392,MDEyOklzc3VlQ29tbWVudDcwODY3MDM5Mg==,9599,simonw,2020-10-14T21:29:22Z,2020-10-14T21:29:22Z,OWNER,"I should also verify (and probably unit-test) that things like the `?_trace=1` mechanism work across the internal request boundary.

`/-/permissions` appears to work across this boundary, but again a test would be useful confirmation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-708669778,https://api.github.com/repos/simonw/datasette/issues/1020,708669778,MDEyOklzc3VlQ29tbWVudDcwODY2OTc3OA==,9599,simonw,2020-10-14T21:27:58Z,2020-10-14T21:27:58Z,OWNER,"Maybe these internal requests should have some kind of flag that lets the underlying code tell that it's being called internally.

One option: add a `x-internal: 1` request header - and ensure that any requests from outside Datasette have that header stripped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1020#issuecomment-708669178,https://api.github.com/repos/simonw/datasette/issues/1020,708669178,MDEyOklzc3VlQ29tbWVudDcwODY2OTE3OA==,9599,simonw,2020-10-14T21:26:37Z,2020-10-14T21:26:37Z,OWNER,"One option: add an optional request=... parameter which can be passed the current request, and will use that to populate the mock request with the exception of the bits that are passed explicitly (like the path):

```python
response = await datasette.client.get(""/db/table.json"", request=request)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication,
https://github.com/simonw/datasette/issues/1019#issuecomment-708139822,https://api.github.com/repos/simonw/datasette/issues/1019,708139822,MDEyOklzc3VlQ29tbWVudDcwODEzOTgyMg==,9599,simonw,2020-10-14T03:53:21Z,2020-10-14T03:53:21Z,OWNER,"Demos:
- https://latest.datasette.io/fixtures/neighborhood_search has the link
- https://latest.datasette.io/fixtures/neighborhood_search?text=ber has the link, and the link passes through the parameter that has already been entered
- https://latest.datasette.io/fixtures/%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC has the link
- https://latest.datasette.io/fixtures/pragma_cache_size does not have the link (`PRAGMA` is not allowed in regular editable queries)
- https://latest.datasette.io/fixtures/magic_parameters does not have the link (magic parameters)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708130290,https://api.github.com/repos/simonw/datasette/issues/1019,708130290,MDEyOklzc3VlQ29tbWVudDcwODEzMDI5MA==,9599,simonw,2020-10-14T03:17:57Z,2020-10-14T03:17:57Z,OWNER,"One last edge-case (I think) - magic parameters aren't supported for non-canned-queries, so if a query includes those it shouldn't show an Edit SQL link: https://latest.datasette.io/fixtures/magic_parameters

I don't have a clean way of detecting if a query contains defined magic parameters, but I can instead do a dumb substring match for `:_` and skip showing the Edit SQL link if that is present.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708128286,https://api.github.com/repos/simonw/datasette/issues/1019,708128286,MDEyOklzc3VlQ29tbWVudDcwODEyODI4Ng==,9599,simonw,2020-10-14T03:11:33Z,2020-10-14T03:14:07Z,OWNER,"Another edge-case: https://latest.datasette.io/fixtures/pragma_cache_size (`PRAGMA cache_size;`) isn't an allowed query usually, so linking to ""Edit SQL"" for it would link to an error page: https://latest.datasette.io/fixtures?sql=PRAGMA+cache_size%3B

Can use `datasette.utils.validate_sql_select(sql)` to check for that - it raises `datasette.utils.InvalidSql` if there's a problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708127294,https://api.github.com/repos/simonw/datasette/issues/1019,708127294,MDEyOklzc3VlQ29tbWVudDcwODEyNzI5NA==,9599,simonw,2020-10-14T03:08:13Z,2020-10-14T03:08:13Z,OWNER,"Maybe like this:

<img width=""1385"" alt=""fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_"" src=""https://user-images.githubusercontent.com/9599/95938930-ce721700-0d8f-11eb-87c8-f67ea4e57d55.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708114636,https://api.github.com/repos/simonw/datasette/issues/1019,708114636,MDEyOklzc3VlQ29tbWVudDcwODExNDYzNg==,9599,simonw,2020-10-14T02:24:56Z,2020-10-14T02:24:56Z,OWNER,"So it would appear on this page: https://latest.datasette.io/fixtures/neighborhood_search#fragment-goes-here

Probably as a blue ""Edit SQL"" link next to those buttons.

![575FB160-4F75-43B2-B2AB-D75E60F10477](https://user-images.githubusercontent.com/9599/95936143-c2835680-0d89-11eb-9cf3-06f86d923103.jpeg)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/1019#issuecomment-708113989,https://api.github.com/repos/simonw/datasette/issues/1019,708113989,MDEyOklzc3VlQ29tbWVudDcwODExMzk4OQ==,9599,simonw,2020-10-14T02:22:40Z,2020-10-14T02:22:40Z,OWNER,"I've found myself wanting this too.

It should respect the allow-sql permission so it doesn't display if the user isn't able to run custom SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721050815,"""Edit SQL"" button on canned queries",
https://github.com/simonw/datasette/issues/690#issuecomment-706845950,https://api.github.com/repos/simonw/datasette/issues/690,706845950,MDEyOklzc3VlQ29tbWVudDcwNjg0NTk1MA==,9599,simonw,2020-10-12T04:07:28Z,2020-10-12T04:07:28Z,OWNER,Next step is to design where these will go and what they will look like - in particular the navigation links that lead to plugins such as `datasette-upload-csvs` and `datasette-configure-fts` and `datasette-edit-schema`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things,
https://github.com/simonw/datasette/issues/1016#issuecomment-706828375,https://api.github.com/repos/simonw/datasette/issues/1016,706828375,MDEyOklzc3VlQ29tbWVudDcwNjgyODM3NQ==,9599,simonw,2020-10-12T03:01:58Z,2020-10-12T03:01:58Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=created&created=2019-01-17+08%3A00%3A00&_facet=state&state=MI

![x-button](https://user-images.githubusercontent.com/9599/95700744-9b4b4e80-0bfc-11eb-85a3-6da82d9e0cbf.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/1016#issuecomment-706821561,https://api.github.com/repos/simonw/datasette/issues/1016,706821561,MDEyOklzc3VlQ29tbWVudDcwNjgyMTU2MQ==,9599,simonw,2020-10-12T02:33:52Z,2020-10-12T02:33:52Z,OWNER,"Should this do the equivalent of clicking Apply for you? I don't think so - that wouldn't fit with how the rest of that block of controls works.

But if it makes the row vanish it may not be clear to the user that they still need to click Apply.

Easiest thing would be for it to select the ""remove filter"" column and blank out the boxes. I will try that first.

The code can go in `table.js`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/1016#issuecomment-706788010,https://api.github.com/repos/simonw/datasette/issues/1016,706788010,MDEyOklzc3VlQ29tbWVudDcwNjc4ODAxMA==,9599,simonw,2020-10-11T23:50:39Z,2020-10-11T23:50:39Z,OWNER,"For consistency can reuse the icon used on selected facets:

<img width=""721"" alt=""evernote__notes__8_rows_where_where_application-data____ACD4B543-4663-40B2-BAF5-666FA54D4204__and_source-url_is_not_blank_sorted_by_id"" src=""https://user-images.githubusercontent.com/9599/95693564-e0ae5280-0be1-11eb-881b-fa0cec8397f5.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,"Add a ""delete"" icon next to filters (in addition to ""remove filter"")",
https://github.com/simonw/datasette/issues/1015#issuecomment-706756879,https://api.github.com/repos/simonw/datasette/issues/1015,706756879,MDEyOklzc3VlQ29tbWVudDcwNjc1Njg3OQ==,9599,simonw,2020-10-11T19:35:03Z,2020-10-11T19:35:03Z,OWNER,"Since plugins are installed via pip this would require Datasette to be restarted. This StackOverflow thread looks relevant to that: https://stackoverflow.com/questions/11329917/restart-python-script-from-within-itself

This recipe looks promising:
```python
import os
import sys
import psutil
import logging

def restart_program():
    """"""Restarts the current program, with file objects and descriptors
       cleanup
    """"""

    try:
        p = psutil.Process(os.getpid())
        for handler in p.get_open_files() + p.connections():
            os.close(handler.fd)
    except Exception, e:
        logging.error(e)

    python = sys.executable
    os.execl(python, python, *sys.argv)
```
https://docs.python.org/3/library/os.html#os.execl says about `os.execl`:

> These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718910318,Research: could Datasette install its own plugins?,
https://github.com/simonw/datasette/issues/782#issuecomment-706745236,https://api.github.com/repos/simonw/datasette/issues/782,706745236,MDEyOklzc3VlQ29tbWVudDcwNjc0NTIzNg==,9599,simonw,2020-10-11T18:16:05Z,2020-10-11T18:16:05Z,OWNER,Here's the `datasette-json-preview` plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706740250,https://api.github.com/repos/simonw/datasette/issues/782,706740250,MDEyOklzc3VlQ29tbWVudDcwNjc0MDI1MA==,9599,simonw,2020-10-11T17:40:48Z,2020-10-11T17:43:07Z,OWNER,"Building this plugin reminded me of an oddity of the `register_output_renderer()` plugin hook: one of the arguments that can be passed to it is `data`, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface.

That's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis.

So maybe that kind of stuff is re-implemented as ""extras"" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution.

To illustrate the problem (in this case the need to access `next_url`) here's my first prototype of the plugin:
```python
from datasette import hookimpl
from datasette.utils.asgi import Response


@hookimpl
def register_output_renderer(datasette):
    return {
        ""extension"": ""json-preview"",
        ""render"": json_preview,
    }


def json_preview(data, columns, rows):
    next_url = data.get(""next_url"")
    headers = {}
    if next_url:
        headers[""link""] = '<{}>; rel=""next""'.format(next_url)
    return Response.json([dict(zip(columns, row)) for row in rows], headers=headers)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706738020,https://api.github.com/repos/simonw/datasette/issues/782,706738020,MDEyOklzc3VlQ29tbWVudDcwNjczODAyMA==,9599,simonw,2020-10-11T17:23:18Z,2020-10-11T17:23:48Z,OWNER,"I'm going to prototype what it would look like if the default shape was a list of objects and `?_extra=` turns that into an object with a `rows` key, in a plugin. As a separate extension (maybe `.json-preview`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706735341,https://api.github.com/repos/simonw/datasette/issues/782,706735341,MDEyOklzc3VlQ29tbWVudDcwNjczNTM0MQ==,9599,simonw,2020-10-11T17:03:29Z,2020-10-11T17:15:34Z,OWNER,"Maybe `.jsonfull` becomes a new renderer that returns ALL of the defined `?_extra=` blocks.

Or... `?_extra=all` turns on ALL of the available information blocks (some of which can come from plugins).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706735200,https://api.github.com/repos/simonw/datasette/issues/782,706735200,MDEyOklzc3VlQ29tbWVudDcwNjczNTIwMA==,9599,simonw,2020-10-11T17:02:11Z,2020-10-11T17:14:51Z,OWNER,"Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe `?_extra=count`.

Based on that, the default JSON shape could look something like this:

```json
{
  ""rows"": [{""id"": 1}, {""id"": 2}],
  ""next"": ""2"",
  ""next_url"": ""/db/table?_next=2""
}
```
And with `?_extra=count`:
```json
{
  ""rows"": [{""id"": 1}, {""id"": 2}],
  ""next"": ""2"",
  ""next_url"": ""/db/table?_next=2"",
  ""count"": 31
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706736541,https://api.github.com/repos/simonw/datasette/issues/782,706736541,MDEyOklzc3VlQ29tbWVudDcwNjczNjU0MQ==,9599,simonw,2020-10-11T17:12:27Z,2020-10-11T17:12:27Z,OWNER,"The core issue that I keep reconsidering is whether the default `.json` representation should be an object or a list.

Arguments in favour of a list:

- It's what I always want. Almost all of the code that I've written against the API myself uses `?_shape=array`.
- It's really easy to use. You can pipe it to e.g. `sqlite-utils insert`, you can load it into JavaScript without thinking about it.

Arguments against:

- Nowhere to put pagination or total counts. I added pagination to the `link:` HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like `""truncated"": true` to indicate that more than 1000 results were returned and they were truncated.
- An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format.

But maybe that last point is a positive? It ensures the default `.json` format remains completely predictable forever.

If `.json` DID default to an array of objects, the `?_shape=` argument could still be used to get back alternative formats.

Maybe `.json?_extra=total` changes the shape of that default to be this instead:

```json
{
  ""rows"": [{""id"": 1}, {""id"": 2}],
  ""total"": 104
}
```

The thing I care about most though is `next_url`. That could be provided like so:

`.json?_extra=total&_extra=next` - alternative syntax `.json?_extra=total,next`:

```json
{
  ""rows"": [{""id"": 1}, {""id"": 2}],
  ""total"": 104,
  ""next"": ""2"",
  ""next_url"": ""/db/table.json?_extra=total&_extra=next&_next=2""
}
```
This is feeling a bit verbose for a common combination though.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-706735280,https://api.github.com/repos/simonw/datasette/issues/782,706735280,MDEyOklzc3VlQ29tbWVudDcwNjczNTI4MA==,9599,simonw,2020-10-11T17:03:01Z,2020-10-11T17:03:01Z,OWNER,"Should that default also include `""columns""` as a list of strings? That would be duplicate data of the keys in the `""rows""` list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1014#issuecomment-706631006,https://api.github.com/repos/simonw/datasette/issues/1014,706631006,MDEyOklzc3VlQ29tbWVudDcwNjYzMTAwNg==,9599,simonw,2020-10-11T00:36:43Z,2020-10-11T00:36:43Z,OWNER,"Demo using [paginate-json](https://github.com/simonw/paginate-json):
```
% paginate-json 'https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array' | jq '. | length' 
https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cd%2Cv
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ch%2Cr
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cl%2Cn
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cp%2Cj
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ct%2Cf
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cx%2Cb
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ca%2Cx
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ce%2Ct
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ci%2Cp
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Cm%2Cl
1001
```
New documentation: https://docs.datasette.io/en/latest/json_api.html#pagination","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706627035,https://api.github.com/repos/simonw/datasette/issues/1014,706627035,MDEyOklzc3VlQ29tbWVudDcwNjYyNzAzNQ==,9599,simonw,2020-10-10T23:50:40Z,2020-10-10T23:50:40Z,OWNER,"First example page to add this to: `?_shape=array`, e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626971,https://api.github.com/repos/simonw/datasette/issues/1014,706626971,MDEyOklzc3VlQ29tbWVudDcwNjYyNjk3MQ==,9599,simonw,2020-10-10T23:49:57Z,2020-10-10T23:49:57Z,OWNER,I won't offer a total-number header. That's expensive to calculate - no need to calculate it unless it's explicitly asked for.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626934,https://api.github.com/repos/simonw/datasette/issues/1014,706626934,MDEyOklzc3VlQ29tbWVudDcwNjYyNjkzNA==,9599,simonw,2020-10-10T23:49:25Z,2020-10-10T23:49:25Z,OWNER,I'm just going to implement the 'next' one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626913,https://api.github.com/repos/simonw/datasette/issues/1014,706626913,MDEyOklzc3VlQ29tbWVudDcwNjYyNjkxMw==,9599,simonw,2020-10-10T23:49:08Z,2020-10-10T23:49:08Z,OWNER,This should be documented on https://docs.datasette.io/en/stable/json_api.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1014#issuecomment-706626796,https://api.github.com/repos/simonw/datasette/issues/1014,706626796,MDEyOklzc3VlQ29tbWVudDcwNjYyNjc5Ng==,9599,simonw,2020-10-10T23:47:23Z,2020-10-10T23:48:26Z,OWNER,"The GitHub API has these:

`link: <https://api.github.com/repositories/107914493/commits?page=2>; rel=""next"", <https://api.github.com/repositories/107914493/commits?page=46>; rel=""last""`

WordPress API: https://css-tricks.com/wp-json/wp/v2/posts

`link: <https://css-tricks.com/wp-json/wp/v2/posts?page=2>; rel=""next""`

It also has these:

```
x-wp-total: 5162
x-wp-totalpages: 517
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,Add Link: pagination HTTP headers,
https://github.com/simonw/datasette/issues/1011#issuecomment-706483680,https://api.github.com/repos/simonw/datasette/issues/1011,706483680,MDEyOklzc3VlQ29tbWVudDcwNjQ4MzY4MA==,9599,simonw,2020-10-10T04:09:32Z,2020-10-10T04:09:32Z,OWNER,"OK, Datasette 0.50.2 is live on PyPI now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,
https://github.com/simonw/datasette/issues/1011#issuecomment-706482542,https://api.github.com/repos/simonw/datasette/issues/1011,706482542,MDEyOklzc3VlQ29tbWVudDcwNjQ4MjU0Mg==,9599,simonw,2020-10-10T03:58:58Z,2020-10-10T03:58:58Z,OWNER,The fix is live on the demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_sort_desc=pk2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,
https://github.com/simonw/datasette/issues/1011#issuecomment-706480791,https://api.github.com/repos/simonw/datasette/issues/1011,706480791,MDEyOklzc3VlQ29tbWVudDcwNjQ4MDc5MQ==,9599,simonw,2020-10-10T03:43:13Z,2020-10-10T03:43:13Z,OWNER,Demo: click column headers on https://latest.datasette.io/fixtures/compound_three_primary_keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,
https://github.com/simonw/datasette/issues/1011#issuecomment-706480763,https://api.github.com/repos/simonw/datasette/issues/1011,706480763,MDEyOklzc3VlQ29tbWVudDcwNjQ4MDc2Mw==,9599,simonw,2020-10-10T03:42:58Z,2020-10-10T03:42:58Z,OWNER,"Oh no! It's the same bug as #1010. Thanks for the report, I shall push out 0.50.2 as quickly as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718521469,column name links broken in 0.50.1,
https://github.com/simonw/datasette/issues/1005#issuecomment-706473306,https://api.github.com/repos/simonw/datasette/issues/1005,706473306,MDEyOklzc3VlQ29tbWVudDcwNjQ3MzMwNg==,9599,simonw,2020-10-10T02:45:46Z,2020-10-10T02:45:46Z,OWNER,"https://github.com/simonw/datasette/blob/9f6dd985bc0eff70f8a9ce65c6578bc43d2e172b/tests/test_html.py#L145-L149

https://github.com/simonw/datasette/blob/9f6dd985bc0eff70f8a9ce65c6578bc43d2e172b/tests/test_html.py#L539-L564

https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/tests/test_api.py#L1190-L1194

https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/tests/test_api.py#L742-L746","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718259202,Remove xfail tests when new httpx is released,
https://github.com/simonw/datasette/issues/1010#issuecomment-706455505,https://api.github.com/repos/simonw/datasette/issues/1010,706455505,MDEyOklzc3VlQ29tbWVudDcwNjQ1NTUwNQ==,9599,simonw,2020-10-10T00:35:41Z,2020-10-10T00:35:41Z,OWNER,I missed the links on the query page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718484082,json / CSV links are broken in Datasette 0.50,
https://github.com/simonw/datasette/issues/1010#issuecomment-706455202,https://api.github.com/repos/simonw/datasette/issues/1010,706455202,MDEyOklzc3VlQ29tbWVudDcwNjQ1NTIwMg==,9599,simonw,2020-10-10T00:33:46Z,2020-10-10T00:33:46Z,OWNER,"Since this is an urgent fix that needs to go out I've marked the `base_url` tests as `xfail`. I need to solve those separately, see #900 and various other bug reports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718484082,json / CSV links are broken in Datasette 0.50,
https://github.com/simonw/datasette/issues/1010#issuecomment-706450418,https://api.github.com/repos/simonw/datasette/issues/1010,706450418,MDEyOklzc3VlQ29tbWVudDcwNjQ1MDQxOA==,9599,simonw,2020-10-10T00:08:27Z,2020-10-10T00:09:13Z,OWNER,"This commit broke it: https://github.com/simonw/datasette/commit/8f97b9b58e77f82fef1f10e9c9f6754b993544b6#diff-ed4bb3a97b7efa4d45be6a6ea6f4636cL114-R165

<img width=""1106"" alt=""Banners_and_Alerts_and_datasette_client_internal_requests_mechanism_·_simonw_datasette_8f97b9b"" src=""https://user-images.githubusercontent.com/9599/95640559-25a37f00-0a52-11eb-838e-9f771ab5a413.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718484082,json / CSV links are broken in Datasette 0.50,
https://github.com/simonw/datasette/issues/1002#issuecomment-706306214,https://api.github.com/repos/simonw/datasette/issues/1002,706306214,MDEyOklzc3VlQ29tbWVudDcwNjMwNjIxNA==,9599,simonw,2020-10-09T17:23:51Z,2020-10-09T17:23:51Z,OWNER,I can start by combining the release notes for the 0.50 alphas.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717783692,Release notes for Datasette 0.50,
https://github.com/simonw/datasette/issues/943#issuecomment-706305784,https://api.github.com/repos/simonw/datasette/issues/943,706305784,MDEyOklzc3VlQ29tbWVudDcwNjMwNTc4NA==,9599,simonw,2020-10-09T17:22:55Z,2020-10-09T17:22:55Z,OWNER,Documentation (from #1006): https://docs.datasette.io/en/latest/internals.html#client,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/1006#issuecomment-706305601,https://api.github.com/repos/simonw/datasette/issues/1006,706305601,MDEyOklzc3VlQ29tbWVudDcwNjMwNTYwMQ==,9599,simonw,2020-10-09T17:22:31Z,2020-10-09T17:22:31Z,OWNER,https://docs.datasette.io/en/latest/internals.html#client,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718264811,Documentation for datasette.client,
https://github.com/simonw/datasette/issues/1003#issuecomment-706281451,https://api.github.com/repos/simonw/datasette/issues/1003,706281451,MDEyOklzc3VlQ29tbWVudDcwNjI4MTQ1MQ==,9599,simonw,2020-10-09T16:33:01Z,2020-10-09T16:33:01Z,OWNER,I think `json_dumps()` and `json_loads()` as aliases for `json.dumps()` and `json.loads()` is the way to go here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718238967,from_json jinja2 filter,
https://github.com/simonw/datasette/issues/1007#issuecomment-706276831,https://api.github.com/repos/simonw/datasette/issues/1007,706276831,MDEyOklzc3VlQ29tbWVudDcwNjI3NjgzMQ==,9599,simonw,2020-10-09T16:23:51Z,2020-10-09T16:23:51Z,OWNER,"I don't appear to be using these anywhere, not sure why I spotted a warning (which I now can't find).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718272593,set-env and add-path commands have been deprecated,
https://github.com/simonw/datasette/issues/1003#issuecomment-706273211,https://api.github.com/repos/simonw/datasette/issues/1003,706273211,MDEyOklzc3VlQ29tbWVudDcwNjI3MzIxMQ==,9599,simonw,2020-10-09T16:16:38Z,2020-10-09T16:16:38Z,OWNER,"I'm not a huge fan of `from_json` as the name for this. Some other options:

- Expose `json` directly so templates can do `json.loads()` and `json.dumps()` - this allows for outputting JSON too, which is useful. But is there anything else on the `json` module that shouldn't be exposed in templates?
- `json_dumps()` and `json_loads()` template functions. I quite like that.
- Something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718238967,from_json jinja2 filter,
https://github.com/simonw/datasette/issues/1003#issuecomment-706272322,https://api.github.com/repos/simonw/datasette/issues/1003,706272322,MDEyOklzc3VlQ29tbWVudDcwNjI3MjMyMg==,9599,simonw,2020-10-09T16:14:56Z,2020-10-09T16:14:56Z,OWNER,Yes I think that makes sense. I added `json` to the template context in Dogsheep Beta just a few days ago because I needed that: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L176,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718238967,from_json jinja2 filter,
https://github.com/simonw/datasette/issues/1006#issuecomment-706270877,https://api.github.com/repos/simonw/datasette/issues/1006,706270877,MDEyOklzc3VlQ29tbWVudDcwNjI3MDg3Nw==,9599,simonw,2020-10-09T16:12:09Z,2020-10-09T16:12:09Z,OWNER,This can become a section on https://docs.datasette.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718264811,Documentation for datasette.client,
https://github.com/simonw/datasette/pull/1000#issuecomment-706269271,https://api.github.com/repos/simonw/datasette/issues/1000,706269271,MDEyOklzc3VlQ29tbWVudDcwNjI2OTI3MQ==,9599,simonw,2020-10-09T16:08:49Z,2020-10-09T16:08:49Z,OWNER,I'm going to document this in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-706263157,https://api.github.com/repos/simonw/datasette/issues/1000,706263157,MDEyOklzc3VlQ29tbWVudDcwNjI2MzE1Nw==,9599,simonw,2020-10-09T15:57:15Z,2020-10-09T15:57:15Z,OWNER,"My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.

I'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705946360,https://api.github.com/repos/simonw/datasette/issues/1000,705946360,MDEyOklzc3VlQ29tbWVudDcwNTk0NjM2MA==,9599,simonw,2020-10-09T03:28:08Z,2020-10-09T03:28:08Z,OWNER,"Here's where `httpx` sets up the ASGI scope: https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L82-L97

```python
        # ASGI scope.
        scheme, host, port, full_path = url
        path, _, query = full_path.partition(b""?"")
        scope = {
            ""type"": ""http"",
            ""asgi"": {""version"": ""3.0""},
            ""http_version"": ""1.1"",
            ""method"": method.decode(),
            ""headers"": [(k.lower(), v) for (k, v) in headers],
            ""scheme"": scheme.decode(""ascii""),
            ""path"": unquote(path.decode(""ascii"")),
            ""query_string"": query,
            ""server"": (host.decode(""ascii""), port),
            ""client"": self.client,
            ""root_path"": self.root_path,
        }
```
Sure enough, it doesn't set the `raw_path`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705946120,https://api.github.com/repos/simonw/datasette/issues/1000,705946120,MDEyOklzc3VlQ29tbWVudDcwNTk0NjEyMA==,9599,simonw,2020-10-09T03:27:05Z,2020-10-09T03:27:05Z,OWNER,"I may need to fuss around with how the `httpx` client sends things to the ASGI app.

https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L26 is relevant:

    Alternatively, you can setup the transport instance explicitly.
    This allows you to include any additional configuration arguments specific
    to the ASGITransport class:
    ```
    transport = httpx.ASGITransport(
        app=app,
        root_path=""/submount"",
        client=(""1.2.3.4"", 123)
    )
    client = httpx.AsyncClient(transport=transport)
    ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705945591,https://api.github.com/repos/simonw/datasette/issues/1000,705945591,MDEyOklzc3VlQ29tbWVudDcwNTk0NTU5MQ==,9599,simonw,2020-10-09T03:24:48Z,2020-10-09T03:24:48Z,OWNER,"I'm testing this with a `print(scope)` and `pytest -k test_table_with_slashes_in_name -s`.

Against the `main` branch:

`{'type': 'http', 'http_version': '1.0', 'method': 'GET', 'path': '/fixtures/table/with/slashes.csv', 'raw_path': b'/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'headers': [[b'host', b'localhost']], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x10e2e6040>}`

Against the broken branch:

`tests/test_api.py {'type': 'http', 'asgi': {'version': '3.0'}, 'http_version': '1.1', 'method': 'GET', 'headers': [(b'host', b'localhost'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'user-agent', b'python-httpx/0.15.0')], 'scheme': 'http', 'path': '/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'server': ('localhost', None), 'client': ('127.0.0.1', 123), 'root_path': '', 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x109e0eca0>}`

This is on my laptop though so both of those pass the tests.

Key difference: the `httpx` version doesn't set a `raw_path` at all. BUT.. it does set `path` and sets it to `'/fixtures/table%2Fwith%2Fslashes.csv'`

The non-httpx version sets `raw_path` to `b'/fixtures/table%2Fwith%2Fslashes.csv'` and `path` to `'/fixtures/table/with/slashes.csv'`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705941580,https://api.github.com/repos/simonw/datasette/issues/1000,705941580,MDEyOklzc3VlQ29tbWVudDcwNTk0MTU4MA==,9599,simonw,2020-10-09T03:08:43Z,2020-10-09T03:08:43Z,OWNER,"Most likely reason for those failures is that `path` and `raw_path` are not being simulated correctly. I used to do those here:

https://github.com/simonw/datasette/blob/402cf870b7d65f9b5fba9e23aa99433294bd4523/datasette/utils/testing.py#L116-L125

But now I'm delegating that to `httpx` to handle.

WEIRD that it passes on my laptop but fails in GitHub Actions CI though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705940507,https://api.github.com/repos/simonw/datasette/issues/1000,705940507,MDEyOklzc3VlQ29tbWVudDcwNTk0MDUwNw==,9599,simonw,2020-10-09T03:04:15Z,2020-10-09T03:04:15Z,OWNER,"This is really weird: new set of test failures that I wasn't seeing before, and those tests aren't failing on my laptop:
```
=========================== short test summary info ============================
FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200
FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200
FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert...
FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5]
FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]
================== 5 failed, 738 passed in 194.73s (0:03:14) ===================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705937696,https://api.github.com/repos/simonw/datasette/issues/1000,705937696,MDEyOklzc3VlQ29tbWVudDcwNTkzNzY5Ng==,9599,simonw,2020-10-09T02:52:53Z,2020-10-09T02:52:53Z,OWNER,"These failures are giving me a severe ""how did this ever work in the first place?"" vibe:
```
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/]
```
I have a fix for them, no idea why they weren't already failing though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705926445,https://api.github.com/repos/simonw/datasette/issues/1000,705926445,MDEyOklzc3VlQ29tbWVudDcwNTkyNjQ0NQ==,9599,simonw,2020-10-09T02:15:38Z,2020-10-09T02:15:38Z,OWNER,"> FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'

That one is caused by `response.cookies` skipping cookies that were set to the empty string. Same fix as this: https://github.com/simonw/datasette/blob/a1687351fb75b01f737fda4ad07e0781029de05c/tests/test_auth.py#L90-L95","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705926035,https://api.github.com/repos/simonw/datasette/issues/1000,705926035,MDEyOklzc3VlQ29tbWVudDcwNTkyNjAzNQ==,9599,simonw,2020-10-09T02:14:14Z,2020-10-09T02:14:14Z,OWNER,"Still need to handle these six failing tests:
```
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'elemen...
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', '...
FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<...
FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<p cla...
FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'
FAILED tests/test_plugins.py::test_hook_register_magic_parameters - AssertionError: assert [{'line': '1.0', 'rowid': 1}] == [{'line': '1.1', 'rowid': 1}]
=========================================================== 6 failed, 731 passed, 6 warnings in 129.17s (0:02:09) ===========================================================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705921006,https://api.github.com/repos/simonw/datasette/issues/1000,705921006,MDEyOklzc3VlQ29tbWVudDcwNTkyMTAwNg==,9599,simonw,2020-10-09T01:55:01Z,2020-10-09T01:55:01Z,OWNER,"With the single client that is reused for all tests:
```
% time pytest tests/test_api.py
...
6.73s user 9.91s system 81% cpu 20.365 total
```
After switching back to this class:
```python
class DatasetteClient:
    def __init__(self, ds):
        self.app = ds.app()

    def _fix(self, path):
        if path.startswith(""/""):
            path = ""http://localhost{}"".format(path)
        return path

    async def get(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.get(self._fix(path), **kwargs)

    async def options(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.options(self._fix(path), **kwargs)

    async def head(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.head(self._fix(path), **kwargs)

    async def post(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.post(self._fix(path), **kwargs)

    async def put(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.put(self._fix(path), **kwargs)

    async def patch(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.patch(self._fix(path), **kwargs)

    async def delete(self, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.delete(self._fix(path), **kwargs)

    async def request(self, method, path, **kwargs):
        async with httpx.AsyncClient(app=self.app) as client:
            return await client.request(method, self._fix(path), **kwargs)
```
The time taken is:
```
% time pytest tests/test_api.py
...
7.26s user 10.02s system 82% cpu 21.014 total
```
That's close enough that I don't feel I need to investigate this further.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705920228,https://api.github.com/repos/simonw/datasette/issues/1000,705920228,MDEyOklzc3VlQ29tbWVudDcwNTkyMDIyOA==,9599,simonw,2020-10-09T01:51:44Z,2020-10-09T01:51:44Z,OWNER,I'm going to switch back to having each request run through a new client. I'm worried about the impact on test performance though. I'll run a microbenchmark before and after.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705920055,https://api.github.com/repos/simonw/datasette/issues/1000,705920055,MDEyOklzc3VlQ29tbWVudDcwNTkyMDA1NQ==,9599,simonw,2020-10-09T01:51:05Z,2020-10-09T01:51:05Z,OWNER,"The topic of disabling cookie persistence is discussed a little here: https://github.com/encode/httpx/issues/422#issuecomment-537906693

> We have a cookiejar abstraction, I think setting it to an always-empty jar like you describe is best. :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705918844,https://api.github.com/repos/simonw/datasette/issues/1000,705918844,MDEyOklzc3VlQ29tbWVudDcwNTkxODg0NA==,9599,simonw,2020-10-09T01:46:06Z,2020-10-09T01:46:06Z,OWNER,"For this failing test I'm suspicious that the AsyncClient may be persisting cookies in between requests:
```
    def test_actor_cookie(app_client):
        ""A valid actor cookie sets request.scope['actor']""
        cookie = app_client.actor_cookie({""id"": ""test""})
        response = app_client.get(""/"", cookies={""ds_actor"": cookie})
>       assert {""id"": ""test""} == app_client.ds._last_request.scope[""actor""]
E       AssertionError: assert {'id': 'test'} == {'id': 'root'}
E         Differing items:
E         {'id': 'test'} != {'id': 'root'}
E         Use -v to get the full diff
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/issues/1001#issuecomment-705917015,https://api.github.com/repos/simonw/datasette/issues/1001,705917015,MDEyOklzc3VlQ29tbWVudDcwNTkxNzAxNQ==,9599,simonw,2020-10-09T01:38:49Z,2020-10-09T01:38:49Z,OWNER,"I actually have a sensible `OPTIONS` implementation here:

https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/datasette/views/base.py#L154-L165

I'm going to set the default one to return a 405 (Method Not Allowed) like Google does.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705916614,https://api.github.com/repos/simonw/datasette/issues/1001,705916614,MDEyOklzc3VlQ29tbWVudDcwNTkxNjYxNA==,9599,simonw,2020-10-09T01:37:06Z,2020-10-09T01:37:06Z,OWNER,"I'm tempted to imitate Django Rest Framework here: https://github.com/encode/django-rest-framework/blob/2e721cdbc85a924d0b0f093b86fe1497b58fe287/rest_framework/views.py#L514-L521
```python
    def options(self, request, *args, **kwargs):
        """"""
        Handler method for HTTP 'OPTIONS' request.
        """"""
        if self.metadata_class is None:
            return self.http_method_not_allowed(request, *args, **kwargs)
        data = self.metadata_class().determine_metadata(request, self)
        return Response(data, status=status.HTTP_200_OK)
```
That `determine_metadata()` default method does this: https://github.com/encode/django-rest-framework/blob/335054a5d36b352a58286b303b608b6bf48152f8/rest_framework/metadata.py#L29

Note the comment at the top:
```
    There are not any formalized standards for `OPTIONS` responses
    for us to base this on.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705905418,https://api.github.com/repos/simonw/datasette/issues/1001,705905418,MDEyOklzc3VlQ29tbWVudDcwNTkwNTQxOA==,9599,simonw,2020-10-09T01:00:54Z,2020-10-09T01:00:54Z,OWNER,https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705905121,https://api.github.com/repos/simonw/datasette/issues/1001,705905121,MDEyOklzc3VlQ29tbWVudDcwNTkwNTEyMQ==,9599,simonw,2020-10-09T01:00:07Z,2020-10-09T01:00:07Z,OWNER,"The www.djangoproject.com site returns an empty page:
```
~ % curl -vv -XOPTIONS https://www.djangoproject.com/
*   Trying 151.101.42.217:443...
* Connected to www.djangoproject.com (151.101.42.217) port 443 (#0)
* ALPN, offering http/1.1
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: osff2.map.fastly.net
* Server certificate: GlobalSign CloudSSL CA - SHA256 - G3
* Server certificate: GlobalSign Root CA
> OPTIONS / HTTP/1.1
> Host: www.djangoproject.com
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: keep-alive
< Content-Length: 0
< Server: nginx
< Content-Type: text/html; charset=utf-8
< Allow: GET, HEAD, OPTIONS
< Content-Language: en
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Access-Control-Allow-Origin: https://code.djangoproject.com
< Accept-Ranges: bytes
< Date: Fri, 09 Oct 2020 00:59:42 GMT
< Via: 1.1 varnish
< X-Served-By: cache-sjc10047-SJC
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1602205182.833493,VS0,VE305
< Vary: Accept-Language, Accept-Encoding
< 
* Connection #0 to host www.djangoproject.com left intact
~ % 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705904917,https://api.github.com/repos/simonw/datasette/issues/1001,705904917,MDEyOklzc3VlQ29tbWVudDcwNTkwNDkxNw==,9599,simonw,2020-10-09T00:59:25Z,2020-10-09T00:59:25Z,OWNER,"```
~ % curl -XOPTIONS https://www.google.com/
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content=""initial-scale=1, minimum-scale=1, width=device-width"">
  <title>Error 405 (Method Not Allowed)!!1</title>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705904759,https://api.github.com/repos/simonw/datasette/issues/1001,705904759,MDEyOklzc3VlQ29tbWVudDcwNTkwNDc1OQ==,9599,simonw,2020-10-09T00:58:47Z,2020-10-09T00:58:47Z,OWNER,"What should an OPTIONS request return, anyway?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705904679,https://api.github.com/repos/simonw/datasette/issues/1001,705904679,MDEyOklzc3VlQ29tbWVudDcwNTkwNDY3OQ==,9599,simonw,2020-10-09T00:58:32Z,2020-10-09T00:58:32Z,OWNER,So the bug is in this code here: https://github.com/simonw/datasette/blob/703439bdc37e724b01bc6d7a1fc1d955795132f2/datasette/views/base.py#L113-L115,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/issues/1001#issuecomment-705904566,https://api.github.com/repos/simonw/datasette/issues/1001,705904566,MDEyOklzc3VlQ29tbWVudDcwNTkwNDU2Ng==,9599,simonw,2020-10-09T00:58:08Z,2020-10-09T00:58:08Z,OWNER,"To get a traceback:
```
datasette . -p 8009 --pdb
```
And then:
```
curl -XOPTIONS http://127.0.0.1:8009
```
This causes the server to open a debugging prompt:
```
INFO:     127.0.0.1:59514 - ""OPTIONS / HTTP/1.1"" 500 Internal Server Error
> /Users/simon/Dropbox/Development/datasette/datasette/views/base.py(115)dispatch_request()
-> return await handler(request, *args, **kwargs)
(Pdb) list
110  	    def database_color(self, database):
111  	        return ""ff0000""
112  	
113  	    async def dispatch_request(self, request, *args, **kwargs):
114  	        handler = getattr(self, request.method.lower(), None)
115  ->	        return await handler(request, *args, **kwargs)
116  	
117  	    async def render(self, templates, request, context=None):
118  	        context = context or {}
119  	        template = self.ds.jinja_env.select_template(templates)
120  	        template_context = {
(Pdb) 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717768441,OPTIONS requests return a 500 error,
https://github.com/simonw/datasette/pull/1000#issuecomment-705902902,https://api.github.com/repos/simonw/datasette/issues/1000,705902902,MDEyOklzc3VlQ29tbWVudDcwNTkwMjkwMg==,9599,simonw,2020-10-09T00:50:49Z,2020-10-09T00:50:49Z,OWNER,"Almost all of the tests are passing:
```
=========================== short test summary info ============================
FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200
FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200
FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert...
FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5]
FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/]
FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/]
FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyE...
FAILED tests/test_plugins.py::test_hook_register_magic_parameters - Assertion...
============ 11 failed, 718 passed, 6 warnings in 225.77s (0:03:45) ============
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705899629,https://api.github.com/repos/simonw/datasette/issues/1000,705899629,MDEyOklzc3VlQ29tbWVudDcwNTg5OTYyOQ==,9599,simonw,2020-10-09T00:37:02Z,2020-10-09T00:37:02Z,OWNER,I'm going to route the existing `TestClient` through this mechanism to exercise it during the tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/pull/1000#issuecomment-705889120,https://api.github.com/repos/simonw/datasette/issues/1000,705889120,MDEyOklzc3VlQ29tbWVudDcwNTg4OTEyMA==,9599,simonw,2020-10-08T23:59:01Z,2020-10-08T23:59:01Z,OWNER,Needs tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,datasette.client internal requests mechanism,
https://github.com/simonw/datasette/issues/943#issuecomment-705887638,https://api.github.com/repos/simonw/datasette/issues/943,705887638,MDEyOklzc3VlQ29tbWVudDcwNTg4NzYzOA==,9599,simonw,2020-10-08T23:53:13Z,2020-10-08T23:53:13Z,OWNER,"I want this in Datasette 0.50, so I can use it in `datasette-graphql` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/997#issuecomment-705883053,https://api.github.com/repos/simonw/datasette/issues/997,705883053,MDEyOklzc3VlQ29tbWVudDcwNTg4MzA1Mw==,9599,simonw,2020-10-08T23:36:55Z,2020-10-08T23:36:55Z,OWNER,Docs are now live here: https://docs.datasette.io/en/latest/deploying.html#deploying-using-buildpacks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716988478,Documentation covering buildpack deployment,
https://github.com/simonw/datasette/issues/514#issuecomment-705878827,https://api.github.com/repos/simonw/datasette/issues/514,705878827,MDEyOklzc3VlQ29tbWVudDcwNTg3ODgyNw==,9599,simonw,2020-10-08T23:33:05Z,2020-10-08T23:33:05Z,OWNER,Documentation is now live at https://docs.datasette.io/en/latest/deploying.html#running-datasette-using-systemd,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459397625,Documentation with recommendations on running Datasette in production without using Docker,
https://github.com/simonw/datasette/issues/969#issuecomment-705874457,https://api.github.com/repos/simonw/datasette/issues/969,705874457,MDEyOklzc3VlQ29tbWVudDcwNTg3NDQ1Nw==,9599,simonw,2020-10-08T23:27:30Z,2020-10-08T23:27:30Z,OWNER,For the moment I'm going to ship this as the `--tar=` option. Can consider detecting `gtar` in the future.,"{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",705057955,"Add --tar option to ""datasette publish heroku""",
https://github.com/simonw/datasette/issues/969#issuecomment-705874317,https://api.github.com/repos/simonw/datasette/issues/969,705874317,MDEyOklzc3VlQ29tbWVudDcwNTg3NDMxNw==,9599,simonw,2020-10-08T23:27:00Z,2020-10-08T23:27:00Z,OWNER,"I'm working on this now. Just ran this test and it worked (no longer showed my the warning message).

    datasette publish heroku fixtures.db -n datasette-issue-999 --branch=main --tar=/usr/local/bin/gtar

But... I wonder if it should automatically do this if it detects that `gtar` is available?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705057955,"Add --tar option to ""datasette publish heroku""",
https://github.com/simonw/datasette/issues/999#issuecomment-705872767,https://api.github.com/repos/simonw/datasette/issues/999,705872767,MDEyOklzc3VlQ29tbWVudDcwNTg3Mjc2Nw==,9599,simonw,2020-10-08T23:21:36Z,2020-10-08T23:21:36Z,OWNER,"I ran this to prove that it breaks:

    datasette publish heroku fixtures.db -n datasette-issue-999

https://datasette-issue-999.herokuapp.com/ returned an error, as expected - here's the relevant logs:

```
$ heroku logs --tail -a datasette-issue-999
2020-10-08T23:19:01.065103+00:00 heroku[web.1]: Starting process with command `datasette serve --host 0.0.0.0 -i fixtures.db --cors --port 22329 --inspect-file inspect-data.json`
2020-10-08T23:19:05.381374+00:00 app[web.1]: WARNING:  You must pass the application as an import string to enable 'reload' or 'workers'.
2020-10-08T23:19:05.617519+00:00 heroku[web.1]: Process exited with status 1
2020-10-08T23:19:05.690673+00:00 heroku[web.1]: State changed from starting to crashed
```
Then I deployed again using:

    datasette publish heroku fixtures.db -n datasette-issue-999 --branch=main

This time the deploy worked OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717729056,Datasette should default to running Uvicorn with workers=1,
https://github.com/simonw/datasette/issues/999#issuecomment-705871513,https://api.github.com/repos/simonw/datasette/issues/999,705871513,MDEyOklzc3VlQ29tbWVudDcwNTg3MTUxMw==,9599,simonw,2020-10-08T23:17:31Z,2020-10-08T23:17:31Z,OWNER,This does mean that any deploys to Heroku will fail with an error until this code has been released in a non-alpha Datasette release (unless you use `--branch=main`) - so I should ship this as soon as possible.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717729056,Datasette should default to running Uvicorn with workers=1,
https://github.com/simonw/datasette/issues/999#issuecomment-705869129,https://api.github.com/repos/simonw/datasette/issues/999,705869129,MDEyOklzc3VlQ29tbWVudDcwNTg2OTEyOQ==,9599,simonw,2020-10-08T23:09:41Z,2020-10-08T23:09:41Z,OWNER,I want to fix this now because then I don't need to document it in #997.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717729056,Datasette should default to running Uvicorn with workers=1,
https://github.com/simonw/datasette/issues/998#issuecomment-705851405,https://api.github.com/repos/simonw/datasette/issues/998,705851405,MDEyOklzc3VlQ29tbWVudDcwNTg1MTQwNQ==,9599,simonw,2020-10-08T22:14:31Z,2020-10-08T22:14:31Z,OWNER,So maybe this is blocked on #987 and adding a new hook in the HTML which plugins like `datasette-cluster-map` can attach their content to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/998#issuecomment-705851211,https://api.github.com/repos/simonw/datasette/issues/998,705851211,MDEyOklzc3VlQ29tbWVudDcwNTg1MTIxMQ==,9599,simonw,2020-10-08T22:14:00Z,2020-10-08T22:14:00Z,OWNER,"I tried this out and it did cause one problem with plugins: `datasette-cluster-map` still injected itself inside the scrolling area, when instead we would want that map to stay outside of the div with the table in it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/datasette/issues/865#issuecomment-705799046,https://api.github.com/repos/simonw/datasette/issues/865,705799046,MDEyOklzc3VlQ29tbWVudDcwNTc5OTA0Ng==,9599,simonw,2020-10-08T20:15:26Z,2020-10-08T20:15:26Z,OWNER,Can anyone give me some detailed steps to reproduce for this issue? I think I need to set up my own local nginx or Apache proxy so I can really dig into this and figure out what's going on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/997#issuecomment-705466873,https://api.github.com/repos/simonw/datasette/issues/997,705466873,MDEyOklzc3VlQ29tbWVudDcwNTQ2Njg3Mw==,9599,simonw,2020-10-08T10:02:02Z,2020-10-08T10:02:02Z,OWNER,"I asked about other hosts for these on Twitter https://twitter.com/simonw/status/1314042631949676544

- https://buildpacks.io/
- https://doc.scalingo.com/languages/python/start
- https://cloud.google.com/run/docs/continuous-deployment-with-cloud-build","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716988478,Documentation covering buildpack deployment,
https://github.com/simonw/datasette/issues/997#issuecomment-705306117,https://api.github.com/repos/simonw/datasette/issues/997,705306117,MDEyOklzc3VlQ29tbWVudDcwNTMwNjExNw==,9599,simonw,2020-10-08T03:22:16Z,2020-10-08T03:22:16Z,OWNER,Note #633 - you need to set WEB_CONCURRENCY=1 on Heroku for this to work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716988478,Documentation covering buildpack deployment,
https://github.com/simonw/datasette/issues/991#issuecomment-705244583,https://api.github.com/repos/simonw/datasette/issues/991,705244583,MDEyOklzc3VlQ29tbWVudDcwNTI0NDU4Mw==,9599,simonw,2020-10-07T23:24:56Z,2020-10-07T23:24:56Z,OWNER,"Sketched out an option:
```diff
diff --git a/datasette/templates/index.html b/datasette/templates/index.html
index c1adfc5..eab3053 100644
--- a/datasette/templates/index.html
+++ b/datasette/templates/index.html
@@ -5,6 +5,14 @@
 {% block body_class %}index{% endblock %}
 
 {% block content %}
+<style>
+.db-table {
+    padding-left: 2em;
+}
+.db-table h3 {
+    margin: 0 0 0.15em 0;
+}
+</style>
 <h1>{{ metadata.title or ""Datasette"" }}{% if private %} 🔒{% endif %}</h1>
 
 {% block description_source_license %}{% include ""_description_source_license.html"" %}{% endblock %}
@@ -21,8 +29,20 @@
             {{ ""{:,}"".format(database.views_count) }} view{% if database.views_count != 1 %}s{% endif %}
         {% endif %}
     </p>
-    <p>{% for table in database.tables_and_views_truncated %}<a href=""{{ database.path }}/{{ table.name|quote_plus 
-    }}""{% if table.count %} title=""{{ table.count }} rows""{% endif %}>{{ table.name }}</a>{% if table.private %} 🔒{% endif %}{% if not loop.last %}, {% endif %}{% endfor %}{% if database.tables_and_views_more %}, <a href=""{{ database.path }}"">...</a>{% endif %}</p>
+    {% for table in database.tables_and_views_truncated %}
+        {% if show_hidden or not table.hidden %}
+        <div class=""db-table"">
+            <h3><a href=""{{ database_url(database.name) }}/{{ table.name|quote_plus }}"">{{ table.name }}</a>{% if table.private %} 🔒{% endif %}{% if table.hidden %}<em> (hidden)</em>{% endif %}</h3>
+            <p><em>{% for column in table.columns[:9] %}{{ column }}{% if not loop.last %}, {% endif %}{% endfor %}{% if table.columns|length > 9 %}...{% endif %}</em></p>
+            <p>{% if table.count is none %}Many rows{% else %}{{ ""{:,}"".format(table.count) }} row{% if table.count == 1 %}{% else %}s{% endif %}{% endif %}</p>
+        </div>
+        {% endif %}
+    {% endfor %}
+
+    {% if database.tables_and_views_more %}
+        <p><a href=""{{ database.path }}"">All {{ database.tables_count }} tables in {{ database.name }}</a></p>
+    {% endif %}
+
 {% endfor %}
 
 {% endblock %}
```

![image](https://user-images.githubusercontent.com/9599/95398019-9c594480-08b9-11eb-90c9-f322d647ee3b.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/996#issuecomment-705119801,https://api.github.com/repos/simonw/datasette/issues/996,705119801,MDEyOklzc3VlQ29tbWVudDcwNTExOTgwMQ==,9599,simonw,2020-10-07T18:34:58Z,2020-10-07T18:34:58Z,OWNER,One option: compare the length of the matches and pick the shortest one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716756082,Better handling of multiple matching template wildcard paths,
https://github.com/simonw/datasette/issues/996#issuecomment-705116123,https://api.github.com/repos/simonw/datasette/issues/996,705116123,MDEyOklzc3VlQ29tbWVudDcwNTExNjEyMw==,9599,simonw,2020-10-07T18:27:57Z,2020-10-07T18:28:36Z,OWNER,"This line was meant to handle this case, by only matching non-/ characters. Looks like it did not have the desired effect: https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1204

Maybe because the finished regex doesn't include `^` and `$` anchors?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716756082,Better handling of multiple matching template wildcard paths,
https://github.com/simonw/datasette/issues/996#issuecomment-705115185,https://api.github.com/repos/simonw/datasette/issues/996,705115185,MDEyOklzc3VlQ29tbWVudDcwNTExNTE4NQ==,9599,simonw,2020-10-07T18:26:11Z,2020-10-07T18:26:45Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1030-L1037

https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L951-L954

https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1197-L1207","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",716756082,Better handling of multiple matching template wildcard paths,
https://github.com/simonw/datasette/pull/995#issuecomment-704395913,https://api.github.com/repos/simonw/datasette/issues/995,704395913,MDEyOklzc3VlQ29tbWVudDcwNDM5NTkxMw==,9599,simonw,2020-10-06T16:25:20Z,2020-10-06T16:25:20Z,OWNER,"This is great, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",715779909,Document setting Google Cloud SDK properties,
https://github.com/simonw/datasette/issues/992#issuecomment-704013305,https://api.github.com/repos/simonw/datasette/issues/992,704013305,MDEyOklzc3VlQ29tbWVudDcwNDAxMzMwNQ==,9599,simonw,2020-10-06T04:00:00Z,2020-10-06T04:00:00Z,OWNER,I can add this well in advance of Datasette 1.0 as an alias to the existing `--config` option (but taking two arguments instead of one).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/993#issuecomment-703963211,https://api.github.com/repos/simonw/datasette/issues/993,703963211,MDEyOklzc3VlQ29tbWVudDcwMzk2MzIxMQ==,9599,simonw,2020-10-06T00:40:48Z,2020-10-06T00:40:48Z,OWNER,"Now live at https://latest.datasette.io/fixtures/facetable

New `await db.table_column_details(table)` method is documented here: https://docs.datasette.io/en/latest/internals.html#database-introspection","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",715072935,Column action menu should show column type,
https://github.com/simonw/datasette/issues/993#issuecomment-703928029,https://api.github.com/repos/simonw/datasette/issues/993,703928029,MDEyOklzc3VlQ29tbWVudDcwMzkyODAyOQ==,9599,simonw,2020-10-05T22:42:45Z,2020-10-05T22:42:59Z,OWNER,"<img width=""574"" alt=""fixtures__facetable__15_rows"" src=""https://user-images.githubusercontent.com/9599/95139485-65e8c180-0721-11eb-9e91-263aad6f6846.png"">

The `NOT NULL` text shows up only for columns that are not null.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",715072935,Column action menu should show column type,
https://github.com/simonw/datasette/issues/992#issuecomment-703351641,https://api.github.com/repos/simonw/datasette/issues/992,703351641,MDEyOklzc3VlQ29tbWVudDcwMzM1MTY0MQ==,9599,simonw,2020-10-05T01:31:36Z,2020-10-05T01:31:36Z,OWNER,If I rename metadata to config in #493 then this should be `--setting` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/741#issuecomment-703350401,https://api.github.com/repos/simonw/datasette/issues/741,703350401,MDEyOklzc3VlQ29tbWVudDcwMzM1MDQwMQ==,9599,simonw,2020-10-05T01:26:12Z,2020-10-05T01:26:12Z,OWNER,Relevant support question on Twitter: https://twitter.com/simonw/status/1312926103627988993,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/778#issuecomment-702493047,https://api.github.com/repos/simonw/datasette/issues/778,702493047,MDEyOklzc3VlQ29tbWVudDcwMjQ5MzA0Nw==,9599,simonw,2020-10-02T02:26:25Z,2020-10-02T02:26:25Z,OWNER,"I think this could work for arbitrary SQL queries too. Those would need querystring configuration that specifies which sorted column(s) should be used for the ""next"" cursor.

One example: I'd like to be able to offer a paginated list of counts of values in a table - e.g. this query:

https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=select+replies%2C+count%28*%29+from+%5Btwitter-ratio%2Fsenators%5D+group+by+replies+order+by+count%28*%29+desc%3B

That could even become a query that gets linked to from the column actions menu.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",626211658,Ability to configure keyset pagination for views and queries,
https://github.com/simonw/datasette/issues/565#issuecomment-702459747,https://api.github.com/repos/simonw/datasette/issues/565,702459747,MDEyOklzc3VlQ29tbWVudDcwMjQ1OTc0Nw==,9599,simonw,2020-10-02T00:09:55Z,2020-10-02T00:09:55Z,OWNER,"Today Datasette uses `""click~=7.1.1""` and Uvicorn uses `""click==7.*""` so I think we are OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473307794,Conflict between datasette and uvicorn click versions,
https://github.com/simonw/datasette/issues/446#issuecomment-702459432,https://api.github.com/repos/simonw/datasette/issues/446,702459432,MDEyOklzc3VlQ29tbWVudDcwMjQ1OTQzMg==,9599,simonw,2020-10-02T00:08:47Z,2020-10-02T00:08:47Z,OWNER,"I've decided not to bother with this, especially having redesigned some hooks to return Datasette standard `Response` objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440134714,Define mechanism for plugins to return structured data,
https://github.com/simonw/datasette/issues/865#issuecomment-702418045,https://api.github.com/repos/simonw/datasette/issues/865,702418045,MDEyOklzc3VlQ29tbWVudDcwMjQxODA0NQ==,9599,simonw,2020-10-01T21:54:00Z,2020-10-01T21:54:00Z,OWNER,"Had a thought: this is likely to break in plugins too, such as `datasette-edit-schema` which constructs URLs for redirects e.g. here: https://github.com/simonw/datasette-edit-schema/blob/dbd0abee6dd3385b114cfe9671f7ead1c4855b60/datasette_edit_schema/__init__.py#L46-L48","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/pull/986#issuecomment-702265255,https://api.github.com/repos/simonw/datasette/issues/986,702265255,MDEyOklzc3VlQ29tbWVudDcwMjI2NTI1NQ==,9599,simonw,2020-10-01T16:51:45Z,2020-10-01T16:51:45Z,OWNER,Thanks for taking a look! The fix ended up being a little different from this because I still want to disable faceting on regular single primary keys (since faceting by those won't ever produce interesting results) - here's what I used: https://github.com/simonw/datasette/commit/5d6bc4c268f9f155e59561671f8617addd3e91bc,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712889459,"Allow facet by primary keys, fixes #985",
https://github.com/simonw/datasette/issues/987#issuecomment-702241397,https://api.github.com/repos/simonw/datasette/issues/987,702241397,MDEyOklzc3VlQ29tbWVudDcwMjI0MTM5Nw==,9599,simonw,2020-10-01T16:11:31Z,2020-10-01T16:11:31Z,OWNER,"This should be a new page in the documentation, which will be expanded later on to cover things like the proposed JavaScript plugin hooks system from #983.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/987#issuecomment-702240916,https://api.github.com/repos/simonw/datasette/issues/987,702240916,MDEyOklzc3VlQ29tbWVudDcwMjI0MDkxNg==,9599,simonw,2020-10-01T16:10:44Z,2020-10-01T16:10:44Z,OWNER,https://github.com/simonw/datasette/blob/141544613f9e76ddb74eee38d6f8ee1e0e70f833/datasette/templates/_table.html#L6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/985#issuecomment-702132019,https://api.github.com/repos/simonw/datasette/issues/985,702132019,MDEyOklzc3VlQ29tbWVudDcwMjEzMjAxOQ==,9599,simonw,2020-10-01T13:23:48Z,2020-10-01T13:23:48Z,OWNER,"The code that skips the ""facet by this"" menu option for primary key columns could take into account if there are multiple primary keys.

https://github.com/simonw/datasette/blob/141544613f9e76ddb74eee38d6f8ee1e0e70f833/datasette/static/table.js#L91-L98","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712839383,Column actions should support facet by compound primary keys,
https://github.com/simonw/datasette/issues/530#issuecomment-701720034,https://api.github.com/repos/simonw/datasette/issues/530,701720034,MDEyOklzc3VlQ29tbWVudDcwMTcyMDAzNA==,9599,simonw,2020-10-01T00:42:08Z,2020-10-01T00:42:08Z,OWNER,"I've changed my mind - especially since Datasette now ships with default JavaScript for the column menus in #981. I'm not going to extract CodeMirror into a plugin, I'll leave it where it is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",460540321,Extract codemirror SQL editor out into a plugin,
https://github.com/simonw/datasette/issues/984#issuecomment-701708072,https://api.github.com/repos/simonw/datasette/issues/984,701708072,MDEyOklzc3VlQ29tbWVudDcwMTcwODA3Mg==,9599,simonw,2020-10-01T00:01:36Z,2020-10-01T00:01:36Z,OWNER,Column action menus are the cog icons on https://latest.datasette.io/fixtures/facetable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712368432,Review accessibility of new column action menus,
https://github.com/simonw/datasette/issues/981#issuecomment-701707116,https://api.github.com/repos/simonw/datasette/issues/981,701707116,MDEyOklzc3VlQ29tbWVudDcwMTcwNzExNg==,9599,simonw,2020-09-30T23:58:17Z,2020-09-30T23:58:17Z,OWNER,"Now live at https://latest.datasette.io/fixtures/facetable

<img width=""1264"" alt=""fixtures__facetable__15_rows"" src=""https://user-images.githubusercontent.com/9599/94751748-1f205380-033e-11eb-940b-bb4ee7728d31.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701704716,https://api.github.com/repos/simonw/datasette/issues/981,701704716,MDEyOklzc3VlQ29tbWVudDcwMTcwNDcxNg==,9599,simonw,2020-09-30T23:48:36Z,2020-09-30T23:48:36Z,OWNER,"Since this menu doesn't provide new functionality, I'm going to ignore the fact that it doesn't exist on portrait mobile view for the moment. Likewise, I'm going to skip making it accessible for the moment since lacking accessibility doesn't prevent functionality from being accessed - the menu-less experience currently works the same as the portrait mobile experience.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701703472,https://api.github.com/repos/simonw/datasette/issues/981,701703472,MDEyOklzc3VlQ29tbWVudDcwMTcwMzQ3Mg==,9599,simonw,2020-09-30T23:43:30Z,2020-09-30T23:43:30Z,OWNER,"I'm going to go based just on the visible values on the current page. I think that's good enough, and it avoids the complexity involved in doing a server-side check for blank values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701697918,https://api.github.com/repos/simonw/datasette/issues/981,701697918,MDEyOklzc3VlQ29tbWVudDcwMTY5NzkxOA==,9599,simonw,2020-09-30T23:24:14Z,2020-09-30T23:25:28Z,OWNER,"I could provide the ""Show non-blank values"" option only for columns where there are blank values visible on the current page.

This could be a bit confusing though, since the absence of that option could suggest that there are no blank values at all when that's actually not true.

One option: run a separate `fetch()` call that figures out if any of the columns contain blank values, which gets a bit of extra time to execute. Only show the ""Show non-blank values"" option in the menu once that has returned.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701679729,https://api.github.com/repos/simonw/datasette/issues/981,701679729,MDEyOklzc3VlQ29tbWVudDcwMTY3OTcyOQ==,9599,simonw,2020-09-30T22:26:33Z,2020-09-30T22:26:33Z,OWNER,"Bug: https://latest.datasette.io/fixtures/sortable?_sort=sortable

![sort-bug](https://user-images.githubusercontent.com/9599/94746304-4e7c9380-0331-11eb-8e81-7245529c4654.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/sqlite-utils/issues/183#issuecomment-701678659,https://api.github.com/repos/simonw/sqlite-utils/issues/183,701678659,MDEyOklzc3VlQ29tbWVudDcwMTY3ODY1OQ==,9599,simonw,2020-09-30T22:23:36Z,2020-09-30T22:23:36Z,OWNER,"It just ran and didn't find anything. I'll leave it enabled for a while but I may turn it off again, it could be that this kind of Python library isn't really what it's useful for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712316959,Try out GitHub code scanning,
https://github.com/simonw/datasette/issues/981#issuecomment-701668526,https://api.github.com/repos/simonw/datasette/issues/981,701668526,MDEyOklzc3VlQ29tbWVudDcwMTY2ODUyNg==,9599,simonw,2020-09-30T21:57:22Z,2020-09-30T21:57:22Z,OWNER,"A bunch of things to fix:

- It clobbers existing querystring parameters - it needs to leave these alone (but replace the current sort order)
- Facet option should not show up if you are already faceting by that column
- There's no way to close the menu once it has opened!
- Accessibility: SVG icon doesn't even have an alt attribute yet. Should use ARIA when the thing appears.

It's also not visible on mobile, need to think about how that will work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701664306,https://api.github.com/repos/simonw/datasette/issues/981,701664306,MDEyOklzc3VlQ29tbWVudDcwMTY2NDMwNg==,9599,simonw,2020-09-30T21:47:08Z,2020-09-30T21:47:08Z,OWNER,"The arrow icon didn't make sense because I already have a triangle icon showing sort order. I'm trying a cog icon instead:

<img width=""303"" alt=""data__dogs__10_rows"" src=""https://user-images.githubusercontent.com/9599/94743176-ce9ffa80-032b-11eb-8f4d-da3962984a4f.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701659197,https://api.github.com/repos/simonw/datasette/issues/981,701659197,MDEyOklzc3VlQ29tbWVudDcwMTY1OTE5Nw==,9599,simonw,2020-09-30T21:34:44Z,2020-09-30T21:34:44Z,OWNER,"Showing ""facet by this"" on the primary key column doesn't make sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701642448,https://api.github.com/repos/simonw/datasette/issues/981,701642448,MDEyOklzc3VlQ29tbWVudDcwMTY0MjQ0OA==,9599,simonw,2020-09-30T20:59:33Z,2020-09-30T20:59:33Z,OWNER,I think I've got everything I need to implement this now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/983#issuecomment-701629984,https://api.github.com/repos/simonw/datasette/issues/983,701629984,MDEyOklzc3VlQ29tbWVudDcwMTYyOTk4NA==,9599,simonw,2020-09-30T20:34:43Z,2020-09-30T20:34:43Z,OWNER,"I had a look around and there isn't an obvious pluggy equivalent in JavaScript world at the moment. Lots of frameworks like jQuery and Vue have their own custom plugin mechanisms.

https://github.com/rekit/js-plugin is a simple standalone plugin mechanism. Not quite as full-featured as Pluggy though - in particular I like how Pluggy supports multiple plugins returning results for the same hook that get concatenated into a list of results.

https://css-tricks.com/designing-a-javascript-plugin-system/ has some ideas.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/sqlite-utils/pull/178#issuecomment-701627158,https://api.github.com/repos/simonw/sqlite-utils/issues/178,701627158,MDEyOklzc3VlQ29tbWVudDcwMTYyNzE1OA==,9599,simonw,2020-09-30T20:29:11Z,2020-09-30T20:29:11Z,OWNER,Thanks for the fix!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709043182,Update README.md,
https://github.com/simonw/sqlite-utils/issues/182#issuecomment-701626134,https://api.github.com/repos/simonw/sqlite-utils/issues/182,701626134,MDEyOklzc3VlQ29tbWVudDcwMTYyNjEzNA==,9599,simonw,2020-09-30T20:27:09Z,2020-09-30T20:27:42Z,OWNER,"It looks like http://maps.natalian.org/data.txt is encoded as `latin-1`, but `sqlite-utils` assumes `utf-8` and hence breaks.

It would be worth improving the error message here. I could also add a `--encoding latin-1` option to `sqlite-utils insert` to help in consuming files that are stored in charsets other than `utf-8`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""",
https://github.com/simonw/datasette/issues/981#issuecomment-701616922,https://api.github.com/repos/simonw/datasette/issues/981,701616922,MDEyOklzc3VlQ29tbWVudDcwMTYxNjkyMg==,9599,simonw,2020-09-30T20:08:02Z,2020-09-30T20:08:02Z,OWNER,It would be neat to provide a JavaScript plugin hook that plugins can use to add their own options to this menu. No idea what that would look like though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701157010,https://api.github.com/repos/simonw/datasette/issues/981,701157010,MDEyOklzc3VlQ29tbWVudDcwMTE1NzAxMA==,9599,simonw,2020-09-30T05:00:42Z,2020-09-30T20:06:59Z,OWNER,"Maybe use this as the icon:
```svg
<svg width=""100px"" height=""100px"" viewBox=""0 0 255 255"">
  <polygon points=""0,64 128,191 255,64""/>
</svg>
```

<img width=""41"" alt=""Real-time_HTML_Editor"" src=""https://user-images.githubusercontent.com/9599/94734258-d22c8500-031d-11eb-982b-82cde4b1920f.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701615291,https://api.github.com/repos/simonw/datasette/issues/981,701615291,MDEyOklzc3VlQ29tbWVudDcwMTYxNTI5MQ==,9599,simonw,2020-09-30T20:04:34Z,2020-09-30T20:05:37Z,OWNER,"Another potential action:

* Show rows where this is not blank (equivalent to `is not blank` filter)

This could be displayed conditionally based on if the column is detected to have any blank rows in it?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/982#issuecomment-701585695,https://api.github.com/repos/simonw/datasette/issues/982,701585695,MDEyOklzc3VlQ29tbWVudDcwMTU4NTY5NQ==,9599,simonw,2020-09-30T19:06:29Z,2020-09-30T19:06:29Z,OWNER,This is a little related to the error display issue #619 in that both will require some reworking of how the code is structured.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712202333,"SQL editor should allow execution of write queries, if you have permission",
https://github.com/simonw/datasette/issues/982#issuecomment-701585075,https://api.github.com/repos/simonw/datasette/issues/982,701585075,MDEyOklzc3VlQ29tbWVudDcwMTU4NTA3NQ==,9599,simonw,2020-09-30T19:05:11Z,2020-09-30T19:05:11Z,OWNER,The form needs to switch from GET to POST if the query is a write query. JavaScript can handle this based on the checkbox - if a user does not have JavaScript submitting the form will cause the form action to be changed to POST and the form to be redisplayed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712202333,"SQL editor should allow execution of write queries, if you have permission",
https://github.com/simonw/datasette/issues/981#issuecomment-701153543,https://api.github.com/repos/simonw/datasette/issues/981,701153543,MDEyOklzc3VlQ29tbWVudDcwMTE1MzU0Mw==,9599,simonw,2020-09-30T04:46:05Z,2020-09-30T05:11:45Z,OWNER,"Prototype:
```html
<style>
body {
font-family: helvetica;
}
.dropdown-menu {
  display: inline-flex;
  border: 1px solid #ccc;
  border-radius: 4px;
  line-height: 1.4;
  font-size: 12px;
  box-shadow: 2px 2px 2px #aaa;
}
.dropdown-menu ul,
.dropdown-menu li {
  list-style-type: none;
  margin: 0;
  padding: 0;
}
.dropdown-menu li {
  border-bottom: 1px solid #ccc;
}
.dropdown-menu li:last-child {
  border: none;
}
.dropdown-menu a:link,
.dropdown-menu a:visited,
.dropdown-menu a:hover,
.dropdown-menu a:focus
.dropdown-menu a:active {
  text-decoration: none;
  display: block;
  padding: 4px 8px 2px 8px;
  color: #222;
  background-color: #fff;
}
.dropdown-menu a:hover {
  background-color: #eee;
}
.hook {
  display: block;
  position: absolute;
  top: 3px;
  left: 12px;
  width: 0; 
  height: 0; 
  border-left: 5px solid transparent;
  border-right: 5px solid transparent;
  border-bottom: 5px solid #ccc;
}
</style>
<div class=""dropdown-menu"">
<div class=""hook""></div>
<ul>
  <li><a href=""#"">Sort descending</a></li>
  <li><a href=""#"">Sort ascending</a></li>
  <li><a href=""#"">Facet by this</a></li>
  <li><a href=""#"">Count by this</a></li>
</ul>
</div>
```
<img width=""148"" alt=""Real-time_HTML_Editor"" src=""https://user-images.githubusercontent.com/9599/94643784-7241ca00-029c-11eb-8554-863fcc255352.png"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701153982,https://api.github.com/repos/simonw/datasette/issues/981,701153982,MDEyOklzc3VlQ29tbWVudDcwMTE1Mzk4Mg==,9599,simonw,2020-09-30T04:47:54Z,2020-09-30T04:47:54Z,OWNER,"I think the accessible way to do this is with absolute positioning - have a menu icon in the `<th>` which, when clicked, causes the dropdown menu to appear as an absolutely positioned `<div>` that is not located within the DOM hierarchy of the`<th>` itself but is positioned to show up in the correct place.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701153822,https://api.github.com/repos/simonw/datasette/issues/981,701153822,MDEyOklzc3VlQ29tbWVudDcwMTE1MzgyMg==,9599,simonw,2020-09-30T04:47:10Z,2020-09-30T04:47:10Z,OWNER,"Future version could have expanding out nested side menus that let you do things like ""calculate sum/avg for this column against this-other-column"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/981#issuecomment-701153600,https://api.github.com/repos/simonw/datasette/issues/981,701153600,MDEyOklzc3VlQ29tbWVudDcwMTE1MzYwMA==,9599,simonw,2020-09-30T04:46:18Z,2020-09-30T04:46:18Z,OWNER,"More options:
```html
<ul>
  <li><a href=""#"">Sort descending</a></li>
  <li><a href=""#"">Sort ascending</a></li>
  <li><a href=""#"">Facet by this</a></li>
  <li><a href=""#"">Unique values</a></li>
  <li><a href=""#"">Count unique values</a></li>
</ul>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns,
https://github.com/simonw/datasette/issues/980#issuecomment-700929721,https://api.github.com/repos/simonw/datasette/issues/980,700929721,MDEyOklzc3VlQ29tbWVudDcwMDkyOTcyMQ==,9599,simonw,2020-09-29T19:21:50Z,2020-09-29T19:21:50Z,OWNER,"That fixed it: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule(%27FREQ%3DHOURLY%3BCOUNT%3D5%27)%2C%0D%0A++dateutil_rrule_date(%0D%0A++++%27FREQ%3DDAILY%3BCOUNT%3D3%27%2C%0D%0A++++%271st+jan+2020%27%0D%0A++)%3B

<img width=""525"" alt=""fixtures__select_dateutil_rrule__FREQ_HOURLY_COUNT_5____dateutil_rrule_date___FREQ_DAILY_COUNT_3____1st_jan_2020____"" src=""https://user-images.githubusercontent.com/9599/94605680-4d266a80-024e-11eb-8818-bc8bd7958df4.png"">

```html
<style>
@media only screen and (max-width: 576px) {

    .rows-and-columns td:nth-of-type(1):before { content: ""dateutil_rrule(\000027FREQ=HOURLY;COUNT=5\000027)""; }

    .rows-and-columns td:nth-of-type(2):before { content: ""dateutil_rrule_date(\00000A    \000027FREQ=DAILY;COUNT=3\000027,\00000A    \0000271st jan 2020\000027\00000A  )""; }

}
</style>
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710819020,Another rendering glitch with column headers on mobile,
https://github.com/simonw/datasette/issues/980#issuecomment-700490225,https://api.github.com/repos/simonw/datasette/issues/980,700490225,MDEyOklzc3VlQ29tbWVudDcwMDQ5MDIyNQ==,9599,simonw,2020-09-29T06:53:37Z,2020-09-29T06:53:37Z,OWNER,"This time it's because there are newlines in the column header:
```html

<style>
@media only screen and (max-width: 576px) {

    .rows-and-columns td:nth-of-type(1):before { content: ""dateutil_rrule(\000027FREQ=HOURLY;COUNT=5\000027)""; }

    .rows-and-columns td:nth-of-type(2):before { content: ""dateutil_rrule_date(
\00000A    \000027FREQ=DAILY;COUNT=3\000027,
\00000A    \0000271st jan 2020\000027
\00000A  )""; }

}
</style>
```
Those need to be escaped somehow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710819020,Another rendering glitch with column headers on mobile,
https://github.com/simonw/datasette/issues/979#issuecomment-700343373,https://api.github.com/repos/simonw/datasette/issues/979,700343373,MDEyOklzc3VlQ29tbWVudDcwMDM0MzM3Mw==,9599,simonw,2020-09-28T23:56:27Z,2020-09-28T23:56:27Z,OWNER,This would benefit https://github.com/simonw/datasette-import-table - which currently ignores the `CREATE TABLE` and derives the schema by inserting rows.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710650633,Default table view JSON should include CREATE TABLE,
https://github.com/simonw/datasette/issues/979#issuecomment-700343229,https://api.github.com/repos/simonw/datasette/issues/979,700343229,MDEyOklzc3VlQ29tbWVudDcwMDM0MzIyOQ==,9599,simonw,2020-09-28T23:55:55Z,2020-09-28T23:55:55Z,OWNER,Here's the code that adds it to the HTML context: https://github.com/simonw/datasette/blob/c11383e6284e000b2641569457efa16ac9e0d6ae/datasette/views/table.py#L835-L837,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710650633,Default table view JSON should include CREATE TABLE,
https://github.com/simonw/datasette/issues/978#issuecomment-700320480,https://api.github.com/repos/simonw/datasette/issues/978,700320480,MDEyOklzc3VlQ29tbWVudDcwMDMyMDQ4MA==,9599,simonw,2020-09-28T22:39:18Z,2020-09-28T22:39:18Z,OWNER,"```python
def escape_css_string(s):
    return _css_re.sub(lambda m: ""\\"" + (""{:X}"".format(ord(m.group())).zfill(6)), s)
```
That fixes it:
<img width=""363"" alt=""data__select_dateutil_parse__10_october_2020_3pm____dateutil_easter__2020____dateutil_parse_fuzzy__This_is_due_10_september____dateutil_parse__1_2_2020____dateutil_parse__2020-03-04____dateutil_parse_dayfirst__2020-03-04____dateutil_easter_"" src=""https://user-images.githubusercontent.com/9599/94493173-c23b6680-01a0-11eb-9468-e972c51b015c.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700319656,https://api.github.com/repos/simonw/datasette/issues/978,700319656,MDEyOklzc3VlQ29tbWVudDcwMDMxOTY1Ng==,9599,simonw,2020-09-28T22:36:44Z,2020-09-28T22:36:44Z,OWNER,"Weirdly even those leading 0s doesn't fix it:

<img width=""600"" alt=""Real-time_HTML_Editor_and_Using_character_escapes_in_markup_and_CSS"" src=""https://user-images.githubusercontent.com/9599/94492937-44775b00-01a0-11eb-9c5f-e991af620404.png"">

But... padding to six characters does! See https://www.w3.org/International/questions/qa-escapes

<img width=""607"" alt=""Real-time_HTML_Editor_and_Using_character_escapes_in_markup_and_CSS"" src=""https://user-images.githubusercontent.com/9599/94492988-61139300-01a0-11eb-8304-bffe448c7d2b.png"">

```
In [32]: print('\\' + ""{:X}"".format(ord('""')).zfill(6))
\000022
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700317760,https://api.github.com/repos/simonw/datasette/issues/978,700317760,MDEyOklzc3VlQ29tbWVudDcwMDMxNzc2MA==,9599,simonw,2020-09-28T22:30:25Z,2020-09-28T22:30:25Z,OWNER,"```python
print('\\' + ""{:X}"".format(ord('""')).zfill(4))
\0022
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700316511,https://api.github.com/repos/simonw/datasette/issues/978,700316511,MDEyOklzc3VlQ29tbWVudDcwMDMxNjUxMQ==,9599,simonw,2020-09-28T22:26:38Z,2020-09-28T22:26:38Z,OWNER,The fix may be to use `\0022` instead of `\22`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700314509,https://api.github.com/repos/simonw/datasette/issues/978,700314509,MDEyOklzc3VlQ29tbWVudDcwMDMxNDUwOQ==,9599,simonw,2020-09-28T22:20:51Z,2020-09-28T22:20:51Z,OWNER,"Here's the HTML for the broken example above:
```html
<style>
@media only screen and (max-width: 576px) {

    .rows-and-columns td:nth-of-type(1):before { content: ""dateutil_parse(\2210 october 2020 3pm\22)""; }

    .rows-and-columns td:nth-of-type(2):before { content: ""dateutil_easter(\222020\22)""; }

    .rows-and-columns td:nth-of-type(3):before { content: ""dateutil_parse_fuzzy(\22This is due 10 september\22)""; }

    .rows-and-columns td:nth-of-type(4):before { content: ""dateutil_parse(\221/2/2020\22)""; }

    .rows-and-columns td:nth-of-type(5):before { content: ""dateutil_parse(\222020-03-04\22)""; }

    .rows-and-columns td:nth-of-type(6):before { content: ""dateutil_parse_dayfirst(\222020-03-04\22)""; }

    .rows-and-columns td:nth-of-type(7):before { content: ""dateutil_easter(2020)""; }

}
</style>
```
The glitch affects the ones where the quote is followed by digits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/datasette/issues/978#issuecomment-700313836,https://api.github.com/repos/simonw/datasette/issues/978,700313836,MDEyOklzc3VlQ29tbWVudDcwMDMxMzgzNg==,9599,simonw,2020-09-28T22:19:05Z,2020-09-28T22:19:05Z,OWNER,Looks like a bug in this function: https://github.com/simonw/datasette/blob/1f021c37110fc9019b0ef70062c28c335e568ae2/datasette/utils/__init__.py#L269-L274,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile,
https://github.com/simonw/sqlite-utils/issues/181#issuecomment-699762881,https://api.github.com/repos/simonw/sqlite-utils/issues/181,699762881,MDEyOklzc3VlQ29tbWVudDY5OTc2Mjg4MQ==,9599,simonw,2020-09-28T04:29:23Z,2020-09-28T04:29:23Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/94fc62857ee2655a21d85f6dae84b67bbfa5956d/sqlite_utils/db.py#L331-L367,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709920027,"pk=[""id""] should have same effect as pk=""id""",
https://github.com/simonw/sqlite-utils/issues/180#issuecomment-699718788,https://api.github.com/repos/simonw/sqlite-utils/issues/180,699718788,MDEyOklzc3VlQ29tbWVudDY5OTcxODc4OA==,9599,simonw,2020-09-28T01:11:45Z,2020-09-28T01:11:45Z,OWNER,https://hypothesis.readthedocs.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709861194,Try running some tests using Hypothesis,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699524671,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699524671,MDEyOklzc3VlQ29tbWVudDY5OTUyNDY3MQ==,9599,simonw,2020-09-26T17:31:23Z,2020-09-27T20:31:50Z,OWNER,"SQL query for detecting integers:
```sql
select
  'contains_non_integer' as result
from
  mytable
where
  cast(cast(mycolumn AS INTEGER) AS TEXT) != mycolumn
limit
  1
```
This will return a single row with a 1 as soon as it comes across a column that contains a non-integer - so it short circuits quickly on TEXT columns with non-integers in them.

If everything in the column is an integer it will scan the whole thing before returning no rows.

More extensive demo:
```sql
select
  value,
  cast(cast(value AS INTEGER) AS TEXT) = value as is_valid_int
from
  (
    select
      '1' as value
    union
    select
      '1.1' as value
    union
    select
      'dog' as value
    union
    select
      null as value
  )
```
https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+INTEGER%29+AS+TEXT%29+%3D+value+as+is_valid_int%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29


value | is_valid_int
-- | --
  |  
1 | 1
1.1 | 0
dog | 0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699684535,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699684535,MDEyOklzc3VlQ29tbWVudDY5OTY4NDUzNQ==,9599,simonw,2020-09-27T20:30:31Z,2020-09-27T20:30:31Z,OWNER,"This recipe looks like it might be the way to detect floats:
```sql
select
  value,
  cast(cast(value AS REAL) AS TEXT) in (value, value || '.0') as is_valid_float
from
  (
    select
      '1' as value
    union
    select
      '1.1' as value
    union
    select
      'dog' as value
    union
    select
      null as value
  )
```
Demo: https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+REAL%29+AS+TEXT%29+in+%28value%2C+value+%7C%7C+%27.0%27%29+as+is_valid_float%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29


value | is_valid_float
-- | --
  |  
1 | 1
1.1 | 1
dog | 0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699526149,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699526149,MDEyOklzc3VlQ29tbWVudDY5OTUyNjE0OQ==,9599,simonw,2020-09-26T17:43:28Z,2020-09-26T17:43:28Z,OWNER,Posed a question about this on the SQLite forum here: https://sqlite.org/forum/forumpost/ab0dcd66ef,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/138#issuecomment-698626768,https://api.github.com/repos/simonw/sqlite-utils/issues/138,698626768,MDEyOklzc3VlQ29tbWVudDY5ODYyNjc2OA==,9599,simonw,2020-09-24T22:46:56Z,2020-09-24T22:46:56Z,OWNER,"Yeah this works fine, added a new confirmatory test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684118950,extracts= doesn't configure foreign keys,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698578959,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698578959,MDEyOklzc3VlQ29tbWVudDY5ODU3ODk1OQ==,9599,simonw,2020-09-24T20:44:35Z,2020-09-24T20:50:19Z,OWNER,"I'm using a `click.File()` at the moment: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L496

I'll need to change that to be something that I can easily measure progress through. Also I should change its name - `json_file` is a bad name when it sometimes handles `csv` or `tsv` instead.

It looks like the argument provided by `click.File` doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698579389,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698579389,MDEyOklzc3VlQ29tbWVudDY5ODU3OTM4OQ==,9599,simonw,2020-09-24T20:45:29Z,2020-09-24T20:45:29Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L550-L560

Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698577508,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698577508,MDEyOklzc3VlQ29tbWVudDY5ODU3NzUwOA==,9599,simonw,2020-09-24T20:41:18Z,2020-09-24T20:41:18Z,OWNER,"I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.

Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/119#issuecomment-698575545,https://api.github.com/repos/simonw/sqlite-utils/issues/119,698575545,MDEyOklzc3VlQ29tbWVudDY5ODU3NTU0NQ==,9599,simonw,2020-09-24T20:36:59Z,2020-09-24T20:36:59Z,OWNER,This was implemented in #161.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770,Ability to remove a foreign key,
https://github.com/simonw/sqlite-utils/issues/176#issuecomment-698572493,https://api.github.com/repos/simonw/sqlite-utils/issues/176,698572493,MDEyOklzc3VlQ29tbWVudDY5ODU3MjQ5Mw==,9599,simonw,2020-09-24T20:30:18Z,2020-09-24T20:30:18Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/cli.html#transforming-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708293114,sqlite-utils transform column order option,
https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698572264,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698572264,MDEyOklzc3VlQ29tbWVudDY5ODU3MjI2NA==,9599,simonw,2020-09-24T20:29:48Z,2020-09-24T20:29:48Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=),
https://github.com/simonw/datasette/issues/976#issuecomment-698488971,https://api.github.com/repos/simonw/datasette/issues/976,698488971,MDEyOklzc3VlQ29tbWVudDY5ODQ4ODk3MQ==,9599,simonw,2020-09-24T17:42:09Z,2020-09-24T17:42:35Z,OWNER,This is complex enough new logic that it will need test coverage - specifically covering tables or databases with strange names.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708289783,Idea: -o could open to a more convenient location,
https://github.com/simonw/sqlite-utils/issues/177#issuecomment-698444567,https://api.github.com/repos/simonw/sqlite-utils/issues/177,698444567,MDEyOklzc3VlQ29tbWVudDY5ODQ0NDU2Nw==,9599,simonw,2020-09-24T16:14:47Z,2020-09-24T16:14:47Z,OWNER,"This is a backwards incompatible change, so technically I should bump the major version to 3. I'm not going to do that, because the feature is brand new and the chance that anyone has written code or shell scripts that use it is vanishingly small.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708301810,Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key,
https://github.com/simonw/sqlite-utils/issues/176#issuecomment-698438043,https://api.github.com/repos/simonw/sqlite-utils/issues/176,698438043,MDEyOklzc3VlQ29tbWVudDY5ODQzODA0Mw==,9599,simonw,2020-09-24T16:02:55Z,2020-09-24T16:02:55Z,OWNER,I think I'll call this option `--column-order` with a shortcut of `-o`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708293114,sqlite-utils transform column order option,
https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698434811,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698434811,MDEyOklzc3VlQ29tbWVudDY5ODQzNDgxMQ==,9599,simonw,2020-09-24T15:57:17Z,2020-09-24T15:57:17Z,OWNER,Landed that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=),
https://github.com/simonw/datasette/issues/970#issuecomment-698434236,https://api.github.com/repos/simonw/datasette/issues/970,698434236,MDEyOklzc3VlQ29tbWVudDY5ODQzNDIzNg==,9599,simonw,2020-09-24T15:56:18Z,2020-09-24T15:56:50Z,OWNER,"Idea: if a database only has a single table, this could open straight to `/db/table`. If it has multiple tables but a single database it could open straight to `/db`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url",
https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698412692,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698412692,MDEyOklzc3VlQ29tbWVudDY5ODQxMjY5Mg==,9599,simonw,2020-09-24T15:19:28Z,2020-09-24T15:19:28Z,OWNER,Need to land #174 first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=),
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698400790,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698400790,MDEyOklzc3VlQ29tbWVudDY5ODQwMDc5MA==,9599,simonw,2020-09-24T14:59:50Z,2020-09-24T14:59:50Z,OWNER,For reusing the lookup table: I'm going to raise an error if a lookup table exists but without the correct columns. The caller can then add those columns and try again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698184166,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698184166,MDEyOklzc3VlQ29tbWVudDY5ODE4NDE2Ng==,9599,simonw,2020-09-24T08:01:07Z,2020-09-24T08:01:07Z,OWNER,I may revert the now unnecessary undocumented tweaks to the `.update()` method made in 66d506587eba9f0715267d6560b97c1fa44cc781 as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182656,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698182656,MDEyOklzc3VlQ29tbWVudDY5ODE4MjY1Ng==,9599,simonw,2020-09-24T07:58:08Z,2020-09-24T07:58:08Z,OWNER,"The way the lookup table works here differs from the previous implementation. In the previous implementation the usage of `.lookup()` meant that an existing table would be modified to fit the new purpose. That no longer happens in this version. Need to make a design decision about how this should work.

It should definitely be possible to use an existing lookup table - imagine a database where several tables have a ""Departments"" column and we want to extract all of those values out to a single shared ""Departments"" table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182037,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698182037,MDEyOklzc3VlQ29tbWVudDY5ODE4MjAzNw==,9599,simonw,2020-09-24T07:56:50Z,2020-09-24T07:56:50Z,OWNER,I could also be a bit smarter about transaction handling. I think it may be possible to run this entire operation in a single transaction now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698181478,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698181478,MDEyOklzc3VlQ29tbWVudDY5ODE4MTQ3OA==,9599,simonw,2020-09-24T07:55:45Z,2020-09-24T07:55:45Z,OWNER,`import functools` is no longer needed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180705,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698180705,MDEyOklzc3VlQ29tbWVudDY5ODE4MDcwNQ==,9599,simonw,2020-09-24T07:54:10Z,2020-09-24T07:54:10Z,OWNER,"After running through the steps in https://simonwillison.net/2020/Sep/23/sqlite-utils-extract/ I get a table that looks like this:

<img width=""1699"" alt=""salaries__salaries__683_558_rows"" src=""https://user-images.githubusercontent.com/9599/94116875-666b8900-fe00-11ea-9e97-2b9ccbfeae29.png"">

The foreign key columns are all at the end of the table. It would be nicer if they were arranged in the same order as the columns they replaced.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180113,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698180113,MDEyOklzc3VlQ29tbWVudDY5ODE4MDExMw==,9599,simonw,2020-09-24T07:53:03Z,2020-09-24T07:53:03Z,OWNER,This could do with a little bit more testing - I'm worried there may be column or table name edge cases that are not covered yet. I also need to remove the progress bar code since that no longer makes sense for this implementation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation",
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-698178101,https://api.github.com/repos/simonw/sqlite-utils/issues/172,698178101,MDEyOklzc3VlQ29tbWVudDY5ODE3ODEwMQ==,9599,simonw,2020-09-24T07:48:57Z,2020-09-24T07:49:20Z,OWNER,"> I wonder if I could make this faster by separating it out into a few steps:
> 
>     * Create the new lookup table with all of the distinct rows
> 
>     * Add the blank foreign key column
> 
>     * run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)`
> 
>     * Drop the value columns
My prototype of this knocked the time down from 10 minutes to 4 seconds, so I think the change is worth it!
```
% date
sqlite-utils extract salaries.db salaries \
   'Department Code' 'Department' \
  --table 'departments' \
  --fk-column 'department_id' \
  --rename 'Department Code' code \
  --rename 'Department' name
date
sqlite-utils extract salaries.db salaries \
   'Union Code' 'Union' \
  --table 'unions' \
  --fk-column 'union_id' \
  --rename 'Union Code' code \
  --rename 'Union' name
date
sqlite-utils extract salaries.db salaries \
   'Job Family Code' 'Job Family' \
  --table 'job_families' \
  --fk-column 'job_family_id' \
  --rename 'Job Family Code' code \
  --rename 'Job Family' name
date
sqlite-utils extract salaries.db salaries \
   'Job Code' 'Job' \
  --table 'jobs' \
  --fk-column 'job_id' \
  --rename 'Job Code' code \
  --rename 'Job' name
date
Thu Sep 24 00:48:16 PDT 2020

Thu Sep 24 00:48:20 PDT 2020

Thu Sep 24 00:48:24 PDT 2020

Thu Sep 24 00:48:28 PDT 2020

Thu Sep 24 00:48:32 PDT 2020
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/datasette/issues/123#issuecomment-698168648,https://api.github.com/repos/simonw/datasette/issues/123,698168648,MDEyOklzc3VlQ29tbWVudDY5ODE2ODY0OA==,9599,simonw,2020-09-24T07:28:38Z,2020-09-24T07:28:38Z,OWNER,@obra there's a plugin for that! https://github.com/simonw/datasette-upload-csvs,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/simonw/datasette/issues/974#issuecomment-698110492,https://api.github.com/repos/simonw/datasette/issues/974,698110492,MDEyOklzc3VlQ29tbWVudDY5ODExMDQ5Mg==,9599,simonw,2020-09-24T04:50:56Z,2020-09-24T04:51:05Z,OWNER,"Come to think of it I've noticed that in the logs when it's running on my laptop, definitely worth fixing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707849175,static assets and favicon aren't cached by the browser,
https://github.com/simonw/datasette/issues/619#issuecomment-698024773,https://api.github.com/repos/simonw/datasette/issues/619,698024773,MDEyOklzc3VlQ29tbWVudDY5ODAyNDc3Mw==,9599,simonw,2020-09-23T23:31:46Z,2020-09-23T23:31:46Z,OWNER,"I'm going to have to untangle Datasette's error handling a bit for this - currently the expectation is that exceptions will be handled at a higher level, but I need to rethink that to make it cleaner for views like the ""execute custom SQL"" view to add their own error handling (and still be able to return the correct HTTP status codes, even with custom pages).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697998045,https://api.github.com/repos/simonw/datasette/issues/619,697998045,MDEyOklzc3VlQ29tbWVudDY5Nzk5ODA0NQ==,9599,simonw,2020-09-23T22:09:06Z,2020-09-23T22:09:06Z,OWNER,"I'll add this to the succesful JSON format:
```json
{
  ""ok"": true,
  ""error"": null
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697995885,https://api.github.com/repos/simonw/datasette/issues/619,697995885,MDEyOklzc3VlQ29tbWVudDY5Nzk5NTg4NQ==,9599,simonw,2020-09-23T22:02:44Z,2020-09-23T22:08:28Z,OWNER,"So the JSON (still served with a 500 code) will look something like this:
```json
{
  ""ok"": false,
  ""status"": 500,
  ""database"": ""fixtures"",
  ""query_name"": null,
  ""rows"": [],
  ""truncated"": false,
  ""error"": ""Error message goes here"",
  ""columns"": [],
  ""query"": {
    ""sql"": ""the query that broke goes here"",
    ""params"": {}
  },
  ""private"": false,
  ""allow_execute_sql"": true,
  ""query_ms"": 0.8716583251953125,
  ""source"": ""tests/fixtures.py"",
  ""source_url"": ""https://github.com/simonw/datasette/blob/master/tests/fixtures.py"",
  ""license"": ""Apache License 2.0"",
  ""license_url"": ""https://github.com/simonw/datasette/blob/master/LICENSE""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697995303,https://api.github.com/repos/simonw/datasette/issues/619,697995303,MDEyOklzc3VlQ29tbWVudDY5Nzk5NTMwMw==,9599,simonw,2020-09-23T22:01:08Z,2020-09-23T22:01:08Z,OWNER,"This is a little tricky to solve, because of the location of the form and the need to return JSON as well as HTML. It would be weird if a JSON request came in and got back the standard output from https://latest.datasette.io/fixtures.json when they were expecting to get back JSON in the shape of https://latest.datasette.io/fixtures.json?sql=select%20*%20from%20sqlite_master

I'm going to return the HTML view that you would get for 0 results for a query - https://latest.datasette.io/fixtures?sql=select%201%20limit%200 - but with an error message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-697980061,https://api.github.com/repos/simonw/datasette/issues/619,697980061,MDEyOklzc3VlQ29tbWVudDY5Nzk4MDA2MQ==,9599,simonw,2020-09-23T21:22:42Z,2020-09-23T21:22:42Z,OWNER,Yeah that sucks. Bumping this up the priority list.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697869886,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697869886,MDEyOklzc3VlQ29tbWVudDY5Nzg2OTg4Ng==,9599,simonw,2020-09-23T18:45:30Z,2020-09-23T18:45:30Z,OWNER,"There's something to be said for making this operation pausable and resumable, especially if I'm going to make it available in a Datasette plugin at some point.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697866885,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697866885,MDEyOklzc3VlQ29tbWVudDY5Nzg2Njg4NQ==,9599,simonw,2020-09-23T18:43:37Z,2020-09-23T18:43:37Z,OWNER,Also what would happen if the table had new rows added to it while that command was running?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697863116,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697863116,MDEyOklzc3VlQ29tbWVudDY5Nzg2MzExNg==,9599,simonw,2020-09-23T18:41:06Z,2020-09-23T18:41:06Z,OWNER,Problem with this approach is it's not compatible with progress bars - but if it's a multiple of times faster it's worth it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697859772,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697859772,MDEyOklzc3VlQ29tbWVudDY5Nzg1OTc3Mg==,9599,simonw,2020-09-23T18:38:43Z,2020-09-23T18:38:52Z,OWNER,"I wonder if I could make this faster by separating it out into a few steps:
- Create the new lookup table with all of the distinct rows
- Add the blank foreign key column
- run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)`
- Drop the value columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697835956,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697835956,MDEyOklzc3VlQ29tbWVudDY5NzgzNTk1Ng==,9599,simonw,2020-09-23T18:22:49Z,2020-09-23T18:22:49Z,OWNER,"I ran `sudo py-spy top -p 123` against the process while it was running and the most time is definitely spent in `.update()`:
```
Total Samples 1000
GIL: 0.00%, Active: 90.00%, Threads: 1

  %Own   %Total  OwnTime  TotalTime  Function (filename:line)                                                                                                                                  
 38.00%  38.00%    3.85s     3.85s   update (sqlite_utils/db.py:1283)
 27.00%  27.00%    2.12s     2.12s   execute (sqlite_utils/db.py:161)
 10.00%  10.00%   0.890s    0.890s   execute (sqlite_utils/db.py:163)
 10.00%  17.00%   0.870s     1.54s   columns (sqlite_utils/db.py:553)
  0.00%   0.00%   0.110s    0.210s   <listcomp> (sqlite_utils/db.py:554)
  0.00%   3.00%   0.100s    0.320s   table_names (sqlite_utils/db.py:191)
  0.00%   0.00%   0.100s    0.100s   __new__ (<string>:1)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-697577646,https://api.github.com/repos/simonw/sqlite-utils/issues/173,697577646,MDEyOklzc3VlQ29tbWVudDY5NzU3NzY0Ng==,9599,simonw,2020-09-23T15:48:51Z,2020-09-23T15:48:51Z,OWNER,"This can only work when it's reading from a file, not when it's reading from standard input.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/datasette/issues/111#issuecomment-697545290,https://api.github.com/repos/simonw/datasette/issues/111,697545290,MDEyOklzc3VlQ29tbWVudDY5NzU0NTI5MA==,9599,simonw,2020-09-23T15:29:11Z,2020-09-23T15:29:11Z,OWNER,This is still a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697473247,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697473247,MDEyOklzc3VlQ29tbWVudDY5NzQ3MzI0Nw==,9599,simonw,2020-09-23T14:45:13Z,2020-09-23T14:45:13Z,OWNER,"`lookup_table.lookup(lookups)` is doing a SQL lookup. This could be cached in-memory, maybe with a LRU cache, to avoid looking up the primary key for records that we have recently used.

The `.update()` method it is calling first does a `get()` and then does a SQL `UPDATE ... WHERE`:

https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L1244-L1264

Batching those updates may have an effect. Or finding a way to skip the `.get()` since we already know we have a valid record.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697467833,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697467833,MDEyOklzc3VlQ29tbWVudDY5NzQ2NzgzMw==,9599,simonw,2020-09-23T14:42:03Z,2020-09-23T14:42:03Z,OWNER,Here's the loop that's taking the time: https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L892-L897,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697466497,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697466497,MDEyOklzc3VlQ29tbWVudDY5NzQ2NjQ5Nw==,9599,simonw,2020-09-23T14:41:17Z,2020-09-23T14:41:17Z,OWNER,"Steps to produce that database:
```
curl -o salaries.csv 'https://data.sfgov.org/api/views/88g8-5mnd/rows.csv?accessType=DOWNLOAD'
sqlite-utils insert salaries.db salaries salaries.csv --csv
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations,
https://github.com/simonw/sqlite-utils/issues/170#issuecomment-697047591,https://api.github.com/repos/simonw/sqlite-utils/issues/170,697047591,MDEyOklzc3VlQ29tbWVudDY5NzA0NzU5MQ==,9599,simonw,2020-09-23T00:14:52Z,2020-09-23T00:14:52Z,OWNER,"
@simonw
@db.register_function decorator, closes #162
4824775
@simonw
table.transform() method - closes #114
987dd12
@simonw
Keyword only arguments for transform()
f8e10df

Also renamed columns= to types=

Closes #165

Commits on Sep 22, 2020
@simonw
Implemented sqlite-utils transform command, closes #164
752d261
@simonw
Applied Black
f29f682
@simonw
table.extract() method, refs #42
f855379
@simonw
Docstring for sqlite-utils transform
c755f28
@simonw
Added table.extract(rename=) option, refs #42
c3210f2
@simonw
Applied Black
317071a
@simonw
New .rows_where(select=) argument
7178231
@simonw
table.extract() now works with rowid tables, refs #42
2db6c5b
@simonw
sqlite-utils extract, closes #42
55cf928
@simonw
Progress bar for ""sqlite-utils extract"", closes #169
5c4d58d
@simonw
Fixed PRAGMA foreign_keys handling for .transform, closes #167 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706768798,Release notes for 2.20,
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697037974,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697037974,MDEyOklzc3VlQ29tbWVudDY5NzAzNzk3NA==,9599,simonw,2020-09-22T23:39:31Z,2020-09-22T23:39:31Z,OWNER,Documentation for `sqlite-utils extract`: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697031174,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697031174,MDEyOklzc3VlQ29tbWVudDY5NzAzMTE3NA==,9599,simonw,2020-09-22T23:16:00Z,2020-09-22T23:16:00Z,OWNER,"Trying this demo again:
```
wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv
sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name
```
It worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697025403,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697025403,MDEyOklzc3VlQ29tbWVudDY5NzAyNTQwMw==,9599,simonw,2020-09-22T22:57:53Z,2020-09-22T22:57:53Z,OWNER,The documentation for the `.extract()` method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697019944,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697019944,MDEyOklzc3VlQ29tbWVudDY5NzAxOTk0NA==,9599,simonw,2020-09-22T22:40:00Z,2020-09-22T22:40:00Z,OWNER,"I tried out the prototype of the CLI on the Global Power Plants data:
```
wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv
sqlite-utils extract global.db power_plants country country_long
```
This threw an error because `rowid` columns are not yet supported. I fixed that like so:
```
sqlite-utils transform global.db power_plants --rename rowid id
sqlite-utils extract global.db power_plants country country_long
```
That worked! But it didn't play great with Datasette, because the resulting extracted table had columns `country` and `country_long` and neither of those are called `name` or `value` or `title`.

Based on this I need to add `rowid` table support AND I need to implement the proposed `rename=` argument for renaming columns on their way into the new table.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697013681,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697013681,MDEyOklzc3VlQ29tbWVudDY5NzAxMzY4MQ==,9599,simonw,2020-09-22T22:22:49Z,2020-09-22T22:22:49Z,OWNER,"The command-line version of this needs to accept a table and one or more columns, then a `--table` and `--fk-column` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697012111,MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ==,9599,simonw,2020-09-22T22:18:13Z,2020-09-22T22:18:13Z,OWNER,"Here's how I'm generating the examples for the documentation:
```
In [2]: import sqlite_utils

In [3]: db = sqlite_utils.Database(memory=True)

In [4]: db[""Trees""].insert({""id"": 1, ""TreeAddress"": ""52 Vine St"", ""CommonName"":
   ...: ""Palm"", ""LatinName"": ""foo""}, pk=""id"")
Out[4]: <Table Trees (id, TreeAddress, CommonName, LatinName)>

In [5]: db[""Trees""].extract([""CommonName"", ""LatinName""], table=""Species"", fk_col
   ...: umn=""species_id"")

In [6]: print(db[""Trees""].schema)
CREATE TABLE ""Trees"" (
   [id] INTEGER PRIMARY KEY,
   [TreeAddress] TEXT,
   [species_id] INTEGER,
   FOREIGN KEY(species_id) REFERENCES Species(id)
)

In [7]: print(db[""Species""].schema)
CREATE TABLE [Species] (
   [id] INTEGER PRIMARY KEY,
   [CommonName] TEXT,
   [LatinName] TEXT
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987925,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987925,MDEyOklzc3VlQ29tbWVudDY5Njk4NzkyNQ==,9599,simonw,2020-09-22T21:19:04Z,2020-09-22T21:19:04Z,OWNER,Need to make sure this works correctly for `rowid` tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987257,MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw==,9599,simonw,2020-09-22T21:17:34Z,2020-09-22T21:17:34Z,OWNER,"What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980709,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980709,MDEyOklzc3VlQ29tbWVudDY5Njk4MDcwOQ==,9599,simonw,2020-09-22T21:05:07Z,2020-09-22T21:05:07Z,OWNER,"So `.extract()` probably takes a `batch_size=` argument too, which defaults to maybe 1000.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980503,MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw==,9599,simonw,2020-09-22T21:04:45Z,2020-09-22T21:04:45Z,OWNER,"`table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979626,MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg==,9599,simonw,2020-09-22T21:03:11Z,2020-09-22T21:03:11Z,OWNER,"And if you want to rename some of the columns in the new table:
```python
db[""trees""].extract([""common_name"", ""latin_name""], table=""species"", rename={""common_name"": ""name""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979168,MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA==,9599,simonw,2020-09-22T21:02:24Z,2020-09-22T21:02:24Z,OWNER,"In Python it looks like this:
```python
# Simple case - species column species_id pointing to species table
db[""trees""].extract(""species"")

# Setting a custom table
db[""trees""].extract(""species"", table=""Species"")

# Custom foreign key column on trees
db[""trees""].extract(""species"", fk_column=""species"")

# Extracting multiple columns
db[""trees""].extract([""common_name"", ""latin_name""])
# (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id)

# Or with explicit table (fk_column here defaults to species_id because of the table name)
db[""trees""].extract([""common_name"", ""latin_name""], table=""species"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696976678,MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA==,9599,simonw,2020-09-22T20:57:57Z,2020-09-22T20:57:57Z,OWNER,"I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID.

That integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893774,MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA==,9599,simonw,2020-09-22T18:15:33Z,2020-09-22T18:15:33Z,OWNER,I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893244,MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA==,9599,simonw,2020-09-22T18:14:33Z,2020-09-22T18:14:45Z,OWNER,"Thinking more about this one:
```
$ sqlite-utils extract my.db \
    dea_sales company_name company_address \
    --table companies
```
The goal here is to pull the company name and address pair out into a separate table.

Some questions:
- should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this.
- what should the foreign key column that gets added to the `companies` table be called?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513262013,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513262013,MDEyOklzc3VlQ29tbWVudDUxMzI2MjAxMw==,9599,simonw,2019-07-19T14:58:23Z,2020-09-22T18:12:11Z,OWNER,"CLI design idea:

    $ sqlite-utils extract my.db \
        dea_sales company_name

Here we just specify the original table and column - the new extracted table will automatically be called ""company_name"" and will have ""id"" and ""value"" columns, by default.

To set a custom extract table:

    $ sqlite-utils extract my.db \
        dea_sales company_name \
        --table companies

And for extracting multiple columns and renaming them on the created table, maybe something like this:

    $ sqlite-utils extract my.db \
        dea_sales company_name company_address \
        --table companies \
        --column company_name name \
        --column company_address address
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/datasette/issues/973#issuecomment-696800410,https://api.github.com/repos/simonw/datasette/issues/973,696800410,MDEyOklzc3VlQ29tbWVudDY5NjgwMDQxMA==,9599,simonw,2020-09-22T15:35:28Z,2020-09-22T15:35:28Z,OWNER,"Confirmed in local dev:
```
% datasette fixtures.db --inspect-file inspect.json
Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette"", line 11, in <module>
    load_entry_point('datasette', 'console_scripts', 'datasette')()
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 406, in serve
    inspect_data = json.load(open(inspect_file))
TypeError: 'bool' object is not callable
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706486323,'bool' object is not callable error,
https://github.com/simonw/datasette/issues/973#issuecomment-696798114,https://api.github.com/repos/simonw/datasette/issues/973,696798114,MDEyOklzc3VlQ29tbWVudDY5Njc5ODExNA==,9599,simonw,2020-09-22T15:31:25Z,2020-09-22T15:31:25Z,OWNER,D'oh because I have a new variable called `open`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706486323,'bool' object is not callable error,
https://github.com/simonw/datasette/issues/969#issuecomment-696788109,https://api.github.com/repos/simonw/datasette/issues/969,696788109,MDEyOklzc3VlQ29tbWVudDY5Njc4ODEwOQ==,9599,simonw,2020-09-22T15:15:14Z,2020-09-22T15:15:14Z,OWNER,"I don't think a standard ""pass these extra arguments to the publish tool"" mechanism will work because there's no guarantee that a publisher uses a CLI tool - or if it does, it might make several calls to different CLI tools. The Cloud Run one runs a couple of commands, as illustrated by this test:

https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/tests/test_publish_cloudrun.py#L63-L73

Adding a `--tar` option for `datasette publish heroku` is a good fix for this though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705057955,"Add --tar option to ""datasette publish heroku""",
https://github.com/simonw/datasette/issues/943#issuecomment-696778735,https://api.github.com/repos/simonw/datasette/issues/943,696778735,MDEyOklzc3VlQ29tbWVudDY5Njc3ODczNQ==,9599,simonw,2020-09-22T15:00:13Z,2020-09-22T15:00:39Z,OWNER,"Am I going to rewrite ALL of my tests to use this instead? It would clean up a lot of test code, at the cost of quite a bit of work.

It would make for much neater plugin tests too, and neater testing documentation: https://docs.datasette.io/en/stable/testing_plugins.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696777886,https://api.github.com/repos/simonw/datasette/issues/943,696777886,MDEyOklzc3VlQ29tbWVudDY5Njc3Nzg4Ng==,9599,simonw,2020-09-22T14:58:54Z,2020-09-22T14:58:54Z,OWNER,"```python
class DatasetteClient:
    def __init__(self, ds):
        self._client = httpx.AsyncClient(app=ds.app())

    def _fix(self, path):
        if path.startswith(""/""):
            path = ""http://localhost{}"".format(path)
        return path

    async def get(self, path, **kwargs):
        return await self._client.get(self._fix(path), **kwargs)

    async def options(self, path, **kwargs):
        return await self._client.options(self._fix(path), **kwargs)

    async def head(self, path, **kwargs):
        return await self._client.head(self._fix(path), **kwargs)

    async def post(self, path, **kwargs):
        return await self._client.post(self._fix(path), **kwargs)

    async def put(self, path, **kwargs):
        return await self._client.put(self._fix(path), **kwargs)

    async def patch(self, path, **kwargs):
        return await self._client.patch(self._fix(path), **kwargs)

    async def delete(self, path, **kwargs):
        return await self._client.delete(self._fix(path), **kwargs)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696776828,https://api.github.com/repos/simonw/datasette/issues/943,696776828,MDEyOklzc3VlQ29tbWVudDY5Njc3NjgyOA==,9599,simonw,2020-09-22T14:57:13Z,2020-09-22T14:57:13Z,OWNER,"I may as well implement all of the HTTP methods supported by the `httpx` client:

- get
- options
- head
- post
- put
- patch
- delete","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696775516,https://api.github.com/repos/simonw/datasette/issues/943,696775516,MDEyOklzc3VlQ29tbWVudDY5Njc3NTUxNg==,9599,simonw,2020-09-22T14:55:10Z,2020-09-22T14:55:10Z,OWNER,"Even smaller `DatasetteClient` implementation:
```python
class DatasetteClient:
    def __init__(self, ds):
        self._client = httpx.AsyncClient(app=ds.app())

    def _fix(self, path):
        if path.startswith(""/""):
            path = ""http://localhost{}"".format(path)
        return path

    async def get(self, path, **kwargs):
        return await self._client.get(self._fix(path), **kwargs)

    async def post(self, path, **kwargs):
        return await self._client.post(self._fix(path), **kwargs)

    async def options(self, path, **kwargs):
        return await self._client.options(self._fix(path), **kwargs)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696774711,https://api.github.com/repos/simonw/datasette/issues/943,696774711,MDEyOklzc3VlQ29tbWVudDY5Njc3NDcxMQ==,9599,simonw,2020-09-22T14:53:56Z,2020-09-22T14:53:56Z,OWNER,"How important is it to use `httpx.AsyncClient` with a context manager?

https://www.python-httpx.org/async/#opening-and-closing-clients says:

> Alternatively, use `await client.aclose()` if you want to close a client explicitly:
> 
> ```
> client = httpx.AsyncClient()
> ...
> await client.aclose()
> ```
The `.aclose()` method has a comment saying ""Close transport and proxies"" - I'm not using proxies, so the relevant implementation seems to be a call to `await self._transport.aclose()` in https://github.com/encode/httpx/blob/f932af9172d15a803ad40061a4c2c0cd891645cf/httpx/_client.py#L1741-L1751

The transport I am using is a class called `ASGITransport` in https://github.com/encode/httpx/blob/master/httpx/_transports/asgi.py

The `aclose()` method on that class does nothing. So it looks like I can instantiate a client without bothering with the `async with httpx.AsyncClient` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696769853,https://api.github.com/repos/simonw/datasette/issues/943,696769853,MDEyOklzc3VlQ29tbWVudDY5Njc2OTg1Mw==,9599,simonw,2020-09-22T14:46:21Z,2020-09-22T14:46:21Z,OWNER,This adds `httpx` as a dependency - I think I'm OK with that. I use it for testing in all of my plugins anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-696769501,https://api.github.com/repos/simonw/datasette/issues/943,696769501,MDEyOklzc3VlQ29tbWVudDY5Njc2OTUwMQ==,9599,simonw,2020-09-22T14:45:49Z,2020-09-22T14:45:49Z,OWNER,"I put together a minimal prototype of this and it feels pretty good:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 20aae7d..fb3bdad 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -4,6 +4,7 @@ import collections
 import datetime
 import glob
 import hashlib
+import httpx
 import inspect
 import itertools
 from itsdangerous import BadSignature
@@ -312,6 +313,7 @@ class Datasette:
         self._register_renderers()
         self._permission_checks = collections.deque(maxlen=200)
         self._root_token = secrets.token_hex(32)
+        self.client = DatasetteClient(self)
 
     async def invoke_startup(self):
         for hook in pm.hook.startup(datasette=self):
@@ -1209,3 +1211,25 @@ def route_pattern_from_filepath(filepath):
 
 class NotFoundExplicit(NotFound):
     pass
+
+
+class DatasetteClient:
+    def __init__(self, ds):
+        self.app = ds.app()
+
+    def _fix(self, path):
+        if path.startswith(""/""):
+            path = ""http://localhost{}"".format(path)
+        return path
+
+    async def get(self, path, **kwargs):
+        async with httpx.AsyncClient(app=self.app) as client:
+            return await client.get(self._fix(path), **kwargs)
+
+    async def post(self, path, **kwargs):
+        async with httpx.AsyncClient(app=self.app) as client:
+            return await client.post(self._fix(path), **kwargs)
+
+    async def options(self, path, **kwargs):
+        async with httpx.AsyncClient(app=self.app) as client:
+            return await client.options(self._fix(path), **kwargs)
```
Used like this in `ipython`:
```
In [1]: from datasette.app import Datasette

In [2]: ds = Datasette([""fixtures.db""])

In [3]: (await ds.client.get(""/-/config.json"")).json()
Out[3]: 
{'default_page_size': 100,
 'max_returned_rows': 1000,
 'num_sql_threads': 3,
 'sql_time_limit_ms': 1000,
 'default_facet_size': 30,
 'facet_time_limit_ms': 200,
 'facet_suggest_time_limit_ms': 50,
 'hash_urls': False,
 'allow_facet': True,
 'allow_download': True,
 'suggest_facets': True,
 'default_cache_ttl': 5,
 'default_cache_ttl_hashed': 31536000,
 'cache_size_kb': 0,
 'allow_csv_stream': True,
 'max_csv_mb': 100,
 'truncate_cells_html': 2048,
 'force_https_urls': False,
 'template_debug': False,
 'base_url': '/'}

In [4]: (await ds.client.get(""/fixtures/facetable.json?_shape=array"")).json()
Out[4]: 
[{'pk': 1,
  'created': '2019-01-14 08:00:00',
  'planet_int': 1,
  'on_earth': 1,
  'state': 'CA',
  'city_id': 1,
  'neighborhood': 'Mission',
  'tags': '[""tag1"", ""tag2""]',
  'complex_array': '[{""foo"": ""bar""}]',
  'distinct_some_null': 'one'},
 {'pk': 2,
  'created': '2019-01-14 08:00:00',
  'planet_int': 1,
  'on_earth': 1,
  'state': 'CA',
  'city_id': 1,
  'neighborhood': 'Dogpatch',
  'tags': '[""tag1"", ""tag3""]',
  'complex_array': '[]',
  'distinct_some_null': 'two'},
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693009048,https://api.github.com/repos/simonw/datasette/issues/943,693009048,MDEyOklzc3VlQ29tbWVudDY5MzAwOTA0OA==,9599,simonw,2020-09-15T22:17:30Z,2020-09-22T14:37:00Z,OWNER,"Maybe instead of implementing `datasette.get()` and `datasette.post()` and `datasette.request()` and `datasette.stream()` I could instead have a nested object called `datasette.client` which is a preconfigured `AsyncClient` instance.

```python
response = await datasette.client.get(""/"")
```
Or perhaps this should be a method in case I ever need to be able to `await` it:
```python
response = await (await datasette.client()).get(""/"")
```
This is a bit cosmetically ugly though, I'd rather avoid that if possible.

Maybe I could get this working by returning an object from `.client()` which provides a `await obj.get()` method:
```python
response = await datasette.client().get(""/"")
```
I don't think there's any benefit to that over `await datasette.client.get()` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/sqlite-utils/issues/168#issuecomment-696573944,https://api.github.com/repos/simonw/sqlite-utils/issues/168,696573944,MDEyOklzc3VlQ29tbWVudDY5NjU3Mzk0NA==,9599,simonw,2020-09-22T08:11:30Z,2020-09-22T08:11:30Z,OWNER,Huh... maybe I don't need to do anything here? It looks like it's been kept up to date: https://github.com/Homebrew/homebrew-core/commits/master/Formula/sqlite-utils.rb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706167456,Automate (as much as possible) updates published to Homebrew,
https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696567988,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696567988,MDEyOklzc3VlQ29tbWVudDY5NjU2Nzk4OA==,9599,simonw,2020-09-22T07:57:50Z,2020-09-22T07:57:50Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#transforming-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command,
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696567460,MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA==,9599,simonw,2020-09-22T07:56:42Z,2020-09-22T07:56:42Z,OWNER,`.transform()` has landed now which should make this a lot easier to solve.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/26#issuecomment-696566750,https://api.github.com/repos/simonw/sqlite-utils/issues/26,696566750,MDEyOklzc3VlQ29tbWVudDY5NjU2Njc1MA==,9599,simonw,2020-09-22T07:55:00Z,2020-09-22T07:55:00Z,OWNER,"Problem: `extract` means something else now, see #47 and the upcoming work in #42.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455486286,Mechanism for turning nested JSON into foreign keys / many-to-many,
https://github.com/simonw/sqlite-utils/issues/167#issuecomment-696565981,https://api.github.com/repos/simonw/sqlite-utils/issues/167,696565981,MDEyOklzc3VlQ29tbWVudDY5NjU2NTk4MQ==,9599,simonw,2020-09-22T07:53:13Z,2020-09-22T07:53:13Z,OWNER,"Confirmed this is a bug, https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes explicitly says you should do the `PRAGMA foreign_keys` bits before and after the transaction, not during.

Right now my code does this INSIDE the transaction: https://github.com/simonw/sqlite-utils/blob/f29f6821f2d08e91c5c6d65d885a1bbc0c743bdd/sqlite_utils/db.py#L790-L793

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706098005,Review the foreign key pragma stuff,
https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696520928,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696520928,MDEyOklzc3VlQ29tbWVudDY5NjUyMDkyOA==,9599,simonw,2020-09-22T05:50:17Z,2020-09-22T05:50:17Z,OWNER,"Idea for CLI options:
```
--type age integer
--drop colname
--rename oldname newname
--not-null col
--not-null-false col
--pk new_id
--pk-none
--default col value
--default-none column
--drop-foreign-key col other_table other_column
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command,
https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696500922,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696500922,MDEyOklzc3VlQ29tbWVudDY5NjUwMDkyMg==,9599,simonw,2020-09-22T04:22:40Z,2020-09-22T04:22:40Z,OWNER,Documentation for the `.transform()` method #114 (now landed) is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696500767,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696500767,MDEyOklzc3VlQ29tbWVudDY5NjUwMDc2Nw==,9599,simonw,2020-09-22T04:21:45Z,2020-09-22T04:21:45Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696494070,MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA==,9599,simonw,2020-09-22T03:48:58Z,2020-09-22T03:48:58Z,OWNER,"One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be:

>  If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. 

And the last steps should be:

> If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints.
>
> Commit the transaction started in step 2.
>
> If foreign keys constraints were originally enabled, reenable them now. 

I need to implement that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696490851,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696490851,MDEyOklzc3VlQ29tbWVudDY5NjQ5MDg1MQ==,9599,simonw,2020-09-22T03:33:54Z,2020-09-22T03:33:54Z,OWNER,It would be neat if `.transform(pk=None)` converted a primary key table to a rowid table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696488201,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696488201,MDEyOklzc3VlQ29tbWVudDY5NjQ4ODIwMQ==,9599,simonw,2020-09-22T03:21:16Z,2020-09-22T03:21:16Z,OWNER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696485791,MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ==,9599,simonw,2020-09-22T03:10:15Z,2020-09-22T03:10:15Z,OWNER,"Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key?

Since I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this:

```python
table.transform(drop_foreign_keys=[(""author_id"", ""author"", ""id"")])
```
It's a bit crufty but it's such a rare use-case that I think this will be good enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696480925,MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ==,9599,simonw,2020-09-22T02:45:47Z,2020-09-22T02:45:47Z,OWNER,"I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this:

    INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs;

So passing `conversions={""name"": ""upper(?)""})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696473559,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696473559,MDEyOklzc3VlQ29tbWVudDY5NjQ3MzU1OQ==,9599,simonw,2020-09-22T02:10:37Z,2020-09-22T02:10:37Z,OWNER,"Maybe something like this:

    sqlite-utils transform mydb.db mytable -c age integer --rename age dog_age
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command,
https://github.com/simonw/sqlite-utils/issues/163#issuecomment-696465788,https://api.github.com/repos/simonw/sqlite-utils/issues/163,696465788,MDEyOklzc3VlQ29tbWVudDY5NjQ2NTc4OA==,9599,simonw,2020-09-22T01:33:04Z,2020-09-22T01:33:04Z,OWNER,This would apply to `.transform()` in #114 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696454485,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696454485,MDEyOklzc3VlQ29tbWVudDY5NjQ1NDQ4NQ==,9599,simonw,2020-09-22T00:42:35Z,2020-09-22T00:42:35Z,OWNER,The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696454084,https://api.github.com/repos/simonw/sqlite-utils/issues/162,696454084,MDEyOklzc3VlQ29tbWVudDY5NjQ1NDA4NA==,9599,simonw,2020-09-22T00:40:44Z,2020-09-22T00:40:44Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705995722,A decorator for registering custom SQL functions,
https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696449345,https://api.github.com/repos/simonw/sqlite-utils/issues/162,696449345,MDEyOklzc3VlQ29tbWVudDY5NjQ0OTM0NQ==,9599,simonw,2020-09-22T00:22:46Z,2020-09-22T00:22:46Z,OWNER,Inspired by the idea of adding `conversions=` to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705995722,A decorator for registering custom SQL functions,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696446658,MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA==,9599,simonw,2020-09-22T00:13:55Z,2020-09-22T00:14:21Z,OWNER,"Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.:

```python
table.transform({""age"": int}, conversions={""name"": ""upper(?)""})
```

https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696445766,MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng==,9599,simonw,2020-09-22T00:10:50Z,2020-09-22T00:11:12Z,OWNER,"A less horrible interface might be the following:
```python
# Ensure the 'age' column is not null:
table.transform(not_null={""age""})
# The 'age' column is not null but I don't want it to be:
table.transform(not_null={""age"": False})
```
So if the argument is a set it means ""make sure these are all not null"" - if the argument is a dictionary it means ""set these to be null or not null depending on if their dictionary value is true or false"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444842,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg==,9599,simonw,2020-09-22T00:07:43Z,2020-09-22T00:09:05Z,OWNER,"Syntax challenge: I could use `.transform(defaults={""age"": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it?

I could do this: `.transform(not_not_null={""age""})` - it's a bit gross but it's also kind of funny. I actually like it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444353,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw==,9599,simonw,2020-09-22T00:06:12Z,2020-09-22T00:06:12Z,OWNER,I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443845,MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ==,9599,simonw,2020-09-22T00:04:31Z,2020-09-22T00:04:44Z,OWNER,"Good news: the `.columns` introspection does tell me those things:
```
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> db.create_table(""foo"", {""id"": int, ""name"": str, ""age"": int}, defaults={""age"": 1}, not_null={""name"", ""age""})
<Table foo (id, name, age)>
>>> db[""foo""]
<Table foo (id, name, age)>
>>> print(db[""foo""].schema)
CREATE TABLE [foo] (
   [id] INTEGER,
   [name] TEXT NOT NULL,
   [age] INTEGER NOT NULL DEFAULT 1
)
>>> db[""foo""].columns
[Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0),
 Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0),
 Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443190,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA==,9599,simonw,2020-09-22T00:02:22Z,2020-09-22T00:02:22Z,OWNER,How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443042,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg==,9599,simonw,2020-09-22T00:01:50Z,2020-09-22T00:01:50Z,OWNER,"When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696442621,MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ==,9599,simonw,2020-09-22T00:00:23Z,2020-09-22T00:00:23Z,OWNER,I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696435194,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696435194,MDEyOklzc3VlQ29tbWVudDY5NjQzNTE5NA==,9599,simonw,2020-09-21T23:34:14Z,2020-09-21T23:35:00Z,OWNER,"I think the fiddliest part of the implementation here is code that takes the existing `columns_dict` of the table and the incoming `columns=` and `drop=` and `rename=` parameters and produces the columns dictionary for the new table, ready to be fed to `.create_table()`.

This logic probably also needs to return a structure that can be used to build the `INSERT INTO ... SELECT ... FROM` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434638,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434638,MDEyOklzc3VlQ29tbWVudDY5NjQzNDYzOA==,9599,simonw,2020-09-21T23:32:26Z,2020-09-21T23:32:26Z,OWNER,A test that confirms that this mechanism can turn a `rowid` into a non-rowid table would be good too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434237,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434237,MDEyOklzc3VlQ29tbWVudDY5NjQzNDIzNw==,9599,simonw,2020-09-21T23:31:07Z,2020-09-21T23:31:57Z,OWNER,"Does it make sense to support the `pk=` argument for changing the primary key?

If the user requests a primary key that doesn't make sense I think an integrity error will be raised when the SQL is being executed, which should hopefully cancel the transaction and raise an error. Need to check that this is what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434097,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434097,MDEyOklzc3VlQ29tbWVudDY5NjQzNDA5Nw==,9599,simonw,2020-09-21T23:30:40Z,2020-09-21T23:30:40Z,OWNER,"Since I have a `column_order=None` argument already, maybe I can ignore the order of the columns in that first argument and use that instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433778,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696433778,MDEyOklzc3VlQ29tbWVudDY5NjQzMzc3OA==,9599,simonw,2020-09-21T23:29:39Z,2020-09-21T23:29:39Z,OWNER,"The `columns=` argument is optional - so you can do just a rename operation like so:
```
table.transform(rename={""age"": ""dog_age""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433542,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696433542,MDEyOklzc3VlQ29tbWVudDY5NjQzMzU0Mg==,9599,simonw,2020-09-21T23:28:58Z,2020-09-21T23:28:58Z,OWNER,"If you want to both change the type of a column AND rename it in the same operation, how would you do that? I think like this:
```python
table.transform({""age"": int}, rename={""age"": ""dog_age""})
```
So any rename logic is applied at the end, after the type transformation or re-ordering logic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696432690,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696432690,MDEyOklzc3VlQ29tbWVudDY5NjQzMjY5MA==,9599,simonw,2020-09-21T23:26:32Z,2020-09-21T23:27:38Z,OWNER,"To expand on what that first argument - the `columns` argument - does. Say you have a table like this:
```
id integer
name text
age text
```
Any columns omitted from the `columns=` argument are left alone - they have to be explicitly dropped using `drop=` if you want to drop them.

Any new columns are added (at the end of the table):

```
table.tranform({""size"": float})
```

Any columns that have their type changed will have their type changed:

```
table.tranform({""age"": int})
```

Should I also re-order columns if the order doesn't match? I think so. Open question as to what happens to columns that aren't mentioned at all in the dictionary though - what order should they go in?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696431058,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696431058,MDEyOklzc3VlQ29tbWVudDY5NjQzMTA1OA==,9599,simonw,2020-09-21T23:21:37Z,2020-09-21T23:21:37Z,OWNER,I may need to do something special for `rowid` tables to ensure that the `rowid` values in the transformed table match those from the old table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696430843,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696430843,MDEyOklzc3VlQ29tbWVudDY5NjQzMDg0Mw==,9599,simonw,2020-09-21T23:21:00Z,2020-09-21T23:21:00Z,OWNER,"For FTS tables associated with the table that is being transformed, should I automatically drop the old FTS table and recreate it against the new one or will it just magically continue to work after the table is renamed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423138,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696423138,MDEyOklzc3VlQ29tbWVudDY5NjQyMzEzOA==,9599,simonw,2020-09-21T22:59:17Z,2020-09-21T23:01:06Z,OWNER,I'm going to sketch out a prototype of this new API design in that branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423066,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696423066,MDEyOklzc3VlQ29tbWVudDY5NjQyMzA2Ng==,9599,simonw,2020-09-21T22:59:01Z,2020-09-21T22:59:01Z,OWNER,"I'm rethinking the API design now. Maybe it could look like this:

To change the type of the `author_id` column from `text` to `int`:
```python
books.transform({""author_id"": int})
```
This would leave the existing columns alone, but would change the type of this column.

To rename `author_id` to `author_identifier`:
```python
books.transform(rename={""author_id"": ""author_identifier""})
```
To drop a column:
```python
books.transform(drop=[""author_id""])
```
Since the parameters all operate on columns they don't need to be called `drop_column` and `rename_column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696421240,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696421240,MDEyOklzc3VlQ29tbWVudDY5NjQyMTI0MA==,9599,simonw,2020-09-21T22:53:48Z,2020-09-21T22:53:48Z,OWNER,"I've decided to call this `table.transform()` - I was over-thinking whether people would remember that `.transform()` actually transforms the table, but that's what documentation is for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/datasette/issues/972#issuecomment-696308847,https://api.github.com/repos/simonw/datasette/issues/972,696308847,MDEyOklzc3VlQ29tbWVudDY5NjMwODg0Nw==,9599,simonw,2020-09-21T19:01:25Z,2020-09-21T19:01:25Z,OWNER,I did a bunch of initial work for this in #427.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705840673,Support faceting against arbitrary SQL queries,
https://github.com/simonw/datasette/issues/971#issuecomment-696307922,https://api.github.com/repos/simonw/datasette/issues/971,696307922,MDEyOklzc3VlQ29tbWVudDY5NjMwNzkyMg==,9599,simonw,2020-09-21T18:59:52Z,2020-09-21T19:00:02Z,OWNER,"Given `dbstat` isn't as widely available as I thought I'm going to let people who want to use `dbstat` run their own `select * from dbstat` queries rather than bake support directly into Datasette.

The experience of exploring `dbstat` will improve if I land support for running facets against arbitrary custom SQL queries, which is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646

https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696304108,https://api.github.com/repos/simonw/datasette/issues/971,696304108,MDEyOklzc3VlQ29tbWVudDY5NjMwNDEwOA==,9599,simonw,2020-09-21T18:52:50Z,2020-09-21T18:52:50Z,OWNER,Looks like the `pysqlite3-binary` package doesn't support `dbstat` either.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696302868,https://api.github.com/repos/simonw/datasette/issues/971,696302868,MDEyOklzc3VlQ29tbWVudDY5NjMwMjg2OA==,9599,simonw,2020-09-21T18:50:40Z,2020-09-21T18:50:40Z,OWNER,Easiest way to get this may be to run `create view dbstat_view as select * from dbstat` on databases that support it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696302020,https://api.github.com/repos/simonw/datasette/issues/971,696302020,MDEyOklzc3VlQ29tbWVudDY5NjMwMjAyMA==,9599,simonw,2020-09-21T18:49:09Z,2020-09-21T18:49:09Z,OWNER,... made harder to work on because I apparently don't have the `DBSTAT_VTAB` module on macOS.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696298614,https://api.github.com/repos/simonw/datasette/issues/971,696298614,MDEyOklzc3VlQ29tbWVudDY5NjI5ODYxNA==,9599,simonw,2020-09-21T18:43:07Z,2020-09-21T18:43:07Z,OWNER,"Or, do this:
```sql
SELECT 1 FROM dbstat limit 1;
```
And see if it returns a ""table does not exist"" error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696297930,https://api.github.com/repos/simonw/datasette/issues/971,696297930,MDEyOklzc3VlQ29tbWVudDY5NjI5NzkzMA==,9599,simonw,2020-09-21T18:41:47Z,2020-09-21T18:41:47Z,OWNER,"https://www.sqlite.org/dbstat.html
> The DBSTAT virtual table is an eponymous virtual table, meaning that is not necessary to run CREATE VIRTUAL TABLE to create an instance of the dbstat virtual table before using it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/971#issuecomment-696297601,https://api.github.com/repos/simonw/datasette/issues/971,696297601,MDEyOklzc3VlQ29tbWVudDY5NjI5NzYwMQ==,9599,simonw,2020-09-21T18:41:07Z,2020-09-21T18:41:07Z,OWNER,"How to detect it? Looks like it's visible in SQLite compile time options: https://latest.datasette.io/-/versions

```
        ""compile_options"": [
            ""COMPILER=gcc-8.3.0"",
            ""ENABLE_COLUMN_METADATA"",
            ""ENABLE_DBSTAT_VTAB"",
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table,
https://github.com/simonw/datasette/issues/970#issuecomment-695896557,https://api.github.com/repos/simonw/datasette/issues/970,695896557,MDEyOklzc3VlQ29tbWVudDY5NTg5NjU1Nw==,9599,simonw,2020-09-21T04:40:12Z,2020-09-21T04:40:12Z,OWNER,The Python standard library has a module for this: https://docs.python.org/3/library/webbrowser.html,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url",
https://github.com/simonw/datasette/issues/970#issuecomment-695895960,https://api.github.com/repos/simonw/datasette/issues/970,695895960,MDEyOklzc3VlQ29tbWVudDY5NTg5NTk2MA==,9599,simonw,2020-09-21T04:36:45Z,2020-09-21T04:36:45Z,OWNER,I like this. It could work with the `--root` option too and automatically sign you in as the root user.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url",
https://github.com/simonw/sqlite-utils/issues/160#issuecomment-695839557,https://api.github.com/repos/simonw/sqlite-utils/issues/160,695839557,MDEyOklzc3VlQ29tbWVudDY5NTgzOTU1Nw==,9599,simonw,2020-09-20T21:37:03Z,2020-09-20T21:37:03Z,OWNER,"Should this support `ignore=True` as well? I'm tempted to skip that - I think `replace=True` is more useful because it implies ""ignore if the options are already the same, but replace if they are different"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705190723,"table.enable_fts(..., replace=True)",
https://github.com/simonw/sqlite-utils/issues/42#issuecomment-695698227,https://api.github.com/repos/simonw/sqlite-utils/issues/42,695698227,MDEyOklzc3VlQ29tbWVudDY5NTY5ODIyNw==,9599,simonw,2020-09-20T04:27:26Z,2020-09-20T04:28:26Z,OWNER,This is going to need #114 (the `transform_table()` method) in order to convert string columns into integer foreign key columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command",
https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776,https://api.github.com/repos/simonw/sqlite-utils/issues/68,695695776,MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng==,9599,simonw,2020-09-20T04:25:47Z,2020-09-20T04:25:47Z,OWNER,This is a dupe of #130 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531583658,Add support for porter stemming in FTS,
https://github.com/simonw/datasette/issues/943#issuecomment-695133768,https://api.github.com/repos/simonw/datasette/issues/943,695133768,MDEyOklzc3VlQ29tbWVudDY5NTEzMzc2OA==,9599,simonw,2020-09-19T00:06:56Z,2020-09-19T00:07:35Z,OWNER,"[dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) could do with this too. It currently [makes a call](https://github.com/dogsheep/dogsheep-beta/blob/ab36101bdae69b11af7c6bd7edee838d052e6ecf/dogsheep_beta/__init__.py#L216-L225) to `TableView` in a similar way to `datasette-graphql` in order to calculate facets.

`dogsheep-beta` would benefit with a mechanism for changing the facet timeout setting during that call (as would `datasette-graphql`, see the [DatasetteSpecialConfig mechanism](https://github.com/simonw/datasette-graphql/blob/f9dc5c518b7cdc94b93873ef20069a7ea2882a95/datasette_graphql/utils.py#L516-L519) it uses).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/507#issuecomment-693775390,https://api.github.com/repos/simonw/datasette/issues/507,693775390,MDEyOklzc3VlQ29tbWVudDY5Mzc3NTM5MA==,9599,simonw,2020-09-17T02:47:35Z,2020-09-17T02:47:35Z,OWNER,"I have a pattern for creating screenshots using Puppeteer running in a GitHub Action now, see https://simonwillison.net/2020/Sep/3/weeknotes-airtable-screenshots-dogsheep/#weeknotes-2020-09-03-social-media-cards-tils","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455852801,Every datasette plugin on the ecosystem page should have a screenshot,
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693695177,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693695177,MDEyOklzc3VlQ29tbWVudDY5MzY5NTE3Nw==,9599,simonw,2020-09-16T22:17:53Z,2020-09-16T22:17:53Z,OWNER,@spdkils can you share a minimal code example that exhibits the behavior you're seeing?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693694968,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693694968,MDEyOklzc3VlQ29tbWVudDY5MzY5NDk2OA==,9599,simonw,2020-09-16T22:17:19Z,2020-09-16T22:17:19Z,OWNER,"That's strange... this test here doesn't manually commit a transaction and passes:

https://github.com/simonw/sqlite-utils/blob/7805d53bcf11199bd1f2b07e05ae90151f9d0eb0/tests/test_delete.py#L17-L23","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693694343,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693694343,MDEyOklzc3VlQ29tbWVudDY5MzY5NDM0Mw==,9599,simonw,2020-09-16T22:15:39Z,2020-09-16T22:15:39Z,OWNER,"Independent of the transaction changes in #121 I may be able to check `self.conn.in_transaction` to see if a transaction is active and, if one is NOT active, execute the delete inside of one. https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.in_transaction","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693589321,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693589321,MDEyOklzc3VlQ29tbWVudDY5MzU4OTMyMQ==,9599,simonw,2020-09-16T18:41:42Z,2020-09-16T18:41:42Z,OWNER,Yeah I'm going to class this as a bug - that's definitely confusing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/pull/158#issuecomment-693199392,https://api.github.com/repos/simonw/sqlite-utils/issues/158,693199392,MDEyOklzc3VlQ29tbWVudDY5MzE5OTM5Mg==,9599,simonw,2020-09-16T06:21:29Z,2020-09-16T06:21:29Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697203800,Fix accidental mega long line in docs,
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-693199049,https://api.github.com/repos/simonw/sqlite-utils/issues/159,693199049,MDEyOklzc3VlQ29tbWVudDY5MzE5OTA0OQ==,9599,simonw,2020-09-16T06:20:26Z,2020-09-16T06:20:26Z,OWNER,"See #121 - I need to think harder about how this all interacts with transactions.

You can do this:

```python
with db.conn:
    db[""mytable""].delete_where()
```
But that should be documented and maybe rethought.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/datasette/issues/943#issuecomment-693010291,https://api.github.com/repos/simonw/datasette/issues/943,693010291,MDEyOklzc3VlQ29tbWVudDY5MzAxMDI5MQ==,9599,simonw,2020-09-15T22:20:55Z,2020-09-15T22:20:55Z,OWNER,"Should I instantiate a single `Client` and reuse it for all internal requests, or can I instantiate a new `Client` for each request?

https://www.python-httpx.org/advanced/#why-use-a-client says that the main benefit of a Client instance is HTTP connection pooling - which isn't an issue for these internal requests since they won't be using the HTTP protocol at all, they'll be calling the ASGI application directly.

So I'm leaning towards instantiating a fresh client for every internal request. I'll run a microbenchmark to check that this doesn't have any unpleasant performance implications.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693008540,https://api.github.com/repos/simonw/datasette/issues/943,693008540,MDEyOklzc3VlQ29tbWVudDY5MzAwODU0MA==,9599,simonw,2020-09-15T22:16:07Z,2020-09-15T22:16:07Z,OWNER,"I think I can use `async with httpx.AsyncClient(base_url=""http://localhost/"") as client:` to ensure I don't need to use `http://localhost/` on every call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693007512,https://api.github.com/repos/simonw/datasette/issues/943,693007512,MDEyOklzc3VlQ29tbWVudDY5MzAwNzUxMg==,9599,simonw,2020-09-15T22:13:30Z,2020-09-15T22:13:30Z,OWNER,"I could solve streaming using something like this:
```python
async with datasette.stream(""GET"", ""/fixtures/compound_three_primary_keys.csv?_stream=on&_size=max"") as response:
    async for chunk in response.aiter_bytes():
        print(chunk)
```
Which would be a wrapper around `AsyncClient.stream(method, url, ...)` from https://www.python-httpx.org/async/#streaming-responses","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693005033,https://api.github.com/repos/simonw/datasette/issues/943,693005033,MDEyOklzc3VlQ29tbWVudDY5MzAwNTAzMw==,9599,simonw,2020-09-15T22:06:58Z,2020-09-15T22:10:58Z,OWNER,"What if `datasette.get()` was an alias for `httpx.get()`, pre-configured to route to the correct application? And with some sugar that added `http://localhost/` to the beginning of the path if it was missing?

This would make `httpx` a dependency of core Datasette, which I think is OK.

It would also solve the return type problem: I would return whatever `httpx` returns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004770,https://api.github.com/repos/simonw/datasette/issues/943,693004770,MDEyOklzc3VlQ29tbWVudDY5MzAwNDc3MA==,9599,simonw,2020-09-15T22:06:13Z,2020-09-15T22:06:13Z,OWNER,I'm tempted to create a `await datasette.request()` method which can take any HTTP verb - then have `datasette.get()` and `datasette.post()` as thin wrappers around it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004572,https://api.github.com/repos/simonw/datasette/issues/943,693004572,MDEyOklzc3VlQ29tbWVudDY5MzAwNDU3Mg==,9599,simonw,2020-09-15T22:05:39Z,2020-09-15T22:05:39Z,OWNER,"Maybe these methods become the way most Datasette tests are written, replacing the existing `TestClient` mechanism?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693004296,https://api.github.com/repos/simonw/datasette/issues/943,693004296,MDEyOklzc3VlQ29tbWVudDY5MzAwNDI5Ng==,9599,simonw,2020-09-15T22:04:54Z,2020-09-15T22:04:54Z,OWNER,"So what should I do about streaming responses?

I could deliberately ignore them - through an exception if you attempt to run `await datasette.get(...)` against a streaming URL.

I could load the entire response into memory and return it as a wrapped object.

I could support some kind of asynchronous iterator mechanism. This would be pretty elegant if I could decide the right syntax for it - it would allow plugins to take advantage of other internal URLs that return streaming content without needing to load that content entirely into memory in order to process it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-693003652,https://api.github.com/repos/simonw/datasette/issues/943,693003652,MDEyOklzc3VlQ29tbWVudDY5MzAwMzY1Mg==,9599,simonw,2020-09-15T22:03:08Z,2020-09-15T22:03:08Z,OWNER,"I'm not going to mess around with formats - you'll get back the exact response that a web client would receive.

Question: what should the response object look like? e.g. if you do:

    response = await datasette.get(""/db/table.json"")

What should `response` be?

I could reuse the Datasette `Response` class from `datasette.utils.asgi`. This would work well for regular responses which just have a status code, some headers and a response body. It wouldn't be great for streaming responses though such as you get back from `?_stream=1` CSV exports.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/891#issuecomment-693001937,https://api.github.com/repos/simonw/datasette/issues/891,693001937,MDEyOklzc3VlQ29tbWVudDY5MzAwMTkzNw==,9599,simonw,2020-09-15T21:58:56Z,2020-09-15T21:58:56Z,OWNER,"Here's what that looks like:
```
Traceback (most recent call last):
  File ""/Users/simon/Dropbox/Development/datasette/plugins/sql_error.py"", line 5, in oh_no_error
    return 100 / 0
ZeroDivisionError: division by zero
ERROR: conn=<sqlite3.Connection object at 0x10bce0030>, sql = 'select oh_no_error()', params = {}: user-defined function raised exception
INFO:     127.0.0.1:54066 - ""GET /data?sql=select+oh_no_error%28%29 HTTP/1.1"" 400 Bad Request
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-693000522,https://api.github.com/repos/simonw/datasette/issues/891,693000522,MDEyOklzc3VlQ29tbWVudDY5MzAwMDUyMg==,9599,simonw,2020-09-15T21:55:11Z,2020-09-15T21:55:11Z,OWNER,I'm going to turn this on. If people complain about it I can turn it off again (or make it a configuration setting).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-692999893,https://api.github.com/repos/simonw/datasette/issues/891,692999893,MDEyOklzc3VlQ29tbWVudDY5Mjk5OTg5Mw==,9599,simonw,2020-09-15T21:53:36Z,2020-09-15T21:53:36Z,OWNER,"Here's the commit (from 15 years ago) where `enable_callback_tracebacks` was first added: https://github.com/ghaering/pysqlite/commit/1e8bd36be93b7d7425910642b72e4152c77b0dfd

> - Exceptions in callbacks lead to the query being aborted now instead of silently leading to generating values.
> - Exceptions in callbacks can be echoed to stderr if you call the module level function enable_callback_tracebacks: enable_callback_tracebacks(1).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-692998061,https://api.github.com/repos/simonw/datasette/issues/891,692998061,MDEyOklzc3VlQ29tbWVudDY5Mjk5ODA2MQ==,9599,simonw,2020-09-15T21:49:03Z,2020-09-15T21:49:03Z,OWNER,"I've been trying to figure out why this is an optional setting that defaults to off.

I think it's because it writes directly to `stderr`, so the maintainers of `sqlite3` reasonably decided that people should be able to opt in to that rather than having weird stuff show up on `stderr` that they weren't expecting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/891#issuecomment-692968792,https://api.github.com/repos/simonw/datasette/issues/891,692968792,MDEyOklzc3VlQ29tbWVudDY5Mjk2ODc5Mg==,9599,simonw,2020-09-15T20:44:15Z,2020-09-15T20:44:15Z,OWNER,"https://github.com/peter-wangxu/persist-queue/issues/74 warns that this might not work with PyPy.

I could solve that with:
```python
if hasattr(sqlite3, ""enable_callback_tracebacks""):
    sqlite3.enable_callback_tracebacks(True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",653529088,Consider using enable_callback_tracebacks(True),
https://github.com/simonw/datasette/issues/877#issuecomment-692967733,https://api.github.com/repos/simonw/datasette/issues/877,692967733,MDEyOklzc3VlQ29tbWVudDY5Mjk2NzczMw==,9599,simonw,2020-09-15T20:42:04Z,2020-09-15T20:42:04Z,OWNER,"I'm not going to drop CSRF protection - it's still needed for older browsers - but I have relaxed the circumstances under which it is applied. It only applies to requests that include cookies for example, so API clients that don't send cookies don't need to worry about it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/889#issuecomment-692967123,https://api.github.com/repos/simonw/datasette/issues/889,692967123,MDEyOklzc3VlQ29tbWVudDY5Mjk2NzEyMw==,9599,simonw,2020-09-15T20:40:52Z,2020-09-15T20:40:52Z,OWNER,Thanks - I've fixed this in `datasette-media` and the other plugins that use that hook now I think.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649907676,asgi_wrapper plugin hook is crashing at startup,
https://github.com/simonw/datasette/issues/888#issuecomment-692966625,https://api.github.com/repos/simonw/datasette/issues/888,692966625,MDEyOklzc3VlQ29tbWVudDY5Mjk2NjYyNQ==,9599,simonw,2020-09-15T20:39:49Z,2020-09-15T20:39:49Z,OWNER,"Thanks, I've fixed that now. It only affected the GitHub release notes - the ones at https://docs.datasette.io/en/stable/changelog.html#v0-45 had the correct links.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649702801,URLs in release notes point to 127.0.0.1,
https://github.com/simonw/datasette/issues/634#issuecomment-692965761,https://api.github.com/repos/simonw/datasette/issues/634,692965761,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTc2MQ==,9599,simonw,2020-09-15T20:37:58Z,2020-09-15T20:37:58Z,OWNER,I fixed this in 5e0b72247ecab4ce0fcec599b77a83d73a480872,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",522352520,Don't run tests twice when releasing a tag,
https://github.com/simonw/datasette/issues/849#issuecomment-692965391,https://api.github.com/repos/simonw/datasette/issues/849,692965391,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTM5MQ==,9599,simonw,2020-09-15T20:37:14Z,2020-09-15T20:37:14Z,OWNER,I've been running on `main` for a while now with no issues.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/956#issuecomment-692965022,https://api.github.com/repos/simonw/datasette/issues/956,692965022,MDEyOklzc3VlQ29tbWVudDY5Mjk2NTAyMg==,9599,simonw,2020-09-15T20:36:34Z,2020-09-15T20:36:34Z,OWNER,https://hub.docker.com/r/datasetteproject/datasette/tags - 0.49.1 was successfully pushed to Docker Hub by https://github.com/simonw/datasette/runs/1119815175?check_suite_focus=true,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-692955850,https://api.github.com/repos/simonw/datasette/issues/956,692955850,MDEyOklzc3VlQ29tbWVudDY5Mjk1NTg1MA==,9599,simonw,2020-09-15T20:17:49Z,2020-09-15T20:17:49Z,OWNER,I think I've fixed this with recent changes I made as part of #941 - but I won't know until I release the next version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/946#issuecomment-692955379,https://api.github.com/repos/simonw/datasette/issues/946,692955379,MDEyOklzc3VlQ29tbWVudDY5Mjk1NTM3OQ==,9599,simonw,2020-09-15T20:16:50Z,2020-09-15T20:16:50Z,OWNER,Can't reproduce this bug now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",682184050,Exception in tracing code,
https://github.com/simonw/datasette/issues/492#issuecomment-692953174,https://api.github.com/repos/simonw/datasette/issues/492,692953174,MDEyOklzc3VlQ29tbWVudDY5Mjk1MzE3NA==,9599,simonw,2020-09-15T20:12:29Z,2020-09-15T20:12:29Z,OWNER,I fixed this in ea340cf320a2566d24517fb4a0c9852c5059e771 for #963 (a duplicate of this issue).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449854604,Facets not correctly persisted in hidden form fields,
https://github.com/simonw/datasette/issues/967#issuecomment-692951144,https://api.github.com/repos/simonw/datasette/issues/967,692951144,MDEyOklzc3VlQ29tbWVudDY5Mjk1MTE0NA==,9599,simonw,2020-09-15T20:08:12Z,2020-09-15T20:08:12Z,OWNER,I think the easiest fix is for me to ensure that calls to `__len__` on the `MagicParameters` class always return at least 1.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692946616,https://api.github.com/repos/simonw/datasette/issues/967,692946616,MDEyOklzc3VlQ29tbWVudDY5Mjk0NjYxNg==,9599,simonw,2020-09-15T19:59:21Z,2020-09-15T19:59:21Z,OWNER,"I wish I could call https://www.sqlite.org/c3ref/bind_parameter_count.html and https://www.sqlite.org/c3ref/bind_parameter_name.html from Python.

Might be possible to do that using `ctypes` - see this example code: https://mail.python.org/pipermail//pypy-commit/2013-February/071372.html

```python
            param_count = lib.sqlite3_bind_parameter_count(self.statement)
            for idx in range(1, param_count + 1):
                param_name = lib.sqlite3_bind_parameter_name(self.statement,
                                                             idx)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692945504,https://api.github.com/repos/simonw/datasette/issues/967,692945504,MDEyOklzc3VlQ29tbWVudDY5Mjk0NTUwNA==,9599,simonw,2020-09-15T19:57:10Z,2020-09-15T19:57:10Z,OWNER,"So the problem actually occurs when the `MagicParameters` class wraps an empty dictionary.

Relevant code:

https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L228-L236

And:

https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L364-L383

I'm passing a special magic parameters dictionary for the Python `sqlite3` module to look up parameters in. When that dictionary is `{}` a `__len__` check is performed on that dictionary, the result comes back as 0 and as a result it assumes there are no parameters.

I tracked down the relevant C code:

https://github.com/python/cpython/blob/81715808716198471fbca0a3db42ac408468dbc5/Modules/_sqlite/statement.c#L218-L237

```c
    Py_BEGIN_ALLOW_THREADS
    num_params_needed = sqlite3_bind_parameter_count(self->st);
    Py_END_ALLOW_THREADS

    if (PyTuple_CheckExact(parameters) || PyList_CheckExact(parameters) || (!PyDict_Check(parameters) && PySequence_Check(parameters))) {
        /* parameters passed as sequence */
        if (PyTuple_CheckExact(parameters)) {
            num_params = PyTuple_GET_SIZE(parameters);
        } else if (PyList_CheckExact(parameters)) {
            num_params = PyList_GET_SIZE(parameters);
        } else {
            num_params = PySequence_Size(parameters);
        }
        if (num_params != num_params_needed) {
            PyErr_Format(pysqlite_ProgrammingError,
                         ""Incorrect number of bindings supplied. The current ""
                         ""statement uses %d, and there are %zd supplied."",
                         num_params_needed, num_params);
            return;
        }
```

It looks to me like this should fail if the number of keys known to be in the dictionary differs from the number of named parameters in the query. But if those numbers fail to match it still works as far as I can tell - it's only dictionary length of 0 that is causing the problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692940375,https://api.github.com/repos/simonw/datasette/issues/967,692940375,MDEyOklzc3VlQ29tbWVudDY5Mjk0MDM3NQ==,9599,simonw,2020-09-15T19:47:09Z,2020-09-15T19:47:09Z,OWNER,"Yes! The tests all pass if I update the test function to do this:
```python
    response = magic_parameters_client.post(
        ""/data/runme_post{}"".format(qs),
        {""ignore_me"": ""1""},
        csrftoken_from=use_csrf or None,
        allow_redirects=False,
    )
```
So the bug only occurs if the POST body is completely empty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692938935,https://api.github.com/repos/simonw/datasette/issues/967,692938935,MDEyOklzc3VlQ29tbWVudDY5MjkzODkzNQ==,9599,simonw,2020-09-15T19:44:21Z,2020-09-15T19:44:41Z,OWNER,"While I'm running the above test, in the rounds that work the `receive()` awaitable returns `{'type': 'http.request', 'body': b'csrftoken=IlpwUGlSMFVVa3Z3ZlVoamQi.uY2U1tF4i0M-5M6x34vnBCmJgr0'}`

In the rounds that fails it returns `{'type': 'http.request'}`

So it looks like the `csrftoken_from=True` parameter may be helping just by ensuring the `body` key is present and not missing. I wonder if it would work if a body of `b''` was present there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692937150,https://api.github.com/repos/simonw/datasette/issues/967,692937150,MDEyOklzc3VlQ29tbWVudDY5MjkzNzE1MA==,9599,simonw,2020-09-15T19:42:57Z,2020-09-15T19:42:57Z,OWNER,"New (failing) test:
```python
@pytest.mark.parametrize(""use_csrf"", [True, False])
@pytest.mark.parametrize(""return_json"", [True, False])
def test_magic_parameters_csrf_json(magic_parameters_client, use_csrf, return_json):
    magic_parameters_client.ds._metadata[""databases""][""data""][""queries""][""runme_post""][
        ""sql""
    ] = ""insert into logs (line) values (:_header_host)""
    qs = """"
    if return_json:
        qs = ""?_json=1""
    response = magic_parameters_client.post(
        ""/data/runme_post{}"".format(qs),
        {},
        csrftoken_from=use_csrf or None,
        allow_redirects=False,
    )
    if return_json:
        assert response.status == 200
        assert response.json[""ok""], response.json
    else:
        assert response.status == 302
        messages = magic_parameters_client.ds.unsign(
            response.cookies[""ds_messages""], ""messages""
        )
        assert [[""Query executed, 1 row affected"", 1]] == messages
    post_actual = magic_parameters_client.get(
        ""/data/logs.json?_sort_desc=rowid&_shape=array""
    ).json[0][""line""]
    assert post_actual == ""localhost""
```
It passes twice, fails twice - failures are for the ones where `use_csrf` is `False`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692927867,https://api.github.com/repos/simonw/datasette/issues/967,692927867,MDEyOklzc3VlQ29tbWVudDY5MjkyNzg2Nw==,9599,simonw,2020-09-15T19:25:23Z,2020-09-15T19:25:23Z,OWNER,Hunch: I think the `asgi-csrf` middleware may be consuming the request body and failing to restore it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692835066,https://api.github.com/repos/simonw/datasette/issues/967,692835066,MDEyOklzc3VlQ29tbWVudDY5MjgzNTA2Ng==,9599,simonw,2020-09-15T16:40:12Z,2020-09-15T16:40:12Z,OWNER,Is the bug here that magic parameters are incompatible with CSRF-exempt requests (e.g. request with no cookies)?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692834670,https://api.github.com/repos/simonw/datasette/issues/967,692834670,MDEyOklzc3VlQ29tbWVudDY5MjgzNDY3MA==,9599,simonw,2020-09-15T16:39:29Z,2020-09-15T16:39:29Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/853c5fc37011a7bc09ca3a1af287102f00827c82/datasette/views/database.py#L222-L236

This issue may not be about `_json=1` interacting with magic parameters after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692834064,https://api.github.com/repos/simonw/datasette/issues/967,692834064,MDEyOklzc3VlQ29tbWVudDY5MjgzNDA2NA==,9599,simonw,2020-09-15T16:38:21Z,2020-09-15T16:38:21Z,OWNER,So the mystery here is why does omitting `csrftoken_from=True` break the `MagicParameters` mechanism?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/967#issuecomment-692832113,https://api.github.com/repos/simonw/datasette/issues/967,692832113,MDEyOklzc3VlQ29tbWVudDY5MjgzMjExMw==,9599,simonw,2020-09-15T16:34:53Z,2020-09-15T16:37:43Z,OWNER,"This is so weird. In the test I wrote for this the following passed:

    response = magic_parameters_client.post(""/data/runme_post?_json=1"", {}, csrftoken_from=True)

But without the `csrftoken_from=True` parameter it failed with the bindings error:

    response = magic_parameters_client.post(""/data/runme_post?_json=1"", {})

Here's the test I wrote:

```python
def test_magic_parameters_json_body(magic_parameters_client):
    magic_parameters_client.ds._metadata[""databases""][""data""][""queries""][""runme_post""][
        ""sql""
    ] = ""insert into logs (line) values (:_header_host)""
    response = magic_parameters_client.post(""/data/runme_post?_json=1"", {}, csrftoken_from=True)
    assert response.status == 200
    assert response.json[""ok""], response.json
    post_actual = magic_parameters_client.get(
        ""/data/logs.json?_sort_desc=rowid&_shape=array""
    ).json[0][""line""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702069429,Writable canned queries with magic parameters fail if POST body is empty,
https://github.com/simonw/datasette/issues/940#issuecomment-692340275,https://api.github.com/repos/simonw/datasette/issues/940,692340275,MDEyOklzc3VlQ29tbWVudDY5MjM0MDI3NQ==,9599,simonw,2020-09-14T22:09:35Z,2020-09-14T22:09:35Z,OWNER,I'm going to cross my fingers and hope that this works - I don't want to leave this issue open until Datasette 0.50.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692339645,https://api.github.com/repos/simonw/datasette/issues/940,692339645,MDEyOklzc3VlQ29tbWVudDY5MjMzOTY0NQ==,9599,simonw,2020-09-14T22:07:58Z,2020-09-14T22:07:58Z,OWNER,"I shipped the Docker build manually by running the following in a tmate session:

    docker login
    # Typed my username and password interactively
    export REPO=datasetteproject/datasette 
    docker build -f Dockerfile -t $REPO:0.49 . 
    docker tag $REPO:0.49 $REPO:latest
    docker push $REPO 
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692337397,https://api.github.com/repos/simonw/datasette/issues/940,692337397,MDEyOklzc3VlQ29tbWVudDY5MjMzNzM5Nw==,9599,simonw,2020-09-14T22:01:56Z,2020-09-14T22:01:56Z,OWNER,"I'm going to switch to using this logic to decide if I should ship to Docker: https://github.community/t/release-prerelease-action-triggers/17275/2

    if: ""!github.event.release.prerelease""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692336564,https://api.github.com/repos/simonw/datasette/issues/940,692336564,MDEyOklzc3VlQ29tbWVudDY5MjMzNjU2NA==,9599,simonw,2020-09-14T21:59:40Z,2020-09-14T21:59:40Z,OWNER,Using https://github.com/marketplace/actions/debugging-with-tmate to manually submit a new build from within an interactive GitHub Actions session.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692332430,https://api.github.com/repos/simonw/datasette/issues/940,692332430,MDEyOklzc3VlQ29tbWVudDY5MjMzMjQzMA==,9599,simonw,2020-09-14T21:48:59Z,2020-09-14T21:48:59Z,OWNER,"So now I've released Datasette 0.49 but failed to push a new Docker image. This is bad, and I need to fix it.

I'd like to push to Docker from GitHub Actions, so I think I'm going to create a one-off workflow task for doing that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692331919,https://api.github.com/repos/simonw/datasette/issues/940,692331919,MDEyOklzc3VlQ29tbWVudDY5MjMzMTkxOQ==,9599,simonw,2020-09-14T21:47:39Z,2020-09-14T21:47:39Z,OWNER,"I bet that's because the `github.ref` actually looks like this:

`${GITHUB_REF#refs/tags/}`

And the `refs/tags/` part has an `a` in it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-692331349,https://api.github.com/repos/simonw/datasette/issues/940,692331349,MDEyOklzc3VlQ29tbWVudDY5MjMzMTM0OQ==,9599,simonw,2020-09-14T21:46:11Z,2020-09-14T21:46:11Z,OWNER,"Just release Datasette 0.49 - which shipped to PyPI just fine but skipped the Docker step for some reason!

https://github.com/simonw/datasette/runs/1114585275?check_suite_focus=true

<img width=""929"" alt=""Release_0_49_·_simonw_datasette_c024952"" src=""https://user-images.githubusercontent.com/9599/93141431-0571e180-f699-11ea-93c3-acaa68bd1272.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/880#issuecomment-692324230,https://api.github.com/repos/simonw/datasette/issues/880,692324230,MDEyOklzc3VlQ29tbWVudDY5MjMyNDIzMA==,9599,simonw,2020-09-14T21:28:15Z,2020-09-14T21:28:21Z,OWNER,Documentation here: https://docs.datasette.io/en/latest/sql_queries.html#json-api-for-writable-canned-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-692299770,https://api.github.com/repos/simonw/datasette/issues/880,692299770,MDEyOklzc3VlQ29tbWVudDY5MjI5OTc3MA==,9599,simonw,2020-09-14T20:36:40Z,2020-09-14T20:36:40Z,OWNER,"The JSON response will look like this:
```json
{
  ""ok"": true,
  ""message"": ""A message"",
  ""redirect"": ""/blah""
}
```
`""ok""` will be `true` if everything went right and `false` if there was an error.

The `""message""` and `""redirect""` will be whatever was configured using the on_success_message - the message shown `on_success_message`, `on_success_redirect`, `on_error_message` and `on_error_redirect` settings, see https://docs.datasette.io/en/stable/sql_queries.html#writable-canned-queries","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-692298011,https://api.github.com/repos/simonw/datasette/issues/880,692298011,MDEyOklzc3VlQ29tbWVudDY5MjI5ODAxMQ==,9599,simonw,2020-09-14T20:33:13Z,2020-09-14T20:33:13Z,OWNER,"I'm going to support several ways of indicating that you would like a JSON response instead of getting a HTTP redirect from your writable canned query submission:

- Use the `Accept: application/json` request header
- Include `?_json=1` in the request query string
- Include `""_json"": 1` in the form submission (or the JSON body submission)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-692272860,https://api.github.com/repos/simonw/datasette/issues/880,692272860,MDEyOklzc3VlQ29tbWVudDY5MjI3Mjg2MA==,9599,simonw,2020-09-14T19:43:47Z,2020-09-14T19:43:47Z,OWNER,"I'm going to add support for POST content that is sent as a JSON document, in addition to the existing support for key=value encoded POST bodies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-692271804,https://api.github.com/repos/simonw/datasette/issues/880,692271804,MDEyOklzc3VlQ29tbWVudDY5MjI3MTgwNA==,9599,simonw,2020-09-14T19:41:37Z,2020-09-14T19:41:37Z,OWNER,Relevant code section: https://github.com/simonw/datasette/blob/1552ac931e4d2cf516caac3ceeab4fd24da1510a/datasette/views/database.py#L209-L232,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/965#issuecomment-692244252,https://api.github.com/repos/simonw/datasette/issues/965,692244252,MDEyOklzc3VlQ29tbWVudDY5MjI0NDI1Mg==,9599,simonw,2020-09-14T18:49:48Z,2020-09-14T18:49:48Z,OWNER,Documented here: https://docs.datasette.io/en/latest/custom_templates.html#custom-error-pages,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",701294727,"Documentation for 404.html, 500.html templates",
https://github.com/simonw/datasette/issues/965#issuecomment-692231257,https://api.github.com/repos/simonw/datasette/issues/965,692231257,MDEyOklzc3VlQ29tbWVudDY5MjIzMTI1Nw==,9599,simonw,2020-09-14T18:25:04Z,2020-09-14T18:25:04Z,OWNER,In documenting this I realized that it's confusing that the default `500.html` template is often used for non-500 errors (404 for example). I think I'll rename that default template to `error.html` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",701294727,"Documentation for 404.html, 500.html templates",
https://github.com/simonw/datasette/issues/964#issuecomment-692212641,https://api.github.com/repos/simonw/datasette/issues/964,692212641,MDEyOklzc3VlQ29tbWVudDY5MjIxMjY0MQ==,9599,simonw,2020-09-14T17:49:44Z,2020-09-14T17:49:44Z,OWNER,Documentation: https://docs.datasette.io/en/latest/custom_templates.html#returning-404s,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",700728217,raise_404 mechanism for custom templates,
https://github.com/simonw/datasette/issues/965#issuecomment-692207341,https://api.github.com/repos/simonw/datasette/issues/965,692207341,MDEyOklzc3VlQ29tbWVudDY5MjIwNzM0MQ==,9599,simonw,2020-09-14T17:40:05Z,2020-09-14T17:40:05Z,OWNER,Also link to these from the docs added in #964.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",701294727,"Documentation for 404.html, 500.html templates",
https://github.com/simonw/datasette/issues/944#issuecomment-691788478,https://api.github.com/repos/simonw/datasette/issues/944,691788478,MDEyOklzc3VlQ29tbWVudDY5MTc4ODQ3OA==,9599,simonw,2020-09-14T03:21:45Z,2020-09-14T03:21:45Z,OWNER,Having tried this out I think it does need a `raise_404()` mechanism - which needs to be smart enough to trigger the default 404 handler without accidentally going into an infinite loop.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/880#issuecomment-691785692,https://api.github.com/repos/simonw/datasette/issues/880,691785692,MDEyOklzc3VlQ29tbWVudDY5MTc4NTY5Mg==,9599,simonw,2020-09-14T03:10:11Z,2020-09-14T03:10:11Z,OWNER,"Answer: no, it's [not safe](https://twitter.com/glenathan/status/1305081266065244162) to skip CSRF if there's an `Accept: application/json` header because of a nasty old `crossdomain.xml` Flash vulnerability: https://blog.appsecco.com/exploiting-csrf-on-json-endpoints-with-flash-and-redirects-681d4ad6b31b?gi=a5ee3d7a8235","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/940#issuecomment-691781345,https://api.github.com/repos/simonw/datasette/issues/940,691781345,MDEyOklzc3VlQ29tbWVudDY5MTc4MTM0NQ==,9599,simonw,2020-09-14T02:53:25Z,2020-09-14T02:53:49Z,OWNER,"That worked: https://github.com/simonw/datasette/runs/1110040212?check_suite_focus=true ran and deployed https://pypi.org/project/datasette/0.49a1/ to PyPI but it skipped the push to Docker step because there was an ""a"" in the tag.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-691779693,https://api.github.com/repos/simonw/datasette/issues/940,691779693,MDEyOklzc3VlQ29tbWVudDY5MTc3OTY5Mw==,9599,simonw,2020-09-14T02:46:39Z,2020-09-14T02:46:39Z,OWNER,I think those should be single quoted.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-691779510,https://api.github.com/repos/simonw/datasette/issues/940,691779510,MDEyOklzc3VlQ29tbWVudDY5MTc3OTUxMA==,9599,simonw,2020-09-14T02:45:53Z,2020-09-14T02:45:53Z,OWNER,This bit here: https://github.com/simonw/datasette/blob/c18117cf08ad67c704dab29e3cb3b88f1de4026b/.github/workflows/publish.yml#L58-L62,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-691779361,https://api.github.com/repos/simonw/datasette/issues/940,691779361,MDEyOklzc3VlQ29tbWVudDY5MTc3OTM2MQ==,9599,simonw,2020-09-14T02:45:04Z,2020-09-14T02:45:04Z,OWNER,"Package deploys are still broken, just got this error trying to ship 0.49a1: https://github.com/simonw/datasette/actions/runs/253099665

> The workflow is not valid. .github/workflows/publish.yml (Line: 61, Col: 9): Unexpected symbol: '""a""'. Located at position 24 within expression: !(contains(github.ref, ""a"") || contains(github.ref, ""b""))

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/944#issuecomment-691774262,https://api.github.com/repos/simonw/datasette/issues/944,691774262,MDEyOklzc3VlQ29tbWVudDY5MTc3NDI2Mg==,9599,simonw,2020-09-14T02:24:08Z,2020-09-14T02:24:08Z,OWNER,"Actually don't need `{{ raise_404(""Museum not found"") }}` because we already have `{{ custom_status(404) }}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/944#issuecomment-691769222,https://api.github.com/repos/simonw/datasette/issues/944,691769222,MDEyOklzc3VlQ29tbWVudDY5MTc2OTIyMg==,9599,simonw,2020-09-14T02:01:33Z,2020-09-14T02:01:33Z,OWNER,I'm going to cache the `list_templates()` result in memory. If you want to add a new template-defined route you will need to restart the server. I think that's acceptable.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/519#issuecomment-691566247,https://api.github.com/repos/simonw/datasette/issues/519,691566247,MDEyOklzc3VlQ29tbWVudDY5MTU2NjI0Nw==,9599,simonw,2020-09-12T22:48:53Z,2020-09-12T22:48:53Z,OWNER,"I think I've figured out what to do about stability of the HTML and the default templates with respect to semantic versioning.

I'm going to announce that the JSON API - including the variables made available to templates - should be considered stable according to semver. I will only break backwards compatibility at that level in a major version release.

The template HTML (and default CSS) will not be considered a stable interface. They won't change on bug fix releases but they may change (albeit described in the release notes) on minor version bumps.

Since the template inputs are stable, you can run your own copy of the previous version's templates if something breaks.

This means users (and plugin authors) who make changes to the default Datasette UI will have to test their changes against every minor release. I think that's OK.

If you write plugins that don't affect the Datasette HTML UI you will be able to expect stability across minor version releases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0,
https://github.com/simonw/datasette/issues/880#issuecomment-691558387,https://api.github.com/repos/simonw/datasette/issues/880,691558387,MDEyOklzc3VlQ29tbWVudDY5MTU1ODM4Nw==,9599,simonw,2020-09-12T22:04:48Z,2020-09-12T22:04:48Z,OWNER,"Is it safe to skip CSRF checks if the incoming request has `Accept: application/json` on it?

I'm not sure that matters since `asgi-csrf` already won't reject requests that either have no cookies or are using a `Authorization: Bearer ...` header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-691557675,https://api.github.com/repos/simonw/datasette/issues/880,691557675,MDEyOklzc3VlQ29tbWVudDY5MTU1NzY3NQ==,9599,simonw,2020-09-12T22:01:02Z,2020-09-12T22:01:11Z,OWNER,"Maybe POST to `.json` doesn't actually make sense. I could instead support `POST /db/queryname` with an optional mechanism for requesting that the response to that POST be in a JSON format.

Could be a `Accept: application/json` header with an option of including `""_accept"": ""json""` as a POST parameter instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/880#issuecomment-691557429,https://api.github.com/repos/simonw/datasette/issues/880,691557429,MDEyOklzc3VlQ29tbWVudDY5MTU1NzQyOQ==,9599,simonw,2020-09-12T21:59:39Z,2020-09-12T21:59:39Z,OWNER,"What should happen when something does a POST to an extension that was registered by a plugin, e.g. `POST /db/table.atom` ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/782#issuecomment-691554088,https://api.github.com/repos/simonw/datasette/issues/782,691554088,MDEyOklzc3VlQ29tbWVudDY5MTU1NDA4OA==,9599,simonw,2020-09-12T21:39:03Z,2020-09-12T21:39:03Z,OWNER,"Plan: release a new release of Datasette (probably 0.49) with the new JSON API design, but provide a plugin called something like `datasette-api-0-48` which runs as ASGI wrapping middleware and internally rewrites incoming requests to e.g. `/db/table.json` to behave if they have the `?_extra=` params on them necessary to produce the 0.48 version of the JSON.

Anyone who has built applications against 0.48 can install that plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/262#issuecomment-691526975,https://api.github.com/repos/simonw/datasette/issues/262,691526975,MDEyOklzc3VlQ29tbWVudDY5MTUyNjk3NQ==,9599,simonw,2020-09-12T18:22:44Z,2020-09-12T18:22:44Z,OWNER,Are there any interesting use-cases for a plugin hook that allows plugins to define their own `?_extra=` blocks?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/782#issuecomment-691526878,https://api.github.com/repos/simonw/datasette/issues/782,691526878,MDEyOklzc3VlQ29tbWVudDY5MTUyNjg3OA==,9599,simonw,2020-09-12T18:21:41Z,2020-09-12T18:22:20Z,OWNER,"Would it be so bad if the default format had a `""rows""` key containing the array of rows? Maybe it wouldn't. The reason I always use `?_shape=array` is because I want an array of objects, rather than an array of arrays that I have to match up again with their columns.

A default format that's an object rather than array also gives something for the `?_extra=` parameter to add its extras to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-691526762,https://api.github.com/repos/simonw/datasette/issues/782,691526762,MDEyOklzc3VlQ29tbWVudDY5MTUyNjc2Mg==,9599,simonw,2020-09-12T18:20:19Z,2020-09-12T18:20:19Z,OWNER,"I'd like to revisit the idea of using `?_extra=x` to opt-in to extra blocks of JSON, from #262","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/262#issuecomment-691526719,https://api.github.com/repos/simonw/datasette/issues/262,691526719,MDEyOklzc3VlQ29tbWVudDY5MTUyNjcxOQ==,9599,simonw,2020-09-12T18:19:50Z,2020-09-12T18:19:50Z,OWNER,"> Idea: `?_extra=sqllog` could output a lot of every individual SQL statement that was executed in order to generate the page - useful for seeing how foreign key expansion and faceting actually works.

I built a version of that a while ago as the `?_trace=1` argument.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/262#issuecomment-389702480,https://api.github.com/repos/simonw/datasette/issues/262,389702480,MDEyOklzc3VlQ29tbWVudDM4OTcwMjQ4MA==,9599,simonw,2018-05-17T00:00:39Z,2020-09-12T18:19:30Z,OWNER,Idea: `?_extra=sqllog` could output a lot of every individual SQL statement that was executed in order to generate the page - useful for seeing how foreign key expansion and faceting actually works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/issues/680#issuecomment-691526635,https://api.github.com/repos/simonw/datasette/issues/680,691526635,MDEyOklzc3VlQ29tbWVudDY5MTUyNjYzNQ==,9599,simonw,2020-09-12T18:18:50Z,2020-09-12T18:18:50Z,OWNER,"I'm happy with the not-quite-automated way I'm doing this, so I'm going to close this issue.

That's documented here https://docs.datasette.io/en/0.48/contributing.html#release-process - I use https://euangoddard.github.io/clipboard2markdown/ to create the GitHub releases markdown version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",569275763,Release automation: automate the bit that posts the GitHub release,
https://github.com/simonw/datasette/issues/782#issuecomment-691526489,https://api.github.com/repos/simonw/datasette/issues/782,691526489,MDEyOklzc3VlQ29tbWVudDY5MTUyNjQ4OQ==,9599,simonw,2020-09-12T18:17:16Z,2020-09-12T18:17:16Z,OWNER,(I think I may have been over-thinking the details of this is for a couple of years now.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-691526416,https://api.github.com/repos/simonw/datasette/issues/782,691526416,MDEyOklzc3VlQ29tbWVudDY5MTUyNjQxNg==,9599,simonw,2020-09-12T18:16:36Z,2020-09-12T18:16:36Z,OWNER,I'm going to hack together a preview of this in a branch and deploy it somewhere so people can see what I've got planned. Much easier to evaluate a working prototype than static examples.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/963#issuecomment-691379980,https://api.github.com/repos/simonw/datasette/issues/963,691379980,MDEyOklzc3VlQ29tbWVudDY5MTM3OTk4MA==,9599,simonw,2020-09-12T01:50:56Z,2020-09-12T01:50:56Z,OWNER,Good bug - looks like a problem with the hidden form fields.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",699947574,Currently selected array facets are not correctly persisted through hidden form fields,
https://github.com/simonw/datasette/issues/782#issuecomment-691323302,https://api.github.com/repos/simonw/datasette/issues/782,691323302,MDEyOklzc3VlQ29tbWVudDY5MTMyMzMwMg==,9599,simonw,2020-09-11T21:38:27Z,2020-09-11T21:40:04Z,OWNER,"Another idea: the default output could be the list of dicts:
```json
[
  {
    ""pk1"": ""a"",
    ""pk2"": ""a"",
    ""pk3"": ""a"",
    ""content"": ""a-a-a""
  },
  ...
]
```
BUT... I could include pagination information in the HTTP headers - as seen in the WordPress REST API or the GitHub API:

```
~ % curl -s -i 'https://api.github.com/repos/simonw/datasette/commits' | head -n 40
HTTP/1.1 200 OK
server: GitHub.com
date: Fri, 11 Sep 2020 21:37:46 GMT
content-type: application/json; charset=utf-8
status: 200 OK
cache-control: public, max-age=60, s-maxage=60
vary: Accept, Accept-Encoding, Accept, X-Requested-With
etag: W/""71c99379743513394e880c6306b66bf9""
last-modified: Fri, 11 Sep 2020 21:32:54 GMT
x-github-media-type: github.v3; format=json
link: <https://api.github.com/repositories/107914493/commits?page=2>; rel=""next"", <https://api.github.com/repositories/107914493/commits?page=44>; rel=""last""
access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
access-control-allow-origin: *
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'
X-Ratelimit-Limit: 60
X-Ratelimit-Remaining: 55
X-Ratelimit-Reset: 1599863850
X-Ratelimit-Used: 5
Accept-Ranges: bytes
Content-Length: 118240
X-GitHub-Request-Id: EC76:0EAD:313F40:5291A4:5F5BEE37

[
  {
    ""sha"": ""d02f6151dae073135a22d0123e8abdc6cbef7c50"",
    ""node_id"": ""MDY6Q29tbWl0MTA3OTE0NDkzOmQwMmY2MTUxZGFlMDczMTM1YTIyZDAxMjNlOGFiZGM2Y2JlZjdjNTA="",
    ""commit"": {
```
Alternative shapes would provide the pagination information (and other extensions) in the JSON, e.g.:

`/squirrels/squirrels.json?_shape=paginated`
```json
{
  ""rows"": [
  {
    ""pk1"": ""a"",
    ""pk2"": ""a"",
    ""pk3"": ""a"",
    ""content"": ""a-a-a""
  }
  ],
  ""pagination"": {
    ""next"": ""234"",
    ""count"": 442
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/947#issuecomment-691318133,https://api.github.com/repos/simonw/datasette/issues/947,691318133,MDEyOklzc3VlQ29tbWVudDY5MTMxODEzMw==,9599,simonw,2020-09-11T21:23:40Z,2020-09-11T21:23:40Z,OWNER,"I'm going to use exit code 1 for any errors, be they 500 or 404.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684111953,datasette --get exit code should reflect HTTP errors,
https://github.com/simonw/datasette/issues/962#issuecomment-691250299,https://api.github.com/repos/simonw/datasette/issues/962,691250299,MDEyOklzc3VlQ29tbWVudDY5MTI1MDI5OQ==,9599,simonw,2020-09-11T18:33:50Z,2020-09-11T18:33:50Z,OWNER,Since this is purely a debugging option I'm going to allow myself not to write a unit test for it!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",699622046,datasette --pdb option for debugging errors,
https://github.com/simonw/sqlite-utils/issues/157#issuecomment-689850509,https://api.github.com/repos/simonw/sqlite-utils/issues/157,689850509,MDEyOklzc3VlQ29tbWVudDY4OTg1MDUwOQ==,9599,simonw,2020-09-09T22:14:49Z,2020-09-09T22:14:49Z,OWNER,It will call this method: https://github.com/simonw/sqlite-utils/blob/367082e787101fb90901ef3214804ab23a92ce46/sqlite_utils/db.py#L405-L411,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697179806,sqlite-utils add-foreign-keys command,
https://github.com/simonw/sqlite-utils/issues/157#issuecomment-689850289,https://api.github.com/repos/simonw/sqlite-utils/issues/157,689850289,MDEyOklzc3VlQ29tbWVudDY4OTg1MDI4OQ==,9599,simonw,2020-09-09T22:14:19Z,2020-09-09T22:14:19Z,OWNER,"This can accept four arguments: table, column, other_table, other_column:
```
sqlite-utils add-foreign-keys calands.db \
  units_with_maps ACCESS_TYP ACCESS_TYP id \
  units_with_maps AGNCY_NAME AGNCY_NAME id \
  units_with_maps AGNCY_LEV AGNCY_LEV id
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697179806,sqlite-utils add-foreign-keys command,
https://github.com/simonw/sqlite-utils/pull/156#issuecomment-689735140,https://api.github.com/repos/simonw/sqlite-utils/issues/156,689735140,MDEyOklzc3VlQ29tbWVudDY4OTczNTE0MA==,9599,simonw,2020-09-09T18:21:06Z,2020-09-09T18:21:06Z,OWNER,"Good spot, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",697030843,Typos in tests,
https://github.com/simonw/datasette/issues/961#issuecomment-689635754,https://api.github.com/repos/simonw/datasette/issues/961,689635754,MDEyOklzc3VlQ29tbWVudDY4OTYzNTc1NA==,9599,simonw,2020-09-09T15:24:31Z,2020-09-09T15:24:31Z,OWNER,"I thought about checking that every database in the `databases:` section exists and ditto for `tables:` - but actually I think it's useful to be able to keep a `metadata.yml` around with configuration for databases or tables that aren't currently attached to Datasette.

I could treat those as warnings and output a warning to standard out when the server starts instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",696908389,Verification checks for metadata.json on startup,
https://github.com/simonw/datasette/issues/961#issuecomment-689635094,https://api.github.com/repos/simonw/datasette/issues/961,689635094,MDEyOklzc3VlQ29tbWVudDY4OTYzNTA5NA==,9599,simonw,2020-09-09T15:23:24Z,2020-09-09T15:23:24Z,OWNER,"Checks can include:

- `facets:` lists columns that exist
- `sort:` and `sort_desc:` columns
- `fts_table` and `fts_pk` are valid","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",696908389,Verification checks for metadata.json on startup,
https://github.com/simonw/sqlite-utils/issues/145#issuecomment-689186423,https://api.github.com/repos/simonw/sqlite-utils/issues/145,689186423,MDEyOklzc3VlQ29tbWVudDY4OTE4NjQyMw==,9599,simonw,2020-09-08T23:21:23Z,2020-09-08T23:21:23Z,OWNER,Fixed in PR #146.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688659182,Bug when first record contains fewer columns than subsequent records,
https://github.com/simonw/sqlite-utils/pull/146#issuecomment-689185393,https://api.github.com/repos/simonw/sqlite-utils/issues/146,689185393,MDEyOklzc3VlQ29tbWVudDY4OTE4NTM5Mw==,9599,simonw,2020-09-08T23:17:42Z,2020-09-08T23:17:42Z,OWNER,"That seems like a reasonable approach to me, especially since this is going to be a pretty rare edge-case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns,
https://github.com/simonw/sqlite-utils/issues/155#issuecomment-689166404,https://api.github.com/repos/simonw/sqlite-utils/issues/155,689166404,MDEyOklzc3VlQ29tbWVudDY4OTE2NjQwNA==,9599,simonw,2020-09-08T22:20:03Z,2020-09-08T22:20:03Z,OWNER,"I'm going to update `sqlite-utils optimize` to also take an optional list of tables, for consistency.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",696045581,rebuild-fts command and table.rebuild_fts() method,
https://github.com/simonw/sqlite-utils/issues/153#issuecomment-689165985,https://api.github.com/repos/simonw/sqlite-utils/issues/153,689165985,MDEyOklzc3VlQ29tbWVudDY4OTE2NTk4NQ==,9599,simonw,2020-09-08T22:18:52Z,2020-09-08T22:18:52Z,OWNER,"I've reverted this change again, because it turns out using the `rebuild` FTS mechanism is a better way of repairing this issue - see #155.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695377804,table.optimize() should delete junk rows from *_fts_docsize,
https://github.com/simonw/sqlite-utils/issues/155#issuecomment-689163158,https://api.github.com/repos/simonw/sqlite-utils/issues/155,689163158,MDEyOklzc3VlQ29tbWVudDY4OTE2MzE1OA==,9599,simonw,2020-09-08T22:10:27Z,2020-09-08T22:10:27Z,OWNER,"For the command version:

    sqlite-utils rebuild-fts mydb.db

This will rebuild all detected FTS tables. You can also specify one or more explicit tables:

    sqlite-utils rebuild-fts mydb.db dogs
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",696045581,rebuild-fts command and table.rebuild_fts() method,
https://github.com/simonw/sqlite-utils/issues/154#issuecomment-688544156,https://api.github.com/repos/simonw/sqlite-utils/issues/154,688544156,MDEyOklzc3VlQ29tbWVudDY4ODU0NDE1Ng==,9599,simonw,2020-09-07T23:47:10Z,2020-09-07T23:47:10Z,OWNER,This is already covered in the tests though: https://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/tests/test_cli.py#L1300-L1328,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695441530,OperationalError: cannot change into wal mode from within a transaction,
https://github.com/simonw/sqlite-utils/issues/154#issuecomment-688543128,https://api.github.com/repos/simonw/sqlite-utils/issues/154,688543128,MDEyOklzc3VlQ29tbWVudDY4ODU0MzEyOA==,9599,simonw,2020-09-07T23:43:10Z,2020-09-07T23:43:10Z,OWNER,"Running this against the same file works:
```
$ sqlite3 beta.db 
SQLite version 3.31.1 2020-01-27 19:55:54
Enter "".help"" for usage hints.
sqlite> PRAGMA journal_mode=wal;
wal
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695441530,OperationalError: cannot change into wal mode from within a transaction,
https://github.com/simonw/sqlite-utils/issues/152#issuecomment-688500704,https://api.github.com/repos/simonw/sqlite-utils/issues/152,688500704,MDEyOklzc3VlQ29tbWVudDY4ODUwMDcwNA==,9599,simonw,2020-09-07T20:28:45Z,2020-09-07T21:17:48Z,OWNER,"The principle reason to turn these on - at least so far - is that without it weird things happen where FTS tables (in particular `*_fts_docsize`) grow without limit over time, because calls to `INSERT OR REPLACE` against the parent table cause additional rows to be inserted into `*_fts_docsize` even if the row was replaced rather than being inserted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695376054,Turn on recursive_triggers by default,
https://github.com/simonw/sqlite-utils/issues/153#issuecomment-688511161,https://api.github.com/repos/simonw/sqlite-utils/issues/153,688511161,MDEyOklzc3VlQ29tbWVudDY4ODUxMTE2MQ==,9599,simonw,2020-09-07T21:07:20Z,2020-09-07T21:07:29Z,OWNER,"FTS4 uses a different column name here: https://datasette-sqlite-fts4.datasette.io/24ways-fts4/articles_fts_docsize

```
CREATE TABLE 'articles_fts_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695377804,table.optimize() should delete junk rows from *_fts_docsize,
https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688508510,https://api.github.com/repos/simonw/sqlite-utils/issues/146,688508510,MDEyOklzc3VlQ29tbWVudDY4ODUwODUxMA==,9599,simonw,2020-09-07T20:56:03Z,2020-09-07T20:56:24Z,OWNER,"The problem with this approach is that it requires us to consume the entire iterator before we can start inserting rows into the table - here on line 1052:

https://github.com/simonw/sqlite-utils/blob/bb131793feac16bc7181ab997568f941b0220ef2/sqlite_utils/db.py#L1047-L1054

I designed the `.insert_all()` to avoid doing this, because I want to be able to pass it an iterator (or more likely a generator) that could produce potentially millions of records. Doing things one batch of 100 records at a time means that the Python process doesn't need to pull millions of records into memory at once.

`db-to-sqlite` is one example of a tool that uses that characteristic, in https://github.com/simonw/db-to-sqlite/blob/63e4ee972f292de13bb11767c0fb64b35339d954/db_to_sqlite/cli.py#L94-L106

So we need to solve this issue without consuming the entire iterator with a `records = list(records)` call.

I think one way to do this is to execute each chunk one at a time and watch out for an exception that indicates that we sent too many parameters - then adjust the chunk size down and try again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688668680,Handle case where subsequent records (after first batch) include extra columns,
https://github.com/simonw/sqlite-utils/issues/153#issuecomment-688506015,https://api.github.com/repos/simonw/sqlite-utils/issues/153,688506015,MDEyOklzc3VlQ29tbWVudDY4ODUwNjAxNQ==,9599,simonw,2020-09-07T20:46:58Z,2020-09-07T20:46:58Z,OWNER,Writing a test for this will be a tiny bit tricky. I think I'll use a test that replicates the bug in #149.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695377804,table.optimize() should delete junk rows from *_fts_docsize,
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688501064,MDEyOklzc3VlQ29tbWVudDY4ODUwMTA2NA==,9599,simonw,2020-09-07T20:30:15Z,2020-09-07T20:30:38Z,OWNER,"The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL:
```sql
DELETE FROM [licenses_fts_docsize] WHERE id NOT IN (
  SELECT rowid FROM [licenses_fts]);
```
I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/152#issuecomment-688500294,https://api.github.com/repos/simonw/sqlite-utils/issues/152,688500294,MDEyOklzc3VlQ29tbWVudDY4ODUwMDI5NA==,9599,simonw,2020-09-07T20:27:07Z,2020-09-07T20:27:07Z,OWNER,I'm going to make this an argument to the `Database()` class constructor which defaults to `True`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695376054,Turn on recursive_triggers by default,
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688499924,MDEyOklzc3VlQ29tbWVudDY4ODQ5OTkyNA==,9599,simonw,2020-09-07T20:25:40Z,2020-09-07T20:25:50Z,OWNER,"https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:

> Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.

So I think the fix is to turn on `recursive_triggers` globally by default for `sqlite-utils`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499650,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688499650,MDEyOklzc3VlQ29tbWVudDY4ODQ5OTY1MA==,9599,simonw,2020-09-07T20:24:35Z,2020-09-07T20:24:35Z,OWNER,"This replicates the problem:
```
(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
 {""table"": ""licenses"", ""count"": 7},
 {""table"": ""licenses_fts_data"", ""count"": 35},
 {""table"": ""licenses_fts_idx"", ""count"": 16},
 {""table"": ""licenses_fts_docsize"", ""count"": 9151},
 {""table"": ""licenses_fts_config"", ""count"": 1},
 {""table"": ""licenses_fts"", ""count"": 7},
(github-to-sqlite) /tmp % github-to-sqlite repos github.db dogsheep       
(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
 {""table"": ""licenses"", ""count"": 7},
 {""table"": ""licenses_fts_data"", ""count"": 45},
 {""table"": ""licenses_fts_idx"", ""count"": 26},
 {""table"": ""licenses_fts_docsize"", ""count"": 9161},
 {""table"": ""licenses_fts_config"", ""count"": 1},
 {""table"": ""licenses_fts"", ""count"": 7},
```
Note how the number of rows in `licenses_fts_docsize` goes from 9151 to 9161.

The number went up by ten. I used tracing from #151 to show that the following SQL executed ten times:
```
INSERT OR REPLACE INTO [licenses] ([key], [name], [node_id], [spdx_id], [url]) VALUES 
    (?, ?, ?, ?, ?);
```
Then I tried executing `PRAGMA recursive_triggers=on;` at the start of the script. This fixed the problem - running the script did not increase the number of rows in `licenses_fts_docsize`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482355,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688482355,MDEyOklzc3VlQ29tbWVudDY4ODQ4MjM1NQ==,9599,simonw,2020-09-07T19:22:51Z,2020-09-07T19:22:51Z,OWNER,"And the SQLite documentation says:
> When the REPLACE conflict resolution strategy deletes rows in order to satisfy a constraint, [delete triggers](https://www.sqlite.org/lang_createtrigger.html) fire if and only if [recursive triggers](https://www.sqlite.org/pragma.html#pragma_recursive_triggers) are enabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482055,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688482055,MDEyOklzc3VlQ29tbWVudDY4ODQ4MjA1NQ==,9599,simonw,2020-09-07T19:21:42Z,2020-09-07T19:21:42Z,OWNER,"Using `replace=True` there executes `INSERT OR REPLACE` - and Dan Kennedy (SQLite maintainer) on the SQLite forums said this:
> Are you using ""REPLACE INTO"", or ""UPDATE OR REPLACE"" on the ""licenses"" table without having first executed ""PRAGMA recursive_triggers = 1""? The docs note that delete triggers will not be fired in this case, which would explain things. Second paragraph under ""REPLACE"" here:
>
> https://www.sqlite.org/lang_conflict.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688481374,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688481374,MDEyOklzc3VlQ29tbWVudDY4ODQ4MTM3NA==,9599,simonw,2020-09-07T19:19:08Z,2020-09-07T19:19:08Z,OWNER,"reading through the code for `github-to-sqlite repos` - one of the things it does is calls `save_license` for each repo:

https://github.com/dogsheep/github-to-sqlite/blob/39b2234253096bd579feed4e25104698b8ccd2ba/github_to_sqlite/utils.py#L259-L262

```python
def save_license(db, license):
    if license is None:
        return None
    return db[""licenses""].insert(license, pk=""key"", replace=True).last_pk
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688480665,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688480665,MDEyOklzc3VlQ29tbWVudDY4ODQ4MDY2NQ==,9599,simonw,2020-09-07T19:16:20Z,2020-09-07T19:16:20Z,OWNER,"Aha! I have managed to replicate the bug:
```
(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
 {""table"": ""licenses"", ""count"": 7},
 {""table"": ""licenses_fts_data"", ""count"": 35},
 {""table"": ""licenses_fts_idx"", ""count"": 16},
 {""table"": ""licenses_fts_docsize"", ""count"": 9151},
 {""table"": ""licenses_fts_config"", ""count"": 1},
 {""table"": ""licenses_fts"", ""count"": 7},
(github-to-sqlite) /tmp % github-to-sqlite repos github.db dogsheep       
(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses
 {""table"": ""licenses"", ""count"": 7},
 {""table"": ""licenses_fts_data"", ""count"": 45},
 {""table"": ""licenses_fts_idx"", ""count"": 26},
 {""table"": ""licenses_fts_docsize"", ""count"": 9161},
 {""table"": ""licenses_fts_config"", ""count"": 1},
 {""table"": ""licenses_fts"", ""count"": 7},
```
Note that the number of records in `licenses_fts_docsize` went from 9151 to 9161.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688464181,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688464181,MDEyOklzc3VlQ29tbWVudDY4ODQ2NDE4MQ==,9599,simonw,2020-09-07T18:19:54Z,2020-09-07T18:19:54Z,OWNER,"Even though that table doesn't declare an integer primary key it does have a `rowid` column: https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+%5Bkey%5D%2C+name%2C+spdx_id%2C+url%2C+node_id+from+licenses+order+by+%5Bkey%5D+limit+101

| rowid | key | name | spdx_id | url | node_id |
| --- | --- | --- | --- | --- | --- |
| 9150 | apache-2.0 | Apache License 2.0 | Apache-2.0 | <https://api.github.com/licenses/apache-2.0> | MDc6TGljZW5zZTI= |
| 112 | bsd-3-clause | BSD 3-Clause ""New"" or ""Revised"" License | BSD-3-Clause | <https://api.github.com/licenses/bsd-3-clause> | MDc6TGljZW5zZTU= |

https://www.sqlite.org/rowidtable.html explains has this clue:

>  If the rowid is not aliased by INTEGER PRIMARY KEY then it is not persistent and might change. In particular the VACUUM command will change rowids for tables that do not declare an INTEGER PRIMARY KEY. Therefore, applications should not normally access the rowid directly, but instead use an INTEGER PRIMARY KEY. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460865,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688460865,MDEyOklzc3VlQ29tbWVudDY4ODQ2MDg2NQ==,9599,simonw,2020-09-07T18:07:14Z,2020-09-07T18:07:14Z,OWNER,"Another likely culprit: `licenses` has a text primary key, so it's not using `rowid`:
```sql
CREATE TABLE [licenses] (
   [key] TEXT PRIMARY KEY,
   [name] TEXT,
   [spdx_id] TEXT,
   [url] TEXT,
   [node_id] TEXT
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460729,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688460729,MDEyOklzc3VlQ29tbWVudDY4ODQ2MDcyOQ==,9599,simonw,2020-09-07T18:06:44Z,2020-09-07T18:06:44Z,OWNER,First posted on SQLite forum here but I'm pretty sure this is a bug in how `sqlite-utils` created those tables: https://sqlite.org/forum/forumpost/51aada1b45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258,"FTS table with 7 rows has _fts_docsize table with 9,141 rows",
https://github.com/simonw/sqlite-utils/issues/148#issuecomment-688434226,https://api.github.com/repos/simonw/sqlite-utils/issues/148,688434226,MDEyOklzc3VlQ29tbWVudDY4ODQzNDIyNg==,9599,simonw,2020-09-07T16:50:33Z,2020-09-07T16:50:33Z,OWNER,"This may be as easy as applying `textwrap.dedent()` to this: https://github.com/simonw/sqlite-utils/blob/0e62744da9a429093e3409575c1f881376b0361f/sqlite_utils/db.py#L778-L787

I could apply that to a few other queries in that code as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695276328,More attractive indentation of created FTS table schema,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-683528149,https://api.github.com/repos/simonw/sqlite-utils/issues/147,683528149,MDEyOklzc3VlQ29tbWVudDY4MzUyODE0OQ==,9599,simonw,2020-08-31T03:17:26Z,2020-08-31T03:17:26Z,OWNER,"+1 to making this something that users can customize. An optional argument to the `Database` constructor would be a neat way to do this.

I think there's a terrifying way that we could find this value... we could perform a binary search for it! Open up a memory connection and try running different bulk inserts against it and catch the exceptions - then adjust and try again.

My hunch is that we could perform just 2 or 3 probes (maybe against carefully selected values) to find the highest value that works. If this process took less than a few ms to run I'd be happy to do it automatically when the class is instantiated (and let users disable that automatic proving by setting a value using the constructor argument).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/datasette/issues/948#issuecomment-683448569,https://api.github.com/repos/simonw/datasette/issues/948,683448569,MDEyOklzc3VlQ29tbWVudDY4MzQ0ODU2OQ==,9599,simonw,2020-08-30T17:39:09Z,2020-08-30T18:34:34Z,OWNER,"So the steps needed are:

- Download and extract latest CodeMirror zip file
- Rename `lib/codemirror.js` to `codemirror-5.57.0.js`
- Rename `lib/codemirror.css` to `codemirror-5.57.0.css`
- Rename `mode/sql/sql.js` to `codemirror-5.57.0-sql.js`
- Edit both JS files to make the top comment a `/* */` block
- Minify JavaScript files like this:
  - `npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'`
  - `npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js --comments '/LICENSE/'`
- Check that the LICENSE comment did indeed survive minification
- Minify CSS file like this:
  - `npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css`
- Edit the `_codemirror.html` template to reference the new files
- `git rm` the old files, `git add` the new files","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-683452613,https://api.github.com/repos/simonw/datasette/issues/948,683452613,MDEyOklzc3VlQ29tbWVudDY4MzQ1MjYxMw==,9599,simonw,2020-08-30T18:16:28Z,2020-08-30T18:16:28Z,OWNER,I added documentation on how to upgrade CodeMirror for the future here: https://docs.datasette.io/en/latest/contributing.html#upgrading-codemirror,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/655#issuecomment-683449837,https://api.github.com/repos/simonw/datasette/issues/655,683449837,MDEyOklzc3VlQ29tbWVudDY4MzQ0OTgzNw==,9599,simonw,2020-08-30T17:51:38Z,2020-08-30T17:51:38Z,OWNER,I think was fixed by #948,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542553350,Copy and paste doesn't work reliably on iPhone for SQL editor,
https://github.com/simonw/datasette/issues/948#issuecomment-683449804,https://api.github.com/repos/simonw/datasette/issues/948,683449804,MDEyOklzc3VlQ29tbWVudDY4MzQ0OTgwNA==,9599,simonw,2020-08-30T17:51:18Z,2020-08-30T17:51:18Z,OWNER,Copy and paste on mobile safari seems to work now. #655 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-683448635,https://api.github.com/repos/simonw/datasette/issues/948,683448635,MDEyOklzc3VlQ29tbWVudDY4MzQ0ODYzNQ==,9599,simonw,2020-08-30T17:39:54Z,2020-08-30T17:39:54Z,OWNER,I'll wait for this to deploy to https://latest.datasette.io/ and then test it in various desktop and mobile browsers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-683445704,https://api.github.com/repos/simonw/datasette/issues/948,683445704,MDEyOklzc3VlQ29tbWVudDY4MzQ0NTcwNA==,9599,simonw,2020-08-30T17:11:58Z,2020-08-30T17:33:30Z,OWNER,"One catch: this stripped the license information from the top of the JS.

I fixed this by editing the license to be a single `/* ... */` block comment instead of multiple `//` lines and running this:

    npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-683445114,https://api.github.com/repos/simonw/datasette/issues/948,683445114,MDEyOklzc3VlQ29tbWVudDY4MzQ0NTExNA==,9599,simonw,2020-08-30T17:06:39Z,2020-08-30T17:06:39Z,OWNER,"Minifying using `npx`:
```
npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js
npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js
npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/957#issuecomment-683357092,https://api.github.com/repos/simonw/datasette/issues/957,683357092,MDEyOklzc3VlQ29tbWVudDY4MzM1NzA5Mg==,9599,simonw,2020-08-30T00:15:51Z,2020-08-30T00:16:02Z,OWNER,"Weirdly even removing this single `datasette` import from `utils/asgi.py` didn't fix the circular import:

https://github.com/simonw/datasette/blob/44cf424a94a85b74552075272660bb96a7432661/datasette/utils/asgi.py#L1-L3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/957#issuecomment-683356440,https://api.github.com/repos/simonw/datasette/issues/957,683356440,MDEyOklzc3VlQ29tbWVudDY4MzM1NjQ0MA==,9599,simonw,2020-08-30T00:08:18Z,2020-08-30T00:10:26Z,OWNER,"Annoyingly this seems to be the line that causes the circular import:
```python
from .utils.asgi import Forbidden, NotFound, Response
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/957#issuecomment-683355993,https://api.github.com/repos/simonw/datasette/issues/957,683355993,MDEyOklzc3VlQ29tbWVudDY4MzM1NTk5Mw==,9599,simonw,2020-08-30T00:02:11Z,2020-08-30T00:04:18Z,OWNER,"I tried doing this and got this error:
```
(datasette) datasette % pytest
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
plugins: asyncio-0.14.0, timeout-1.4.2
collected 1 item / 23 errors                                                                                                                                 

=========================================================================== ERRORS ===========================================================================
_____________________________________________________________ ERROR collecting tests/test_api.py _____________________________________________________________
ImportError while importing test module '/Users/simon/Dropbox/Development/datasette/tests/test_api.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_api.py:1: in <module>
    from datasette.plugins import DEFAULT_PLUGINS
datasette/__init__.py:2: in <module>
    from .app import Datasette
datasette/app.py:30: in <module>
    from .views.base import DatasetteError, ureg
datasette/views/base.py:12: in <module>
    from datasette.plugins import pm
datasette/plugins.py:26: in <module>
    mod = importlib.import_module(plugin)
/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
datasette/publish/heroku.py:2: in <module>
    from datasette import hookimpl
E   ImportError: cannot import name 'hookimpl' from partially initialized module 'datasette' (most likely due to a circular import) (/Users/simon/Dropbox/Development/datasette/datasette/__init__.py)
```
That's with `datasette/__init__.py` looking like this:
```python
from datasette.version import __version_info__, __version__  # noqa
from .app import Datasette
from .utils.asgi import Forbidden, NotFound, Response
from .utils import actor_matches_allow, QueryInterrupted
from .hookspecs import hookimpl  # noqa
from .hookspecs import hookspec  # noqa


__all__ = [
    ""actor_matches_allow"",
    ""hookimpl"",
    ""hookspec"",
    ""QueryInterrupted"",
    ""Forbidden"",
    ""NotFound"",
    ""Response"",
    ""Datasette"",
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/957#issuecomment-683355598,https://api.github.com/repos/simonw/datasette/issues/957,683355598,MDEyOklzc3VlQ29tbWVudDY4MzM1NTU5OA==,9599,simonw,2020-08-29T23:55:10Z,2020-08-29T23:55:34Z,OWNER,"Of these I think I'm going to promote the following to being importable directly `from datasette`:

- `from datasette.app import Datasette`
- `from datasette.utils import QueryInterrupted`
- `from datasette.utils.asgi import Response, Forbidden, NotFound`
- `from datasette.utils import actor_matches_allow`

All of the rest are infrequently used enough (or clearly named enough) that I'm happy to leave them as-is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/957#issuecomment-683355508,https://api.github.com/repos/simonw/datasette/issues/957,683355508,MDEyOklzc3VlQ29tbWVudDY4MzM1NTUwOA==,9599,simonw,2020-08-29T23:54:01Z,2020-08-29T23:54:01Z,OWNER,"Reviewing https://github.com/search?q=user%3Asimonw+%22from+datasette%22&type=Code I spotted these others:

```python
# Various:
from datasette.utils import path_with_replaced_args
from datasette.plugins import pm
from datasette.utils import QueryInterrupted
from datasette.utils.asgi import Response, Forbidden, NotFound

# datasette-publish-vercel:
from datasette.publish.common import (
    add_common_publish_arguments_and_options,
    fail_if_publish_binary_not_installed
)
from datasette.utils import temporary_docker_directory

# datasette-insert
from datasette.utils import actor_matches_allow, sqlite3

# obsolete: russian-ira-facebook-ads-datasette 
from datasette.utils import TableFilter

# simonw/museums
from datasette.utils.asgi import asgi_send

# datasette-media
from datasette.utils.asgi import Response, asgi_send_file

# datasette/tests/plugins/my_plugin.py
from datasette.facets import Facet

# datasette-graphql
from datasette.views.table import TableView
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148,Simplify imports of common classes,
https://github.com/simonw/datasette/issues/956#issuecomment-683214102,https://api.github.com/repos/simonw/datasette/issues/956,683214102,MDEyOklzc3VlQ29tbWVudDY4MzIxNDEwMg==,9599,simonw,2020-08-29T01:32:21Z,2020-08-29T01:32:21Z,OWNER,Maybe the bug here is the double colon?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-683213973,https://api.github.com/repos/simonw/datasette/issues/956,683213973,MDEyOklzc3VlQ29tbWVudDY4MzIxMzk3Mw==,9599,simonw,2020-08-29T01:31:39Z,2020-08-29T01:31:39Z,OWNER,"Here's how the old Travis mechanism worked: https://github.com/simonw/datasette/blob/52eabb019d4051084b21524bd0fd9c2731126985/.travis.yml#L41-L47

So I was assuming that the eqivalent of `$REPO:$TRAVIS_TAG` in GitHub Actions is `$REPO::${GITHUB_REF#refs/tags/}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-683212960,https://api.github.com/repos/simonw/datasette/issues/956,683212960,MDEyOklzc3VlQ29tbWVudDY4MzIxMjk2MA==,9599,simonw,2020-08-29T01:25:34Z,2020-08-29T01:25:34Z,OWNER,So I guess this bit is wrong: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L71-L73,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-683212421,https://api.github.com/repos/simonw/datasette/issues/956,683212421,MDEyOklzc3VlQ29tbWVudDY4MzIxMjQyMQ==,9599,simonw,2020-08-29T01:22:23Z,2020-08-29T01:22:23Z,OWNER,"Here's the error message again:

> invalid argument `""***/datasette::0.49a0""` for `""-t, --tag""` flag: invalid reference format","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/956#issuecomment-683212246,https://api.github.com/repos/simonw/datasette/issues/956,683212246,MDEyOklzc3VlQ29tbWVudDY4MzIxMjI0Ng==,9599,simonw,2020-08-29T01:21:26Z,2020-08-29T01:21:26Z,OWNER,I added this but I have no idea if I got it right or not: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L58-L63,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751,Push to Docker Hub failed - but it shouldn't run for alpha releases anyway,
https://github.com/simonw/datasette/issues/955#issuecomment-683189334,https://api.github.com/repos/simonw/datasette/issues/955,683189334,MDEyOklzc3VlQ29tbWVudDY4MzE4OTMzNA==,9599,simonw,2020-08-28T23:30:48Z,2020-08-28T23:30:48Z,OWNER,Also https://github.com/simonw/datasette-copyable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687711713,Release updated datasette-atom and datasette-ics,
https://github.com/simonw/datasette/issues/955#issuecomment-683185861,https://api.github.com/repos/simonw/datasette/issues/955,683185861,MDEyOklzc3VlQ29tbWVudDY4MzE4NTg2MQ==,9599,simonw,2020-08-28T23:17:09Z,2020-08-28T23:17:09Z,OWNER,I released 0.49a0 which means I can update the main branches of those two plugins - I'll push a release of them once 0.49 is fully shipped.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687711713,Release updated datasette-atom and datasette-ics,
https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683180581,https://api.github.com/repos/simonw/sqlite-utils/issues/144,683180581,MDEyOklzc3VlQ29tbWVudDY4MzE4MDU4MQ==,9599,simonw,2020-08-28T22:57:04Z,2020-08-28T22:57:04Z,OWNER,"That worked! https://github.com/simonw/sqlite-utils/runs/1043640785?check_suite_focus=true

<img width=""1001"" alt=""Add_numpy_to_the_matrix__refs__144_·_simonw_sqlite-utils_cbc22ef"" src=""https://user-images.githubusercontent.com/9599/91621188-1b686e00-e947-11ea-906d-129ceadbb434.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688395275,Run some tests against numpy,
https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683179678,https://api.github.com/repos/simonw/sqlite-utils/issues/144,683179678,MDEyOklzc3VlQ29tbWVudDY4MzE3OTY3OA==,9599,simonw,2020-08-28T22:53:17Z,2020-08-28T22:53:17Z,OWNER,I'm going to try doing this as a GitHub Actions test matrix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688395275,Run some tests against numpy,
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-683178570,https://api.github.com/repos/simonw/sqlite-utils/issues/139,683178570,MDEyOklzc3VlQ29tbWVudDY4MzE3ODU3MA==,9599,simonw,2020-08-28T22:48:51Z,2020-08-28T22:48:51Z,OWNER,"Thanks @simonwiles, this is now released in 2.16.1: https://sqlite-utils.readthedocs.io/en/stable/changelog.html","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/143#issuecomment-683175491,https://api.github.com/repos/simonw/sqlite-utils/issues/143,683175491,MDEyOklzc3VlQ29tbWVudDY4MzE3NTQ5MQ==,9599,simonw,2020-08-28T22:37:15Z,2020-08-28T22:37:15Z,OWNER,"I'm going to start running black exclusively in the GitHub Actions workflow, rather than having it run by the unit tests themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688389933,Move to GitHub Actions CI,
https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683173375,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683173375,MDEyOklzc3VlQ29tbWVudDY4MzE3MzM3NQ==,9599,simonw,2020-08-28T22:29:02Z,2020-08-28T22:29:02Z,OWNER,Yeah I think that failure is actually because there's a brand new release of Black out and it subtly changes some of the formatting rules. I'll merge this and then run Black against the entire codebase.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172829,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683172829,MDEyOklzc3VlQ29tbWVudDY4MzE3MjgyOQ==,9599,simonw,2020-08-28T22:27:05Z,2020-08-28T22:27:05Z,OWNER,"Looks like it failed the ""black"" formatting test - possibly because there's a new release if black out. I'm going to merge despite that failure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172082,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683172082,MDEyOklzc3VlQ29tbWVudDY4MzE3MjA4Mg==,9599,simonw,2020-08-28T22:24:25Z,2020-08-28T22:24:25Z,OWNER,Thanks very much!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/119#issuecomment-683146200,https://api.github.com/repos/simonw/sqlite-utils/issues/119,683146200,MDEyOklzc3VlQ29tbWVudDY4MzE0NjIwMA==,9599,simonw,2020-08-28T21:05:37Z,2020-08-28T21:05:37Z,OWNER,Maybe use `transform_table()` in #114 for this? Would be less efficient as it would copy the whole table but it would reduce library complexity a bit.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770,Ability to remove a foreign key,
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682771226,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682771226,MDEyOklzc3VlQ29tbWVudDY4Mjc3MTIyNg==,9599,simonw,2020-08-28T15:57:42Z,2020-08-28T15:57:42Z,OWNER,"That pull request should update this section of the documentation too:

> If you have more than one record to insert, the insert_all() method is a much more efficient way of inserting them. Just like insert() it will automatically detect the columns that should be created, but it will inspect the first batch of 100 items to help decide what those column types should be.

https://github.com/simonw/sqlite-utils/blob/ea87c2b943fdd162c42a900ac0aea5ecc2f4b9d9/docs/python-api.rst#L393","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682762911,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682762911,MDEyOklzc3VlQ29tbWVudDY4Mjc2MjkxMQ==,9599,simonw,2020-08-28T15:54:57Z,2020-08-28T15:55:20Z,OWNER,"Here's a suggested test update:
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index a8791c3..12fa2f2 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1074,6 +1074,13 @@ class Table(Queryable):
                 all_columns = list(sorted(all_columns))
                 if hash_id:
                     all_columns.insert(0, hash_id)
+            else:
+                all_columns += [
+                    column
+                    for record in chunk
+                    for column in record
+                    if column not in all_columns
+                ]
             validate_column_names(all_columns)
             first = False
             # values is the list of insert data that is passed to the
diff --git a/tests/test_create.py b/tests/test_create.py
index a84eb8d..3a7fafc 100644
--- a/tests/test_create.py
+++ b/tests/test_create.py
@@ -707,13 +707,15 @@ def test_insert_thousands_using_generator(fresh_db):
     assert 10000 == fresh_db[""test""].count
 
 
-def test_insert_thousands_ignores_extra_columns_after_first_100(fresh_db):
+def test_insert_thousands_adds_extra_columns_after_first_100(fresh_db):
+    # https://github.com/simonw/sqlite-utils/issues/139
     fresh_db[""test""].insert_all(
         [{""i"": i, ""word"": ""word_{}"".format(i)} for i in range(100)]
-        + [{""i"": 101, ""extra"": ""This extra column should cause an exception""}]
+        + [{""i"": 101, ""extra"": ""Should trigger ALTER""}],
+        alter=True,
     )
     rows = fresh_db.execute_returning_dicts(""select * from test where i = 101"")
-    assert [{""i"": 101, ""word"": None}] == rows
+    assert [{""i"": 101, ""word"": None, ""extra"": ""Should trigger ALTER""}] == rows
 
 
 def test_insert_ignore(fresh_db):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/datasette/issues/954#issuecomment-682312736,https://api.github.com/repos/simonw/datasette/issues/954,682312736,MDEyOklzc3VlQ29tbWVudDY4MjMxMjczNg==,9599,simonw,2020-08-28T04:05:01Z,2020-08-28T04:05:10Z,OWNER,>  It can also return a dictionary with the following keys. This format is **deprecated** as-of Datasette 0.49 and will be removed by Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687694947,Remove old register_output_renderer dict mechanism in Datasette 1.0,
https://github.com/simonw/datasette/issues/953#issuecomment-682312494,https://api.github.com/repos/simonw/datasette/issues/953,682312494,MDEyOklzc3VlQ29tbWVudDY4MjMxMjQ5NA==,9599,simonw,2020-08-28T04:03:56Z,2020-08-28T04:03:56Z,OWNER,"Documentation says that the old dictionary mechanism will be deprecated by 1.0:

https://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687681018,register_output_renderer render function should be able to return a Response,
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682285212,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682285212,MDEyOklzc3VlQ29tbWVudDY4MjI4NTIxMg==,9599,simonw,2020-08-28T02:12:51Z,2020-08-28T02:12:51Z,OWNER,"I'd be happy to accept a PR for this, provided it included updated unit tests that illustrate it working. I think this is a really good improvement.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682284908,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682284908,MDEyOklzc3VlQ29tbWVudDY4MjI4NDkwOA==,9599,simonw,2020-08-28T02:11:40Z,2020-08-28T02:11:40Z,OWNER,"This is deliberate behaviour, but I'm not at all attached to it - you're right in pointing out that it's actually pretty unexpected.

I'd be happy to change this behaviour so if you pass `alter=True` and then use `.insert_all()` on more than 100 rows it works as you would expect, instead of silently ignoring new columns past the first 100 rows. I don't expect that anyone would be depending on the current behaviour (ignore new columns after the first 100) such that this should be considered a backwards incompatible change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131,"insert_all(..., alter=True) should work for new columns introduced after the first 100 records",
https://github.com/simonw/datasette/issues/950#issuecomment-680374196,https://api.github.com/repos/simonw/datasette/issues/950,680374196,MDEyOklzc3VlQ29tbWVudDY4MDM3NDE5Ng==,9599,simonw,2020-08-26T00:43:50Z,2020-08-26T00:43:50Z,OWNER,"The problem with the term ""private"" is that it could be confused with the concept of databases that aren't visible to the public due to the permissions system - the ones that are displayed with the padlock icon e.g. on https://datasette-auth-passwords-demo.datasette.io/

<img width=""595"" alt=""datasette-auth-passwords_demo__public__private"" src=""https://user-images.githubusercontent.com/9599/91241752-807b5400-e6fa-11ea-9f71-9da37c371b45.png"">

So I think ""secret"" is a better term for these.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/950#issuecomment-680264202,https://api.github.com/repos/simonw/datasette/issues/950,680264202,MDEyOklzc3VlQ29tbWVudDY4MDI2NDIwMg==,9599,simonw,2020-08-25T20:53:13Z,2020-08-25T20:53:13Z,OWNER,Forcing people to spell out `datasette github.db --private private.db` isn't terrible though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/950#issuecomment-680263999,https://api.github.com/repos/simonw/datasette/issues/950,680263999,MDEyOklzc3VlQ29tbWVudDY4MDI2Mzk5OQ==,9599,simonw,2020-08-25T20:52:47Z,2020-08-25T20:52:47Z,OWNER,"Naming challenge: secret databases or private databases?

I prefer private. But `datasette -p` is already taken by `--port`. `datasette -s` is currently available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/950#issuecomment-680263427,https://api.github.com/repos/simonw/datasette/issues/950,680263427,MDEyOklzc3VlQ29tbWVudDY4MDI2MzQyNw==,9599,simonw,2020-08-25T20:51:30Z,2020-08-25T20:52:13Z,OWNER,"`datasette-graphql` currently dispatches requests through the `TableView` class, so if that couldn't access private databases then it would not be able to either. See also the concept for `datasette.get(...)` as an internal API in #943 - that might need to have a mechanism for also being able to query private databases, maybe `datasette.get(path, allow_private=True)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/949#issuecomment-679367931,https://api.github.com/repos/simonw/datasette/issues/949,679367931,MDEyOklzc3VlQ29tbWVudDY3OTM2NzkzMQ==,9599,simonw,2020-08-24T21:09:50Z,2020-08-24T21:09:50Z,OWNER,"I'm attracted to this because of how good GraphiQL is for auto-completing queries. But I realize there's a problem here: GraphQL is designed to be autocomplete-friendly, but SQL is not. If you type `select ` and it doesn't know what's going in the `from` clause it can't give you good column autocomplete, for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints,
https://github.com/simonw/datasette/issues/949#issuecomment-679363710,https://api.github.com/repos/simonw/datasette/issues/949,679363710,MDEyOklzc3VlQ29tbWVudDY3OTM2MzcxMA==,9599,simonw,2020-08-24T21:00:43Z,2020-08-24T21:00:43Z,OWNER,"I think this requires three extra files from https://github.com/codemirror/CodeMirror/tree/5.57.0/addon/hint

- `show-hint.css`
- `show-hint.js`
- `sql-hint.js`

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints,
https://github.com/simonw/datasette/issues/948#issuecomment-679355426,https://api.github.com/repos/simonw/datasette/issues/948,679355426,MDEyOklzc3VlQ29tbWVudDY3OTM1NTQyNg==,9599,simonw,2020-08-24T20:43:07Z,2020-08-24T20:43:07Z,OWNER,"It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/datasette/issues/948#issuecomment-679333717,https://api.github.com/repos/simonw/datasette/issues/948,679333717,MDEyOklzc3VlQ29tbWVudDY3OTMzMzcxNw==,9599,simonw,2020-08-24T19:55:59Z,2020-08-24T19:55:59Z,OWNER,CodeMirror 6 is in pre-release at the moment and is a complete rewrite. I'll stick with the 5.x series for now. https://github.com/codemirror/codemirror.next/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907,Upgrade CodeMirror,
https://github.com/simonw/sqlite-utils/issues/138#issuecomment-678732667,https://api.github.com/repos/simonw/sqlite-utils/issues/138,678732667,MDEyOklzc3VlQ29tbWVudDY3ODczMjY2Nw==,9599,simonw,2020-08-23T05:46:10Z,2020-08-23T05:46:10Z,OWNER,"Actually the `TEXT` column thing wasn't a `sqlite-utils` issue, it was unique to how `shapefile-to-spatialite` was creating the table when using the SpatiaLite extension.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684118950,extracts= doesn't configure foreign keys,
https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678508056,https://api.github.com/repos/simonw/sqlite-utils/issues/136,678508056,MDEyOklzc3VlQ29tbWVudDY3ODUwODA1Ng==,9599,simonw,2020-08-21T21:13:41Z,2020-08-21T21:13:41Z,OWNER,"The `--spatialite` option should be available for other useful commands too, refs #137.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642,--load-extension=spatialite shortcut option,
https://github.com/simonw/sqlite-utils/issues/137#issuecomment-678507502,https://api.github.com/repos/simonw/sqlite-utils/issues/137,678507502,MDEyOklzc3VlQ29tbWVudDY3ODUwNzUwMg==,9599,simonw,2020-08-21T21:13:19Z,2020-08-21T21:13:19Z,OWNER,Adding `--spatialite` too would be great for usability: #136,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683830416,--load-extension for other sqlite-utils commands,
https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678497497,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678497497,MDEyOklzc3VlQ29tbWVudDY3ODQ5NzQ5Nw==,9599,simonw,2020-08-21T21:06:26Z,2020-08-21T21:06:26Z,OWNER,"Ended up needing two skipIfs:

https://github.com/simonw/sqlite-utils/blob/7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d/tests/test_cli.py#L888-L893","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172,--load-extension option for sqlite-utils query,
https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678480969,https://api.github.com/repos/simonw/sqlite-utils/issues/136,678480969,MDEyOklzc3VlQ29tbWVudDY3ODQ4MDk2OQ==,9599,simonw,2020-08-21T20:33:45Z,2020-08-21T20:33:45Z,OWNER,"I think this should initialize SpatiaLite against the current database if it has not been initialized already.

Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126

```python
def init_spatialite(db, lib):
    db.conn.enable_load_extension(True)
    db.conn.load_extension(lib)
    # Initialize SpatiaLite if not yet initialized
    if ""spatial_ref_sys"" in db.table_names():
        return
    db.conn.execute(""select InitSpatialMetadata(1)"")


def ensure_table_has_geometry(db, table, table_srid):
    if ""geometry"" not in db[table].columns_dict:
        db.conn.execute(
            ""SELECT AddGeometryColumn(?, 'geometry', ?, 'GEOMETRY', 2);"",
            [table, table_srid],
        )
```
Not sure if I should add a utility function or CLI command for that `ensure_table_has_geometry` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642,--load-extension=spatialite shortcut option,
https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678479741,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678479741,MDEyOklzc3VlQ29tbWVudDY3ODQ3OTc0MQ==,9599,simonw,2020-08-21T20:30:37Z,2020-08-21T20:30:37Z,OWNER,Docs: https://github.com/simonw/sqlite-utils/blob/bf4c6b7c82fab6b2400e48424f8dac1ae2f0a2dc/docs/python-api.rst#finding-spatialite,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434,Code for finding SpatiaLite in the usual locations,
https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678476842,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678476842,MDEyOklzc3VlQ29tbWVudDY3ODQ3Njg0Mg==,9599,simonw,2020-08-21T20:23:13Z,2020-08-21T20:23:13Z,OWNER,I'm going to start with just the first two - I'm not convinced I understand the `.so.5` variants.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434,Code for finding SpatiaLite in the usual locations,
https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678476338,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678476338,MDEyOklzc3VlQ29tbWVudDY3ODQ3NjMzOA==,9599,simonw,2020-08-21T20:22:02Z,2020-08-21T20:22:02Z,OWNER,I think that adds it as `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172,--load-extension option for sqlite-utils query,
https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678475578,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678475578,MDEyOklzc3VlQ29tbWVudDY3ODQ3NTU3OA==,9599,simonw,2020-08-21T20:20:05Z,2020-08-21T20:20:05Z,OWNER,"https://github.com/simonw/cryptozoology/blob/2ad69168f3b78ebd90a2cbeea8136c9115e2a9b7/build_cryptids_database.py#L16-L22
```python
try_these = (
    ""mod_spatialite"",
    ""/usr/local/lib/mod_spatialite.dylib"",
    ""/usr/lib/x86_64-linux-gnu/mod_spatialite.so"",
    ""/usr/lib/x86_64-linux-gnu/libspatialite.so.5"",
    ""/usr/lib/x86_64-linux-gnu/libspatialite.so.7"",
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434,Code for finding SpatiaLite in the usual locations,
https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474928,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678474928,MDEyOklzc3VlQ29tbWVudDY3ODQ3NDkyOA==,9599,simonw,2020-08-21T20:18:33Z,2020-08-21T20:18:33Z,OWNER,"This should get me SpatiaLite in the GitHub Actions Ubuntu:
```
apt install libsqlite3-mod-spatialite
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172,--load-extension option for sqlite-utils query,
https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474018,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678474018,MDEyOklzc3VlQ29tbWVudDY3ODQ3NDAxOA==,9599,simonw,2020-08-21T20:16:20Z,2020-08-21T20:16:20Z,OWNER,"Trickiest part of this is how to write a test for it.

I'll do a `pytest.skipIf` that only executes the test if SpatiaLite is available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172,--load-extension option for sqlite-utils query,
https://github.com/simonw/datasette/issues/945#issuecomment-676556377,https://api.github.com/repos/simonw/datasette/issues/945,676556377,MDEyOklzc3VlQ29tbWVudDY3NjU1NjM3Nw==,9599,simonw,2020-08-19T17:21:16Z,2020-08-19T17:21:16Z,OWNER,Documented here: https://docs.datasette.io/en/latest/plugins.html#installing-plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",682005535,datasette install -U for upgrading packages,
https://github.com/simonw/datasette/issues/943#issuecomment-675889865,https://api.github.com/repos/simonw/datasette/issues/943,675889865,MDEyOklzc3VlQ29tbWVudDY3NTg4OTg2NQ==,9599,simonw,2020-08-19T06:57:00Z,2020-08-19T06:57:00Z,OWNER,Maybe `.get` vs `.get_html`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675889551,https://api.github.com/repos/simonw/datasette/issues/943,675889551,MDEyOklzc3VlQ29tbWVudDY3NTg4OTU1MQ==,9599,simonw,2020-08-19T06:56:06Z,2020-08-19T06:56:17Z,OWNER,"I'm leaning towards defaulting to JSON as the requested format - you can pass `format=""html""` if you want HTML.

But weird that it's different from the web UI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675884980,https://api.github.com/repos/simonw/datasette/issues/943,675884980,MDEyOklzc3VlQ29tbWVudDY3NTg4NDk4MA==,9599,simonw,2020-08-19T06:44:26Z,2020-08-19T06:44:26Z,OWNER,"Need to decide what to do about JSON responses.

When called from a template it's likely the intent will be to further loop through the JSON data returned. It would be annoying to have to run `json.loads` here.

Maybe a `.get_json()` method then? Or even return a response that has `.json()` and `.text` similar to `httpx` - or just return an `httpx` response.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/944#issuecomment-675830678,https://api.github.com/repos/simonw/datasette/issues/944,675830678,MDEyOklzc3VlQ29tbWVudDY3NTgzMDY3OA==,9599,simonw,2020-08-19T03:30:10Z,2020-08-19T03:30:10Z,OWNER,"These templates will need a way to raise a 404 - so that if the template itself is deciding if the page exists (for example using `datasette-template-sql` or the proposed `datasette.get()` method from #943 or the `graphql()` template function in https://github.com/simonw/datasette-graphql/issues/50) it can return a regular 404 page.

This can imitate the `custom_redirect()` function from https://docs.datasette.io/en/stable/custom_templates.html#custom-redirects:
```html+jinja
{{ custom_redirect(""https://github.com/simonw/datasette"", 301) }}
```
It could be as simple as this:
```
{{ raise_404(""Museum not found"") }}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/944#issuecomment-675829942,https://api.github.com/repos/simonw/datasette/issues/944,675829942,MDEyOklzc3VlQ29tbWVudDY3NTgyOTk0Mg==,9599,simonw,2020-08-19T03:27:25Z,2020-08-19T03:27:25Z,OWNER,"I created a template file called `templates/pages/museums/{slug}.html` and used the debugger to see if Jinja could see it. This worked:
```
(Pdb) self.ds.jinja_env.list_templates()
['500.html', '_codemirror.html', '_codemirror_foot.html', '_description_source_license.html', '_footer.html',
'_table.html', 'allow_debug.html', 'base.html', 'database.html', 'default:500.html', 'default:_codemirror.html',
'default:_codemirror_foot.html', 'default:_description_source_license.html', 'default:_footer.html',
'default:_table.html', 'default:allow_debug.html', 'default:base.html', 'default:database.html',
'default:index.html', 'default:logout.html', 'default:messages_debug.html', 'default:patterns.html',
'default:permissions_debug.html', 'default:query.html', 'default:row.html', 'default:show_json.html',
'default:table.html', 'forbidden.html', 'index.html', 'logout.html', 'messages_debug.html',
'pages/about.html', 'pages/museums/{slug}.html', 'patterns.html', 'permissions_debug.html',
'query.html', 'row.html', 'show_json.html', 'table.html']
```
The `pages/museums/{slug}.html` template is in that list.

Here's the implementation of that `list_templates()` method - it does some filesystem walking so it may be a bit expensive to run it on every request: https://github.com/pallets/jinja/blob/ca8b0b0287e320fe1f4a74f36910ef7ae3303d99/src/jinja2/loaders.py#L197-L212

But caching it would be pretty easy - either until the server is restarted or as an in-memory cache for a few seconds.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976,Path parameters for custom pages,
https://github.com/simonw/datasette/issues/943#issuecomment-675788203,https://api.github.com/repos/simonw/datasette/issues/943,675788203,MDEyOklzc3VlQ29tbWVudDY3NTc4ODIwMw==,9599,simonw,2020-08-19T00:46:08Z,2020-08-19T00:46:23Z,OWNER,Also fun: the inevitable plugin that exposes this to the template language - so Datasette templates can stitch together data from multiple other internal API calls. Fun way to take advantage of `async` support in Jinja.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675787416,https://api.github.com/repos/simonw/datasette/issues/943,675787416,MDEyOklzc3VlQ29tbWVudDY3NTc4NzQxNg==,9599,simonw,2020-08-19T00:42:38Z,2020-08-19T00:42:38Z,OWNER,"I just realised that this mechanism is kind of like being able to use microservices - make API calls within your application - except that everything runs in the same process against SQLite databases so calls will be _lightning fast_.

It also means that a plugin can add a new internal API to Datasette that's accessible to other plugins by registering a new route with `register_routes`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675753114,https://api.github.com/repos/simonw/datasette/issues/943,675753114,MDEyOklzc3VlQ29tbWVudDY3NTc1MzExNA==,9599,simonw,2020-08-18T22:34:55Z,2020-08-18T22:34:55Z,OWNER,"Maybe allow this:

    response = await datasette.get(""/{database}/{table}.json"", database=database, table=table)

This could cause problems if users ever need to pass literal `{` in their paths. Maybe allow this too:

    response = await datasette.get(""/{database}/{table}.json"", interpolate=False)

Not convinced this is useful - it's a bit unintuitive.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675752436,https://api.github.com/repos/simonw/datasette/issues/943,675752436,MDEyOklzc3VlQ29tbWVudDY3NTc1MjQzNg==,9599,simonw,2020-08-18T22:32:44Z,2020-08-18T22:32:44Z,OWNER,"One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted.

If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values.

Here's how the `row.html` table handles that right now: https://github.com/simonw/datasette/blob/b21ed237ab940768574c834aa5a7130724bd3a2d/datasette/templates/row.html#L19-L23

It would be an improvement to have this logic abstracted out somewhere and documented so plugins can use it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675751719,https://api.github.com/repos/simonw/datasette/issues/943,675751719,MDEyOklzc3VlQ29tbWVudDY3NTc1MTcxOQ==,9599,simonw,2020-08-18T22:30:27Z,2020-08-18T22:30:27Z,OWNER,"Right now calling `datasette.app()` instantiates an ASGI application - complete with a bunch of routes and wrappers - and returns that application object. Calling it twice instantiates another ASGI application.

I think a single `Datasette` instance should only ever create a single ASGI app - so the `.app()` method should cache the ASGI app that it returns the first time and return the same application again on future calls.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/915#issuecomment-675751136,https://api.github.com/repos/simonw/datasette/issues/915,675751136,MDEyOklzc3VlQ29tbWVudDY3NTc1MTEzNg==,9599,simonw,2020-08-18T22:28:36Z,2020-08-18T22:28:36Z,OWNER,I'm closing this in favour of an internal requests mechanism in #943.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671763164,Refactor TableView class so things like datasette-graphql can reuse the logic,
https://github.com/simonw/datasette/issues/943#issuecomment-675750845,https://api.github.com/repos/simonw/datasette/issues/943,675750845,MDEyOklzc3VlQ29tbWVudDY3NTc1MDg0NQ==,9599,simonw,2020-08-18T22:27:43Z,2020-08-18T22:27:43Z,OWNER,"What about authentication checks etc? Won't they run twice?

I think that's OK too, in fact it's desirable: think of the case of `datasette-graphql` where a bunch of different TableView calls are being made as part of the same GraphQL queries. Having those calls take advantage of finely grained per-table authentication and permission checks seems like a good feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675750382,https://api.github.com/repos/simonw/datasette/issues/943,675750382,MDEyOklzc3VlQ29tbWVudDY3NTc1MDM4Mg==,9599,simonw,2020-08-18T22:26:15Z,2020-08-18T22:26:15Z,OWNER,"Should internal requests executed in this way be handled by plugins that used the `asgi_wrapper()` hook?

Hard to be sure one way or the other. I'm worried about logging middleware triggering twice - but actually anyone doing serious logging of their Datasette instance is probably doing it in a different layer (uvicorn logs or nginx proxy or whatever) so they wouldn't be affected. There aren't any ASGI logging middlewares out there that I've seen.

Also: if you run into a situation where your stuff is breaking because `datasette.get()` is calling ASGI middleware twice you can fix it by running your ASGI middleware outside of the `asgi_wrapper` plugin hook mechanism.

So I think it DOES execute `asgi_wrapper()` middleware.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675749319,https://api.github.com/repos/simonw/datasette/issues/943,675749319,MDEyOklzc3VlQ29tbWVudDY3NTc0OTMxOQ==,9599,simonw,2020-08-18T22:23:01Z,2020-08-18T22:23:01Z,OWNER,"Actually no - `requests.get()` and `httpx.get()` prove that having a `.get()` method for an HTTP-related API isn't confusing to people at all.

`datasette.get()` it is.

(I'll probably add `datasette.post()` in the future too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675749076,https://api.github.com/repos/simonw/datasette/issues/943,675749076,MDEyOklzc3VlQ29tbWVudDY3NTc0OTA3Ng==,9599,simonw,2020-08-18T22:22:21Z,2020-08-18T22:22:21Z,OWNER,"Alternative name possibilities:

- `datasette.http_get(...)` - slightly misleading since it's not going over the HTTP protocol
- `datasette.internal_get(...)` - the `internal_` might suggest its not an API for external use, which isn't true - it's for plugins
- `datasette.get(...)` - clashes with `dict.get()` but I'm not at all sure that's a good reason not to use it","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675748573,https://api.github.com/repos/simonw/datasette/issues/943,675748573,MDEyOklzc3VlQ29tbWVudDY3NTc0ODU3Mw==,9599,simonw,2020-08-18T22:20:52Z,2020-08-18T22:20:52Z,OWNER,"Should it default to treating things as if they had the `.json` extension? There are use-cases for the non-JSON method, such as https://github.com/natbat/tidepools_near_me/commit/ec102c6da5a5d86f17628740d90b6365b671b5e1

I think I'm OK with people having to add `.json` to their internal calls. Maybe they could use `format=""json""`) as an optional parameter which would automatically handle the very weird edge-cases where you need to use `?_format=json` instead of `.json` (due to table names existing with a `.json` suffix).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/943#issuecomment-675747878,https://api.github.com/repos/simonw/datasette/issues/943,675747878,MDEyOklzc3VlQ29tbWVudDY3NTc0Nzg3OA==,9599,simonw,2020-08-18T22:18:46Z,2020-08-18T22:19:12Z,OWNER,"Could be as simple as `response = await datasette.get(""/path/blah"")` - which could also be re-used by the implementation of the `datasette --get /` CLI option introduced in #927.

Bit weird calling it `.get()` since that clashes with Python's dictionary `.get()` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests,
https://github.com/simonw/datasette/issues/915#issuecomment-675746544,https://api.github.com/repos/simonw/datasette/issues/915,675746544,MDEyOklzc3VlQ29tbWVudDY3NTc0NjU0NA==,9599,simonw,2020-08-18T22:14:41Z,2020-08-18T22:14:41Z,OWNER,"I'm actually pretty happy with how `datasette-graphql` works now - maybe the trick here is to redesign the JSON format in #782 such that it can be used as a documented interface by things like `datasette-graphql` and then ensure Datasette has a documented mechanism for dispatching internal requests.

I just did a horrible hack here that simulates an internal request, so supporting them as a feature would definitely make sense: https://github.com/natbat/tidepools_near_me/commit/ec102c6da5a5d86f17628740d90b6365b671b5e1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671763164,Refactor TableView class so things like datasette-graphql can reuse the logic,
https://github.com/simonw/datasette/issues/268#issuecomment-675725464,https://api.github.com/repos/simonw/datasette/issues/268,675725464,MDEyOklzc3VlQ29tbWVudDY3NTcyNTQ2NA==,9599,simonw,2020-08-18T21:18:07Z,2020-08-18T21:18:35Z,OWNER,"I want this on the table page - but that means that the table page will need to run a slightly more complex query since it needs access to a `rank` column to sort by - which it gets from running a join.

BUT... that join needs to be constructed in a way that keeps existing filters, `?_where=` clauses etc intact.

Here's a prototype using SQLite CTEs: https://register-of-members-interests.datasettes.com/regmem?sql=with+original+as+%28select+rowid%2C+*+from+items%29%0D%0Aselect%0D%0A++original.*%2C%0D%0A++items_fts.rank+as+items_fts_rank%0D%0Afrom%0D%0A++original+join+items_fts+on+original.rowid+%3D+items_fts.rowid%0D%0Awhere%0D%0A++items_fts+match+escape_fts%28%3Asearch%29%0D%0Aorder+by+items_fts_rank+desc+limit+10&search=hotel

```sql
with original as (
  select
    rowid,
    *
  from
    items
)
select
  original.*,
  items_fts.rank as items_fts_rank
from
  original
  join items_fts on original.rowid = items_fts.rowid
where
  items_fts match escape_fts(:search)
order by
  items_fts_rank desc
limit
  10
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/942#issuecomment-675720040,https://api.github.com/repos/simonw/datasette/issues/942,675720040,MDEyOklzc3VlQ29tbWVudDY3NTcyMDA0MA==,9599,simonw,2020-08-18T21:05:24Z,2020-08-18T21:05:24Z,OWNER,"Is `columns` the right key for this in the table metadata block? I might want to use that for initial values for `?_col=` in #615.

Alternative names:

- `column_descriptions`
- `column_info`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-675718593,https://api.github.com/repos/simonw/datasette/issues/942,675718593,MDEyOklzc3VlQ29tbWVudDY3NTcxODU5Mw==,9599,simonw,2020-08-18T21:02:11Z,2020-08-18T21:02:24Z,OWNER,"Easiest solution: if you provide column metadata it gets displayed above the table, something like on https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act

<img width=""500"" alt=""fivethirtyeight__antiquities-act_actions_under_antiquities_act__344_rows"" src=""https://user-images.githubusercontent.com/9599/90565187-57d3e700-e15b-11ea-89c8-0270e3040a50.png"">

HTML `title=` tooltips are also added to the table headers, which won't be visible on touch devices but that's OK because the information is visible on the page already.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-675715472,https://api.github.com/repos/simonw/datasette/issues/942,675715472,MDEyOklzc3VlQ29tbWVudDY3NTcxNTQ3Mg==,9599,simonw,2020-08-18T20:55:02Z,2020-08-18T20:55:02Z,OWNER,"Could display these as tooltips on icons something like this (from the experimental `datasette-inspect-columns` plugin):

<img width=""500"" alt=""fixtures__facetable__15_rows_and_NOAA_tides_second_attempt_-_Jupyter_Notebook"" src=""https://user-images.githubusercontent.com/9599/90564416-37eff380-e15a-11ea-95e5-b556e9039a2a.png"">

This would need to take accessibility into account, and would need a different display for the mobile web layout. Need to consider how it will interact with the column menu suggested in #690.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/873#issuecomment-675610275,https://api.github.com/repos/simonw/datasette/issues/873,675610275,MDEyOklzc3VlQ29tbWVudDY3NTYxMDI3NQ==,9599,simonw,2020-08-18T17:24:05Z,2020-08-18T17:26:10Z,OWNER,"Maybe I can do this with ASGI after all. Here's the output of `/-/asgi-scope` with `datasette-debug-asgi` installed:
```
{'asgi': {'spec_version': '2.1', 'version': '3.0'},
 'client': ('127.0.0.1', 62035),
 'headers': [(b'host', b'127.0.0.1:62029'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:79.0) Gecko'
              b'/20100101 Firefox/79.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'accept-encoding', b'gzip, deflate'),
             (b'dnt', b'1'),
             (b'connection', b'keep-alive'),
             (b'upgrade-insecure-requests', b'1'),
             (b'cache-control', b'max-age=0')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope',
 'root_path': '',
 'scheme': 'http',
 'server': ('127.0.0.1', 62029),
 'type': 'http'}
```
That `'server': ('127.0.0.1', 62029)` bit has the correct port. Question is, can I access that programmatically on server startup?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-675609109,https://api.github.com/repos/simonw/datasette/issues/873,675609109,MDEyOklzc3VlQ29tbWVudDY3NTYwOTEwOQ==,9599,simonw,2020-08-18T17:21:51Z,2020-08-18T17:21:51Z,OWNER,Asked about this on the encode gitter here: https://gitter.im/encode/community?at=5f3c0dcaa8c17801765940c0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/940#issuecomment-675538586,https://api.github.com/repos/simonw/datasette/issues/940,675538586,MDEyOklzc3VlQ29tbWVudDY3NTUzODU4Ng==,9599,simonw,2020-08-18T15:11:36Z,2020-08-18T15:11:36Z,OWNER,I tested this new publish pattern (running the tests in parallel before the deploy step) on `github-to-sqlite` - skipping the Docker step - and it worked: https://github.com/dogsheep/github-to-sqlite/actions/runs/213809864,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-675253373,https://api.github.com/repos/simonw/datasette/issues/940,675253373,MDEyOklzc3VlQ29tbWVudDY3NTI1MzM3Mw==,9599,simonw,2020-08-18T05:10:17Z,2020-08-18T05:10:17Z,OWNER,I'll close this after the next release successfully goes out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-675251613,https://api.github.com/repos/simonw/datasette/issues/940,675251613,MDEyOklzc3VlQ29tbWVudDY3NTI1MTYxMw==,9599,simonw,2020-08-18T05:05:15Z,2020-08-18T05:05:15Z,OWNER,I think this is ready. I'll only know for sure the first time I push a release through it though!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-674590583,https://api.github.com/repos/simonw/datasette/issues/940,674590583,MDEyOklzc3VlQ29tbWVudDY3NDU5MDU4Mw==,9599,simonw,2020-08-16T23:15:51Z,2020-08-18T05:04:43Z,OWNER,This example of jobs depending on each other and sharing data via artifacts looks relevant: https://docs.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts#passing-data-between-jobs-in-a-workflow,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-675250280,https://api.github.com/repos/simonw/datasette/issues/940,675250280,MDEyOklzc3VlQ29tbWVudDY3NTI1MDI4MA==,9599,simonw,2020-08-18T05:01:34Z,2020-08-18T05:01:42Z,OWNER,I think `${GITHUB_REF#refs/tags/}` is the equivalent of `$TRAVIS_TAG`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-674589472,https://api.github.com/repos/simonw/datasette/issues/940,674589472,MDEyOklzc3VlQ29tbWVudDY3NDU4OTQ3Mg==,9599,simonw,2020-08-16T23:05:57Z,2020-08-16T23:05:57Z,OWNER,When I figure this out I'll update the https://github.com/simonw/datasette-plugin/blob/main/datasette-%7B%7Bcookiecutter.hyphenated%7D%7D/.github/workflows/publish.yml default workflow to do this - right now it runs the tests once on just a single version of Python as part of the package deploy to PyPI step.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-674589321,https://api.github.com/repos/simonw/datasette/issues/940,674589321,MDEyOklzc3VlQ29tbWVudDY3NDU4OTMyMQ==,9599,simonw,2020-08-16T23:04:34Z,2020-08-16T23:04:34Z,OWNER,"https://docs.github.com/en/actions/getting-started-with-github-actions/core-concepts-for-github-actions#job

> A set of steps that execute on the same runner. You can define the dependency rules for how jobs run in a workflow file. Jobs can run at the same time in parallel or run sequentially depending on the status of a previous job. For example, a workflow can have two sequential jobs that build and test code, where the test job is dependent on the status of the build job. If the build job fails, the test job will not run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/940#issuecomment-674589035,https://api.github.com/repos/simonw/datasette/issues/940,674589035,MDEyOklzc3VlQ29tbWVudDY3NDU4OTAzNQ==,9599,simonw,2020-08-16T23:02:23Z,2020-08-16T23:02:23Z,OWNER,"I'd like to set these up as different workflows that depend on each other, if that's possible.

I want to start three test runs in parallel (on three different Python versions), then if all three pass kick off the PyPI push (without running more tests), then if that passes do the Docker build and push.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/914#issuecomment-674578388,https://api.github.com/repos/simonw/datasette/issues/914,674578388,MDEyOklzc3VlQ29tbWVudDY3NDU3ODM4OA==,9599,simonw,2020-08-16T21:10:27Z,2020-08-16T21:10:27Z,OWNER,"Demo of the fix: https://latest.datasette.io/fixtures/binary_data.json?_sort_desc=data&_shape=array&_nl=on

```
{""rowid"": 2, ""data"": {""$base64"": true, ""encoded"": ""FRwDx60F/g==""}}
{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671056788,"""Object of type bytes is not JSON serializable"" for _nl=on",
https://github.com/simonw/datasette/issues/940#issuecomment-674566618,https://api.github.com/repos/simonw/datasette/issues/940,674566618,MDEyOklzc3VlQ29tbWVudDY3NDU2NjYxOA==,9599,simonw,2020-08-16T19:20:58Z,2020-08-16T19:20:58Z,OWNER,I need to figure out how to build and push the Docker image on releases. Here's the Travis code for that: https://github.com/simonw/datasette/blob/52eabb019d4051084b21524bd0fd9c2731126985/.travis.yml#L38-L47,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679808124,Move CI to GitHub Issues,
https://github.com/simonw/datasette/issues/938#issuecomment-674558631,https://api.github.com/repos/simonw/datasette/issues/938,674558631,MDEyOklzc3VlQ29tbWVudDY3NDU1ODYzMQ==,9599,simonw,2020-08-16T18:10:23Z,2020-08-16T18:10:23Z,OWNER,Documentation: https://github.com/simonw/datasette/blob/3a4c8ed36aa97211e46849d32a09f2f386f342dd/docs/plugin_hooks.rst#extra-template-vars-template-database-table-columns-view-name-request-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679700269,Pass columns to extra CSS/JS/etc plugin hooks,
https://github.com/simonw/datasette/issues/938#issuecomment-674551826,https://api.github.com/repos/simonw/datasette/issues/938,674551826,MDEyOklzc3VlQ29tbWVudDY3NDU1MTgyNg==,9599,simonw,2020-08-16T17:07:57Z,2020-08-16T17:07:57Z,OWNER,extra_tenplate_vars should he documented first and should be the only one to document the arguments and the async return feature. Others should refer back to it rather than duplicating that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679700269,Pass columns to extra CSS/JS/etc plugin hooks,
https://github.com/simonw/datasette/issues/939#issuecomment-674548163,https://api.github.com/repos/simonw/datasette/issues/939,674548163,MDEyOklzc3VlQ29tbWVudDY3NDU0ODE2Mw==,9599,simonw,2020-08-16T16:34:30Z,2020-08-16T16:34:30Z,OWNER,Docs also need to note that `request` may be `None` for all of these hooks.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674547811,https://api.github.com/repos/simonw/datasette/issues/939,674547811,MDEyOklzc3VlQ29tbWVudDY3NDU0NzgxMQ==,9599,simonw,2020-08-16T16:31:20Z,2020-08-16T16:31:20Z,OWNER,And the docs need to be updated too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674547788,https://api.github.com/repos/simonw/datasette/issues/939,674547788,MDEyOklzc3VlQ29tbWVudDY3NDU0Nzc4OA==,9599,simonw,2020-08-16T16:31:08Z,2020-08-16T16:31:08Z,OWNER,Also: they should all be able to return `async def` inner functions. At the moment only `extra_template_vars` supports that pattern.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674545058,https://api.github.com/repos/simonw/datasette/issues/939,674545058,MDEyOklzc3VlQ29tbWVudDY3NDU0NTA1OA==,9599,simonw,2020-08-16T16:07:20Z,2020-08-16T16:07:20Z,OWNER,"I'm going to implement this first as a single commit, then implement `columns` from #938 as a separate change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674544973,https://api.github.com/repos/simonw/datasette/issues/939,674544973,MDEyOklzc3VlQ29tbWVudDY3NDU0NDk3Mw==,9599,simonw,2020-08-16T16:06:34Z,2020-08-16T16:06:34Z,OWNER,"They should also be next to each other in the documentation - right now they're a bit scattered in terms of page order:

<img width=""598"" alt=""Plugin_hooks_—_Datasette_documentation"" src=""https://user-images.githubusercontent.com/9599/90338620-c5d3af00-df9f-11ea-908b-3f888947dd03.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/939#issuecomment-674544875,https://api.github.com/repos/simonw/datasette/issues/939,674544875,MDEyOklzc3VlQ29tbWVudDY3NDU0NDg3NQ==,9599,simonw,2020-08-16T16:05:36Z,2020-08-16T16:05:36Z,OWNER,"They should all take:

    template, database, table, view_name, request, datasette, columns

`columns` is new, see #938.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679779797,extra_ plugin hooks should take the same arguments,
https://github.com/simonw/datasette/issues/938#issuecomment-674544691,https://api.github.com/repos/simonw/datasette/issues/938,674544691,MDEyOklzc3VlQ29tbWVudDY3NDU0NDY5MQ==,9599,simonw,2020-08-16T16:03:52Z,2020-08-16T16:03:52Z,OWNER,"Four plugin hooks need this extra `columns` argument:

- `extra_css_urls(template, database, table, datasette)`
- `extra_js_urls(template, database, table, datasette)`
- `extra_body_script(template, database, table, view_name, datasette)`
- `extra_template_vars(template, database, table, view_name, request, datasette)`

It's weird that these take different arguments. I should fix that too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679700269,Pass columns to extra CSS/JS/etc plugin hooks,
https://github.com/simonw/datasette/pull/936#issuecomment-674453318,https://api.github.com/repos/simonw/datasette/issues/936,674453318,MDEyOklzc3VlQ29tbWVudDY3NDQ1MzMxOA==,9599,simonw,2020-08-15T22:29:15Z,2020-08-15T22:32:58Z,OWNER,"Test failure:
```
tests/test_canned_queries.py F
>>>  captured stdout >>> 
__enter__
>>>  traceback >>> 

canned_write_client = <datasette.utils.testing.TestClient object at 0x108ec5040>

    def test_insert(canned_write_client):
        response = canned_write_client.post(
            ""/data/add_name"",
            {""name"": ""Hello""},
            allow_redirects=False,
            csrftoken_from=True,
            cookies={""foo"": ""bar""},
        )
        assert 302 == response.status
>       assert ""/data/add_name?success"" == response.headers[""Location""]
E       AssertionError: assert '/data/add_name?success' == '/data/add_name'
E         - /data/add_name
E         + /data/add_name?success
E         ?               ++++++++

/Users/simon/Dropbox/Development/datasette/tests/test_canned_queries.py:66: AssertionError
```
No idea why this change would affect that test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679650632,Don't hang in db.execute_write_fn() if connection fails,
https://github.com/simonw/datasette/issues/935#issuecomment-674451012,https://api.github.com/repos/simonw/datasette/issues/935,674451012,MDEyOklzc3VlQ29tbWVudDY3NDQ1MTAxMg==,9599,simonw,2020-08-15T21:56:13Z,2020-08-15T21:56:13Z,OWNER,"This implementation seems to fix it, need to work out how to test though.
```diff
diff --git a/datasette/database.py b/datasette/database.py
index ffa7a79..7ba1456 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -89,14 +89,22 @@ class Database:
     def _execute_writes(self):
         # Infinite looping thread that protects the single write connection
         # to this database
-        conn = self.connect(write=True)
+        conn_exception = None
+        conn = None
+        try:
+            conn = self.connect(write=True)
+        except Exception as e:
+            conn_exception = e
         while True:
             task = self._write_queue.get()
-            try:
-                result = task.fn(conn)
-            except Exception as e:
-                print(e)
-                result = e
+            if conn_exception is not None:
+                result = conn_exception
+            else:
+                try:
+                    result = task.fn(conn)
+                except Exception as e:
+                    print(e)
+                    result = e
             task.reply_queue.sync_q.put(result)
 
     async def execute_fn(self, fn):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710,"db.execute_write_fn(create_tables, block=True) hangs a thread if connection fails",
https://github.com/simonw/datasette/issues/935#issuecomment-674450652,https://api.github.com/repos/simonw/datasette/issues/935,674450652,MDEyOklzc3VlQ29tbWVudDY3NDQ1MDY1Mg==,9599,simonw,2020-08-15T21:51:22Z,2020-08-15T21:51:22Z,OWNER,"The easiest way to recreate this is to attempt a write against an immutable database, which triggers this assertion error:

https://github.com/simonw/datasette/blob/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8/datasette/database.py#L47-L55","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710,"db.execute_write_fn(create_tables, block=True) hangs a thread if connection fails",
https://github.com/simonw/datasette/issues/935#issuecomment-674450607,https://api.github.com/repos/simonw/datasette/issues/935,674450607,MDEyOklzc3VlQ29tbWVudDY3NDQ1MDYwNw==,9599,simonw,2020-08-15T21:50:41Z,2020-08-15T21:50:41Z,OWNER,"The bug is here:

https://github.com/simonw/datasette/blob/13b3b51087964d5e1a8c1cdd2495e07bdbe176b8/datasette/database.py#L89-L100

If `conn = self.connect(write=True)` raises an exception the entire server hangs, like this:
```
% datasette -i fixtures.db  --get /            
Exception in thread Thread-1:
Traceback (most recent call last):
  File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py"", line 932, in _bootstrap_inner
    self.run()
  File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py"", line 870, in run
    self._target(*self._args, **self._kwargs)
  File ""/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py"", line 92, in _execute_writes
    conn = self.connect(write=True)
  File ""/Users/simon/.local/share/virtualenvs/latest-datasette-with-all-plugins-PJL_Xy9e/lib/python3.8/site-packages/datasette/database.py"", line 55, in connect
    assert not (write and not self.is_mutable)
AssertionError

... server hangs here ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",679646710,"db.execute_write_fn(create_tables, block=True) hangs a thread if connection fails",
https://github.com/simonw/datasette/issues/932#issuecomment-674144798,https://api.github.com/repos/simonw/datasette/issues/932,674144798,MDEyOklzc3VlQ29tbWVudDY3NDE0NDc5OA==,9599,simonw,2020-08-14T16:02:24Z,2020-08-14T16:02:24Z,OWNER,"Things to go in here:

- What is Datasette?
- A *database* contains *tables* full of *records*. A table has *rows* and *columns*.
- Understanding faceting
- How to use the filter interface
- How to export data
- How to link to data
- How to run SQL","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation,
https://github.com/simonw/datasette/issues/932#issuecomment-673735299,https://api.github.com/repos/simonw/datasette/issues/932,673735299,MDEyOklzc3VlQ29tbWVudDY3MzczNTI5OQ==,9599,simonw,2020-08-13T22:10:40Z,2020-08-13T22:11:06Z,OWNER,"Idea: plugins can provide their own user-facing documentation. Datasette can like to eg `datasette.io/help?plugins=datasette-vega,datasette-cluster-map` to get the user manual with extra sections for those plugins.

Or... link to `?url=datasette-url` and the documentation site can hit `/-/plugins.json` to figure out what extra manual sections to display!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation,
https://github.com/simonw/datasette/issues/932#issuecomment-673734387,https://api.github.com/repos/simonw/datasette/issues/932,673734387,MDEyOklzc3VlQ29tbWVudDY3MzczNDM4Nw==,9599,simonw,2020-08-13T22:08:06Z,2020-08-13T22:08:06Z,OWNER,One challenge: how does this interact with plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation,
https://github.com/simonw/datasette/issues/932#issuecomment-673733904,https://api.github.com/repos/simonw/datasette/issues/932,673733904,MDEyOklzc3VlQ29tbWVudDY3MzczMzkwNA==,9599,simonw,2020-08-13T22:06:50Z,2020-08-13T22:06:50Z,OWNER,Title: **Using Datasette**. `using.rst`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",678760988,End-user documentation,
https://github.com/simonw/datasette/issues/931#issuecomment-673123213,https://api.github.com/repos/simonw/datasette/issues/931,673123213,MDEyOklzc3VlQ29tbWVudDY3MzEyMzIxMw==,9599,simonw,2020-08-12T21:36:20Z,2020-08-12T21:36:20Z,OWNER,That worked: https://hub.docker.com/r/datasetteproject/datasette/tags,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673104851,https://api.github.com/repos/simonw/datasette/issues/931,673104851,MDEyOklzc3VlQ29tbWVudDY3MzEwNDg1MQ==,9599,simonw,2020-08-12T20:52:08Z,2020-08-12T20:52:08Z,OWNER,"```
docker run -p 8001:8001 -v `pwd`:/mnt \                     
  datasette-updated-spatialite \
  datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db --load-extension=/usr/local/lib/mod_spatialite.so
```
This seems to work `/-/versions` reports the SpatiaLite versions as expected:
```
        ""extensions"": {
            ""json1"": null,
            ""spatialite"": ""4.4.0-RC0""
        },
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673088110,https://api.github.com/repos/simonw/datasette/issues/931,673088110,MDEyOklzc3VlQ29tbWVudDY3MzA4ODExMA==,9599,simonw,2020-08-12T20:15:28Z,2020-08-12T20:15:28Z,OWNER,"I changed the Dockerfile and built it on my laptop using:

    docker build -f Dockerfile -t datasette-updated-spatialite .
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673074297,https://api.github.com/repos/simonw/datasette/issues/931,673074297,MDEyOklzc3VlQ29tbWVudDY3MzA3NDI5Nw==,9599,simonw,2020-08-12T19:46:29Z,2020-08-12T19:46:29Z,OWNER,"Looks like the old files have moved to:

- http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/
- http://www.gaia-gis.it/gaia-sins/libspatialite-sources/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673070308,https://api.github.com/repos/simonw/datasette/issues/931,673070308,MDEyOklzc3VlQ29tbWVudDY3MzA3MDMwOA==,9599,simonw,2020-08-12T19:37:55Z,2020-08-12T19:37:55Z,OWNER,"Project news here: https://groups.google.com/g/spatialite-users/c/8AG3hQzXDmo

> FreeXL
> ==============================
> the current stable version is now 1.0.6
> (supporting few minor bug fixes and refreshed
> build scripts)
> DONE: no further development is expected in
> the near future.
> 
> http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz
> http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.zip
> 
> 
> ReadOSM
> ===============================
> the current stable version is now 1.1.0a
> (supporting few minor bug fixes and refreshed
> build scripts)
> DONE: no further development is expected in
> the near future.
> 
> http://www.gaia-gis.it/gaia-sins/readosm-1.1.0a.tar.gz
> http://www.gaia-gis.it/gaia-sins/readosm-1.1.0a.zip
> 
> 
> VirtualPG
> ===============================
> the current stanle version is now 2.0.0
> (supporting few minor bug fixes and refreshed
> build scripts)
> DONE: no further development is expected in
> the near future.
> 
> http://www.gaia-gis.it/gaia-sins/virtualpg-2.0.0.tar.gz
> http://www.gaia-gis.it/gaia-sins/virtualpg-2.0.0.zip
> 
> 
> libspatialite
> ===============================
> 5.0.0 Release Candidate 1 (RC1) is now ready
> to become a stable release.
> I'll simply wait for more or less a week so to
> allow for a reasonable community testing; if
> no critical issue will be reported in the meanwhile
> I'll go ASAP to release 5.0.0 ""stable""
> 
> http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.0-RC1.tar.gz
> http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.0-RC1.zip
> 
> 
> librasterlite2
> ================================
> still under active development.
> a very relevant milestone has been achieved,
> now the DB layout required for fully integrating
> libspatialte and librasterlie2 is definetely 
> consolidated, no further changes are expected.
> 
> http://www.gaia-gis.it/gaia-sins/librasterlite2-sources/librasterlite2-1.1.0-beta1.tar.gz
> http://www.gaia-gis.it/gaia-sins/librasterlite2-sources/librasterlite2-1.1.0-beta1.zip
> 
> more development activity is still required for
> fully implementing SQL-driven graphic rendering 
> of both Vector and Raster Coverages based on
> standard SLD/SE styles.
> 
> a reasonable estimate is two man-month (and
> not necessarily one development month equals
> one calendar month)
> 
> 
> spatialite_gui
> =================================
> the GUI tool is expected to take full profit from
> the advanced features of RasterLite2 so to
> become a complete self-contained GIS viewer 
> with map editing capabilities.
>  
> consequently it wiil surely be the last
> member of the SpatiaLite family to be
> released in a final stable form. 
> several transient development versions
> will be possibly released so to closely
> follow the evolution of RasterLite2.
> 
> http://www.gaia-gis.it/gaia-sins/spatialite-gui-sources/spatialite_gui-2.1.0-beta1.tar.gz
> http://www.gaia-gis.it/gaia-sins/spatialite-gui-sources/spatialite_gui-2.1.0-beta1.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673068919,https://api.github.com/repos/simonw/datasette/issues/931,673068919,MDEyOklzc3VlQ29tbWVudDY3MzA2ODkxOQ==,9599,simonw,2020-08-12T19:34:57Z,2020-08-12T19:34:57Z,OWNER,"Looks like SpatiaLite released new versions: https://www.gaia-gis.it/fossil/freexl/index says ""Sources: current version is 1.0.6 (released on 2020-08-02)"" and links to http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/datasette/issues/931#issuecomment-673068327,https://api.github.com/repos/simonw/datasette/issues/931,673068327,MDEyOklzc3VlQ29tbWVudDY3MzA2ODMyNw==,9599,simonw,2020-08-12T19:33:42Z,2020-08-12T19:33:42Z,OWNER,"https://hub.docker.com/r/datasetteproject/datasette/tags shows this:

<img width=""1282"" alt=""datasetteproject_datasette_Tags_-_Docker_Hub"" src=""https://user-images.githubusercontent.com/9599/90059445-0b3e6680-dc98-11ea-9e21-c473b6e9b55d.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677926613,Docker container is no longer being pushed (it's stuck on 0.45),
https://github.com/simonw/sqlite-utils/issues/133#issuecomment-672997703,https://api.github.com/repos/simonw/sqlite-utils/issues/133,672997703,MDEyOklzc3VlQ29tbWVudDY3Mjk5NzcwMw==,9599,simonw,2020-08-12T17:05:06Z,2020-08-12T17:05:06Z,OWNER,Released.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677839979,Release a sdist to PyPI,
https://github.com/simonw/datasette/issues/930#issuecomment-672550662,https://api.github.com/repos/simonw/datasette/issues/930,672550662,MDEyOklzc3VlQ29tbWVudDY3MjU1MDY2Mg==,9599,simonw,2020-08-12T03:30:59Z,2020-08-12T03:30:59Z,OWNER,https://datasette.readthedocs.io/en/stable/changelog.html#v0-47-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672519787,https://api.github.com/repos/simonw/datasette/issues/930,672519787,MDEyOklzc3VlQ29tbWVudDY3MjUxOTc4Nw==,9599,simonw,2020-08-12T02:52:46Z,2020-08-12T02:52:46Z,OWNER,"Homebrew install is now fixed, using a temporary URL while waiting for Travis to ship the new release to PyPI: https://github.com/simonw/homebrew-datasette/issues/6","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672488293,https://api.github.com/repos/simonw/datasette/issues/930,672488293,MDEyOklzc3VlQ29tbWVudDY3MjQ4ODI5Mw==,9599,simonw,2020-08-12T02:35:34Z,2020-08-12T02:35:34Z,OWNER,"OK, this has fixed it - I used `sdist` to create the `.tar.gz` and then `pip installed` it into a new environment, and everything worked just fine. Going to ship this as Datasette 0.47.1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672480811,https://api.github.com/repos/simonw/datasette/issues/930,672480811,MDEyOklzc3VlQ29tbWVudDY3MjQ4MDgxMQ==,9599,simonw,2020-08-12T02:31:38Z,2020-08-12T02:31:38Z,OWNER,The root cause appears to be that the `templates/` are missing from the `sdist` `.tar.gz`: https://github.com/simonw/homebrew-datasette/issues/6#issuecomment-672477352,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672472518,https://api.github.com/repos/simonw/datasette/issues/930,672472518,MDEyOklzc3VlQ29tbWVudDY3MjQ3MjUxOA==,9599,simonw,2020-08-12T02:24:49Z,2020-08-12T02:24:49Z,OWNER,I checked and `datasette publish` deploys a 0.47 version that works fine too: https://datasette-0-47-j7hipcg4aq-uc.a.run.app/-/versions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/930#issuecomment-672471431,https://api.github.com/repos/simonw/datasette/issues/930,672471431,MDEyOklzc3VlQ29tbWVudDY3MjQ3MTQzMQ==,9599,simonw,2020-08-12T02:21:19Z,2020-08-12T02:21:19Z,OWNER,"Correction: pip installed Datasette works fine, it's just the Homebrew release that is broken.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677326155,Datasette sdist is missing templates (hence broken when installing from Homebrew),
https://github.com/simonw/datasette/issues/926#issuecomment-672393737,https://api.github.com/repos/simonw/datasette/issues/926,672393737,MDEyOklzc3VlQ29tbWVudDY3MjM5MzczNw==,9599,simonw,2020-08-12T00:26:17Z,2020-08-12T00:26:17Z,OWNER,"```
$ datasette --get '/:memory:.json?sql=select+sqlite_version()' | jq .
{
  ""database"": "":memory:"",
  ""query_name"": null,
  ""rows"": [
    [
      ""3.32.3""
    ]
  ],
  ""truncated"": false,
  ""columns"": [
    ""sqlite_version()""
  ],
  ""query"": {
    ""sql"": ""select sqlite_version()"",
    ""params"": {}
  },
  ""private"": false,
  ""allow_execute_sql"": true,
  ""query_ms"": 1.165628433227539
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677250834,"datasette fixtures.db --get ""/fixtures.json""",
https://github.com/simonw/datasette/pull/927#issuecomment-672391299,https://api.github.com/repos/simonw/datasette/issues/927,672391299,MDEyOklzc3VlQ29tbWVudDY3MjM5MTI5OQ==,9599,simonw,2020-08-12T00:19:20Z,2020-08-12T00:19:20Z,OWNER,Docs: https://github.com/simonw/datasette/blob/2111da01a03cfc62303b6a4b59ea9f96d22c0f78/docs/getting_started.rst#datasette---get,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/pull/927#issuecomment-672382108,https://api.github.com/repos/simonw/datasette/issues/927,672382108,MDEyOklzc3VlQ29tbWVudDY3MjM4MjEwOA==,9599,simonw,2020-08-12T00:09:18Z,2020-08-12T00:09:18Z,OWNER,Documentation can go here: https://datasette.readthedocs.io/en/latest/getting_started.html#datasette-serve-options,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/issues/928#issuecomment-672373061,https://api.github.com/repos/simonw/datasette/issues/928,672373061,MDEyOklzc3VlQ29tbWVudDY3MjM3MzA2MQ==,9599,simonw,2020-08-11T23:56:19Z,2020-08-11T23:56:19Z,OWNER,"New implementation of the `install` command:
https://github.com/simonw/datasette/blob/afdeda8216d4d3027f87583ccdbef17ad85022ef/datasette/cli.py#L235-L240","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618,Test failures caused by failed attempts to mock pip,
https://github.com/simonw/datasette/issues/928#issuecomment-672372465,https://api.github.com/repos/simonw/datasette/issues/928,672372465,MDEyOklzc3VlQ29tbWVudDY3MjM3MjQ2NQ==,9599,simonw,2020-08-11T23:54:28Z,2020-08-11T23:54:28Z,OWNER,"While debugging this I found a useful clue in https://github.com/pypa/pip/blob/e060970d51c5946beac8447eb95585d83019582d/src/pip/_internal/cli/main.py#L23-L47

```
# Do not import and use main() directly! Using it directly is actively
# discouraged by pip's maintainers. The name, location and behavior of
# this function is subject to change, so calling it directly is not
# portable across different pip versions.

# In addition, running pip in-process is unsupported and unsafe. This is
# elaborated in detail at
# https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program.
# That document also provides suggestions that should work for nearly
# all users that are considering importing and using main() directly.

# However, we know that certain users will still want to invoke pip
# in-process. If you understand and accept the implications of using pip
# in an unsupported manner, the best approach is to use runpy to avoid
# depending on the exact location of this entry point.

# The following example shows how to use runpy to invoke pip in that
# case:
#
#     sys.argv = [""pip"", your, args, here]
#     runpy.run_module(""pip"", run_name=""__main__"")
#
# Note that this will exit the process after running, unlike a direct
# call to main. As it is not safe to do any processing after calling
# main, this should not be an issue in practice.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618,Test failures caused by failed attempts to mock pip,
https://github.com/simonw/datasette/issues/928#issuecomment-672372197,https://api.github.com/repos/simonw/datasette/issues/928,672372197,MDEyOklzc3VlQ29tbWVudDY3MjM3MjE5Nw==,9599,simonw,2020-08-11T23:53:38Z,2020-08-11T23:53:38Z,OWNER,Caused by the tests for #925,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677272618,Test failures caused by failed attempts to mock pip,
https://github.com/simonw/datasette/pull/927#issuecomment-672357176,https://api.github.com/repos/simonw/datasette/issues/927,672357176,MDEyOklzc3VlQ29tbWVudDY3MjM1NzE3Ng==,9599,simonw,2020-08-11T23:32:08Z,2020-08-11T23:33:09Z,OWNER,Needs documentation and tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/pull/927#issuecomment-672357902,https://api.github.com/repos/simonw/datasette/issues/927,672357902,MDEyOklzc3VlQ29tbWVudDY3MjM1NzkwMg==,9599,simonw,2020-08-11T23:32:39Z,2020-08-11T23:32:39Z,OWNER,"It works:
```
$ datasette --get '/:memory:.json?sql=select * from sqlite_master' | jq .                            
{
  ""database"": "":memory:"",
  ""query_name"": null,
  ""rows"": [],
  ""truncated"": false,
  ""columns"": [
    ""type"",
    ""name"",
    ""tbl_name"",
    ""rootpage"",
    ""sql""
  ],
  ""query"": {
    ""sql"": ""select * from sqlite_master"",
    ""params"": {}
  },
  ""private"": false,
  ""allow_execute_sql"": true,
  ""query_ms"": 0.8032321929931641
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677265716,"'datasette --get' option, refs #926",
https://github.com/simonw/datasette/issues/926#issuecomment-672338113,https://api.github.com/repos/simonw/datasette/issues/926,672338113,MDEyOklzc3VlQ29tbWVudDY3MjMzODExMw==,9599,simonw,2020-08-11T22:57:28Z,2020-08-11T22:57:28Z,OWNER,"I partly want this so I can easily implement a better `test` method for the Homebrew package. The test I have right now looks like this:

https://github.com/simonw/homebrew-datasette/blob/8aa30aa183158051a987a7e3f50e7e3ee05d8ee9/Formula/datasette.rb#L125-L127

```
  test do
    system bin/""datasette"", ""--help""
  end
```
The Homebrew docs at https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula say:

> We want tests that don't require any user input and test the basic functionality of the application. For example `foo build-foo input.foo` is a good test and (despite their widespread use) `foo --version` and `foo --help` are bad tests. However, a bad test is better than no test at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677250834,"datasette fixtures.db --get ""/fixtures.json""",
https://github.com/simonw/datasette/issues/923#issuecomment-672336720,https://api.github.com/repos/simonw/datasette/issues/923,672336720,MDEyOklzc3VlQ29tbWVudDY3MjMzNjcyMA==,9599,simonw,2020-08-11T22:53:07Z,2020-08-11T22:53:07Z,OWNER,https://github.com/simonw/datasette/blob/5126ecb1267ed3850bf3b0ab270accd031a02e79/docs/installation.rst#using-homebrew,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/923#issuecomment-672329101,https://api.github.com/repos/simonw/datasette/issues/923,672329101,MDEyOklzc3VlQ29tbWVudDY3MjMyOTEwMQ==,9599,simonw,2020-08-11T22:35:13Z,2020-08-11T22:35:13Z,OWNER,I added the `datasette install name-of-plugin` command in #925 mainly to simplify the process of installing plugins if Datasette itself was installed using homebrew.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/925#issuecomment-672328807,https://api.github.com/repos/simonw/datasette/issues/925,672328807,MDEyOklzc3VlQ29tbWVudDY3MjMyODgwNw==,9599,simonw,2020-08-11T22:34:37Z,2020-08-11T22:34:37Z,OWNER,"This will simplify the instructions for installing plugins with Datasette install via homebrew, refs #923","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912,"""datasette install"" and ""datasette uninstall"" commands",
https://github.com/simonw/datasette/issues/925#issuecomment-672328436,https://api.github.com/repos/simonw/datasette/issues/925,672328436,MDEyOklzc3VlQ29tbWVudDY3MjMyODQzNg==,9599,simonw,2020-08-11T22:33:32Z,2020-08-11T22:33:42Z,OWNER,"```
$ datasette install --help
Usage: datasette install [OPTIONS] PACKAGES...

  Install Python packages - e.g. Datasette plugins - into the same
  environment as Datasette

Options:
  --help  Show this message and exit.
$ datasette uninstall --help
Usage: datasette uninstall [OPTIONS] PACKAGES...

  Uninstall Python packages (e.g. plugins) from the Datasette environment

Options:
  -y, --yes  Don't ask for confirmation
  --help     Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912,"""datasette install"" and ""datasette uninstall"" commands",
https://github.com/simonw/datasette/issues/925#issuecomment-672304650,https://api.github.com/repos/simonw/datasette/issues/925,672304650,MDEyOklzc3VlQ29tbWVudDY3MjMwNDY1MA==,9599,simonw,2020-08-11T22:04:48Z,2020-08-11T22:04:48Z,OWNER,Prototyped in this thread: https://github.com/simonw/datasette/issues/335#issuecomment-671005731,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677227912,"""datasette install"" and ""datasette uninstall"" commands",
https://github.com/simonw/datasette/issues/923#issuecomment-672288845,https://api.github.com/repos/simonw/datasette/issues/923,672288845,MDEyOklzc3VlQ29tbWVudDY3MjI4ODg0NQ==,9599,simonw,2020-08-11T21:28:17Z,2020-08-11T21:28:17Z,OWNER,"Here's a pattern for installing plugins:
```
$ datasette plugins
[]
$ /usr/local/opt/datasette/libexec/bin/pip install datasette-vega
Collecting datasette-vega
  Using cached datasette_vega-0.6.2-py3-none-any.whl (1.8 MB)
Requirement already satisfied: datasette in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette-vega) (0.46)
Requirement already satisfied: click~=7.1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (7.1.2)
Requirement already satisfied: click-default-group~=1.2.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.2.2)
Requirement already satisfied: Jinja2<2.12.0,>=2.10.3 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (2.11.2)
Requirement already satisfied: hupper~=1.9 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.10.2)
Requirement already satisfied: pint~=0.9 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.14)
Requirement already satisfied: pluggy~=0.13.0 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.13.1)
Requirement already satisfied: uvicorn~=0.11 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.11.8)
Requirement already satisfied: aiofiles<0.6,>=0.4 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.5.0)
Requirement already satisfied: janus<0.6,>=0.4 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.5.0)
Requirement already satisfied: asgi-csrf>=0.6 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (0.6.1)
Requirement already satisfied: PyYAML~=5.3 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (5.3.1)
Requirement already satisfied: mergedeep<1.4.0,>=1.1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.3.0)
Requirement already satisfied: itsdangerous~=1.1 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.1.0)
Requirement already satisfied: python-baseconv==1.2.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from datasette->datasette-vega) (1.2.2)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from Jinja2<2.12.0,>=2.10.3->datasette->datasette-vega) (1.1.1)
Requirement already satisfied: setuptools in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from pint~=0.9->datasette->datasette-vega) (49.3.1)
Requirement already satisfied: packaging in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from pint~=0.9->datasette->datasette-vega) (20.4)
Requirement already satisfied: h11<0.10,>=0.8 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.9.0)
Requirement already satisfied: websockets==8.* in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (8.1)
Requirement already satisfied: httptools==0.1.* in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.1.1)
Requirement already satisfied: uvloop>=0.14.0 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from uvicorn~=0.11->datasette->datasette-vega) (0.14.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from packaging->pint~=0.9->datasette->datasette-vega) (2.4.7)
Requirement already satisfied: six in /usr/local/Cellar/datasette/0.46/libexec/lib/python3.8/site-packages (from packaging->pint~=0.9->datasette->datasette-vega) (1.15.0)
Installing collected packages: datasette-vega
Successfully installed datasette-vega-0.6.2
$ datasette plugins                                              
[
    {
        ""name"": ""datasette-vega"",
        ""static"": true,
        ""templates"": false,
        ""version"": ""0.6.2"",
        ""hooks"": [
            ""extra_css_urls"",
            ""extra_js_urls""
        ]
    }
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/923#issuecomment-672287754,https://api.github.com/repos/simonw/datasette/issues/923,672287754,MDEyOklzc3VlQ29tbWVudDY3MjI4Nzc1NA==,9599,simonw,2020-08-11T21:25:33Z,2020-08-11T21:25:33Z,OWNER,.. and confirm if `brew tap ...` is even needed if you run `brew install simonw/datasette/datasette`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/923#issuecomment-672089281,https://api.github.com/repos/simonw/datasette/issues/923,672089281,MDEyOklzc3VlQ29tbWVudDY3MjA4OTI4MQ==,9599,simonw,2020-08-11T16:54:50Z,2020-08-11T16:54:50Z,OWNER,Also need to talk about how you install plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",677037043,Add homebrew installation to documentation,
https://github.com/simonw/datasette/issues/335#issuecomment-672088880,https://api.github.com/repos/simonw/datasette/issues/335,672088880,MDEyOklzc3VlQ29tbWVudDY3MjA4ODg4MA==,9599,simonw,2020-08-11T16:54:06Z,2020-08-11T16:54:06Z,OWNER,"It works!
```
$ brew tap simonw/datasette
$ brew install simonw/datasette/datasette
$ datasette --version
datasette, version 0.46
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-671733187,https://api.github.com/repos/simonw/datasette/issues/335,671733187,MDEyOklzc3VlQ29tbWVudDY3MTczMzE4Nw==,9599,simonw,2020-08-11T05:25:23Z,2020-08-11T05:25:23Z,OWNER,I got this almost working in `simonw/homebrew-datasette` - see https://github.com/simonw/homebrew-datasette/issues/2 for the last missing detail.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671151461,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671151461,MDEyOklzc3VlQ29tbWVudDY3MTE1MTQ2MQ==,9599,simonw,2020-08-10T03:54:06Z,2020-08-10T03:54:06Z,OWNER,For the moment I'll build it without the `--retry` mode.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671151170,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671151170,MDEyOklzc3VlQ29tbWVudDY3MTE1MTE3MA==,9599,simonw,2020-08-10T03:52:02Z,2020-08-10T03:52:02Z,OWNER,"I'm having trouble figuring out how to write a test that locks a SQLite database (so I can test that `--retry` actually works). I tried this recipe but it didn't seem to prevent another process from running `pragma journal_mode='wal';` against that database:

```python
import time
import sys
import sqlite3

filename = sys.argv[-1]

db = sqlite3.connect(filename)
with db:
    db.execute(""create table if not exists counter(id integer primary key, counter text)"")
    time.sleep(100)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671147344,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671147344,MDEyOklzc3VlQ29tbWVudDY3MTE0NzM0NA==,9599,simonw,2020-08-10T03:29:00Z,2020-08-10T03:29:00Z,OWNER,"The CLI options should take multiple database files:

    $ sqlite-utils enable-wal *.db

It's possible for this to fail if the DB is locked. How about a `--retry` option that causes it to retry a bunch of times if that happens?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671147148,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671147148,MDEyOklzc3VlQ29tbWVudDY3MTE0NzE0OA==,9599,simonw,2020-08-10T03:27:50Z,2020-08-10T03:27:50Z,OWNER,"https://www.sqlite.org/pragma.html#pragma_journal_mode lists six modes: DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF

I'm only going to implement utilities for DELETE (wal-off) and WAL (wal-on) - the other modes look like they're for specialist purposes that I don't need to support.

If it turns out I do need them I can add those to `sqlite-utils` later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/132#issuecomment-671146948,https://api.github.com/repos/simonw/sqlite-utils/issues/132,671146948,MDEyOklzc3VlQ29tbWVudDY3MTE0Njk0OA==,9599,simonw,2020-08-10T03:26:51Z,2020-08-10T03:26:51Z,OWNER,"For the CLI:

    $ sqlite-utils enable-wal github.db
    $ sqlite-utils disable-wal github.db

For the Python library:

```python
import sqlite_utils

db = sqlite_utils.Database(""github.db"")
db.enable_wal()
db.disable_wal()
mode = db.journal_mode # ""wal"" or ""delete"" or others
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675839512,Features for enabling and disabling WAL mode,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-671088832,https://api.github.com/repos/simonw/sqlite-utils/issues/131,671088832,MDEyOklzc3VlQ29tbWVudDY3MTA4ODgzMg==,9599,simonw,2020-08-09T19:00:41Z,2020-08-09T19:00:41Z,OWNER,"Should be consistent with the `create-table` command as much as possible:
```
$ sqlite-utils create-table mydb.db mytable \
    id integer \
    name text \
    age integer \
    is_good integer \
    --not-null name \
    --not-null age \
    --default is_good 1 \
    --pk=id
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/datasette/issues/335#issuecomment-671077168,https://api.github.com/repos/simonw/datasette/issues/335,671077168,MDEyOklzc3VlQ29tbWVudDY3MTA3NzE2OA==,9599,simonw,2020-08-09T17:10:15Z,2020-08-09T18:13:39Z,OWNER,"Here's the issue that explains that warning: https://github.com/pypa/pip/issues/5599

This should fix it (risky):

    from pip._internal.cli.main import main","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-671076975,https://api.github.com/repos/simonw/datasette/issues/335,671076975,MDEyOklzc3VlQ29tbWVudDY3MTA3Njk3NQ==,9599,simonw,2020-08-09T17:08:34Z,2020-08-09T17:09:21Z,OWNER,"Quick prototype of `datasette install`:
```diff
diff --git a/datasette/cli.py b/datasette/cli.py
index 287195a..95b6eb7 100644
--- a/datasette/cli.py
+++ b/datasette/cli.py
@@ -231,6 +231,18 @@ def package(
         call(args)
 
 
+@cli.command()
+@click.argument(""packages"", nargs=-1, required=True)
+def install(packages):
+    ""Install Python packages - e.g. Datasette plugins - into the same environment as Datasett""
+    import pip
+
+    try:
+        pip.main([""install""] + list(packages))
+    except SystemExit as e:
+        pass
+
+
 @cli.command()
 @click.argument(""files"", type=click.Path(exists=True), nargs=-1)
 @click.option(
```

```
$ datasette install
Usage: datasette install [OPTIONS] PACKAGES...
Try 'datasette install --help' for help.

Error: Missing argument 'PACKAGES...'.
$ datasette install datasette-vega
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Collecting datasette-vega
  Using cached datasette_vega-0.6.2-py3-none-any.whl (1.8 MB)
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-671005731,https://api.github.com/repos/simonw/datasette/issues/335,671005731,MDEyOklzc3VlQ29tbWVudDY3MTAwNTczMQ==,9599,simonw,2020-08-09T04:44:13Z,2020-08-09T17:04:21Z,OWNER,"Telling people how to figure out that `pip` location is going to be pretty unpleasant.

How about instead providing a `datasette plugins --install=datasette-graphql` command?

Or `datasette install datasette-vega`

It would run `pip install` in the same virtualenv as Datasette itself.

http://jelly.codes/articles/python-pip-module/ shows how to do this:

```python

import pip

try:
    pip.main([""install"", ""plumbum""])
except SystemExit as e:
    pass
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/918#issuecomment-671075764,https://api.github.com/repos/simonw/datasette/issues/918,671075764,MDEyOklzc3VlQ29tbWVudDY3MTA3NTc2NA==,9599,simonw,2020-08-09T16:56:48Z,2020-08-09T16:56:48Z,OWNER,GitHub security advisory: https://github.com/simonw/datasette/security/advisories/GHSA-q6j3-c4wc-63vw,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951,Security issue: read-only canned queries leak CSRF token in URL,
https://github.com/simonw/datasette/issues/915#issuecomment-671073223,https://api.github.com/repos/simonw/datasette/issues/915,671073223,MDEyOklzc3VlQ29tbWVudDY3MTA3MzIyMw==,9599,simonw,2020-08-09T16:35:20Z,2020-08-09T16:36:10Z,OWNER,"`datasette-graphql` uses the logic from `TableView` right now. It wasn't too unpleasant, but I do worry about the two of them being coupled together in this way.

https://github.com/simonw/datasette-graphql/blob/cc65ec294b0bf8e26213fc68bb5487066de9caab/datasette_graphql/utils.py#L412-L417

```python
        request = Request.fake(path_with_query_string)

        view = TableView(DatasetteSpecialConfig(datasette))
        data, _, _ = await view.data(
            request, database=database_name, hash=None, table=table_name, _next=after
        )
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671763164,Refactor TableView class so things like datasette-graphql can reuse the logic,
https://github.com/simonw/datasette/issues/919#issuecomment-671072223,https://api.github.com/repos/simonw/datasette/issues/919,671072223,MDEyOklzc3VlQ29tbWVudDY3MTA3MjIyMw==,9599,simonw,2020-08-09T16:26:17Z,2020-08-09T16:26:17Z,OWNER,Should be released in a couple of minutes: https://travis-ci.org/github/simonw/datasette/builds/716328883,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",675727366,"Travis should not build the master branch, only the main branch",
https://github.com/simonw/datasette/issues/918#issuecomment-671071710,https://api.github.com/repos/simonw/datasette/issues/918,671071710,MDEyOklzc3VlQ29tbWVudDY3MTA3MTcxMA==,9599,simonw,2020-08-09T16:21:41Z,2020-08-09T16:21:41Z,OWNER,Submitting the form on https://latest.datasette.io/fixtures/neighborhood_search demonstrates that this is fixed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951,Security issue: read-only canned queries leak CSRF token in URL,
https://github.com/simonw/datasette/issues/919#issuecomment-671071461,https://api.github.com/repos/simonw/datasette/issues/919,671071461,MDEyOklzc3VlQ29tbWVudDY3MTA3MTQ2MQ==,9599,simonw,2020-08-09T16:19:37Z,2020-08-09T16:19:37Z,OWNER,That appears to have worked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675727366,"Travis should not build the master branch, only the main branch",
https://github.com/simonw/datasette/issues/918#issuecomment-671070528,https://api.github.com/repos/simonw/datasette/issues/918,671070528,MDEyOklzc3VlQ29tbWVudDY3MTA3MDUyOA==,9599,simonw,2020-08-09T16:12:16Z,2020-08-09T16:12:16Z,OWNER,"It's worth noting that in order to exploit this issue the following would all need to be true:

- A user is running a copy of Datasette protected by a cookie-based authentication plugin AND configured with at least one writable canned query
- An attacker is in control of a URL that could concievably be returned on a page that is displayed as the result of submitting a read-only canned query
- An authenticated user of that Datasette instance, who is running a browser that doesn't support the `SameSite=lax` cookie parameter (which is [widely supported](https://caniuse.com/#feat=same-site-cookie-attribute) by modern browsers), submits the read-only canned query form and then clicks a link to the attacker's off-site page, exposing their CSRFToken in the attacker's HTTP referer logs
- The attacker then tricks that user into visiting their own malicious web page which includes a POST form that auto-submits against the writable canned query that the attacker wishes to exploit, including the CSRF token as a hidden field

The attacker would need full knowledge of the URL and form layout of the Datasette instance that they are exploiting.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951,Security issue: read-only canned queries leak CSRF token in URL,
https://github.com/simonw/datasette/issues/918#issuecomment-671070486,https://api.github.com/repos/simonw/datasette/issues/918,671070486,MDEyOklzc3VlQ29tbWVudDY3MTA3MDQ4Ng==,9599,simonw,2020-08-09T16:11:59Z,2020-08-09T16:11:59Z,OWNER,Fix has been released in Datasette 0.46: https://datasette.readthedocs.io/en/latest/changelog.html#v0-46,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675724951,Security issue: read-only canned queries leak CSRF token in URL,
https://github.com/simonw/datasette/issues/335#issuecomment-671001457,https://api.github.com/repos/simonw/datasette/issues/335,671001457,MDEyOklzc3VlQ29tbWVudDY3MTAwMTQ1Nw==,9599,simonw,2020-08-09T03:37:39Z,2020-08-09T03:37:39Z,OWNER,"Here's what happened when I installed `homebrew-vd`: https://gist.github.com/simonw/7bfd971a62743d7ca248e6b5e696c240

It worked! And from digging around, it has a virtual environment at `/usr/local/Cellar/visidata/1.5.2/libexec/`

Which means `/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip` is a working `pip`

And I tried running these commands and confirmed that I get a `datasette` with an additional plugin:

```
/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install datasette
/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install datasette-graphql
/usr/local/Cellar/visidata/1.5.2/libexec/bin/datasette plugins
[
    {
        ""name"": ""datasette-graphql"",
        ""static"": false,
        ""templates"": true,
        ""version"": ""0.11"",
        ""hooks"": [
            ""register_routes"",
            ""startup""
        ]
    }
]
```
So I can package Datasette as a homebrew package AND I can give people instructions for installing plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-670999860,https://api.github.com/repos/simonw/datasette/issues/335,670999860,MDEyOklzc3VlQ29tbWVudDY3MDk5OTg2MA==,9599,simonw,2020-08-09T03:12:44Z,2020-08-09T03:12:44Z,OWNER,How would plugin installation work if Datasette was installed via homebrew?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/datasette/issues/335#issuecomment-670999832,https://api.github.com/repos/simonw/datasette/issues/335,670999832,MDEyOklzc3VlQ29tbWVudDY3MDk5OTgzMg==,9599,simonw,2020-08-09T03:12:14Z,2020-08-09T03:12:14Z,OWNER,Another useful example: https://github.com/Homebrew/homebrew-core/blob/master/Formula/trailscraper.rb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",339505204,Package datasette for installation using homebrew,
https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585598,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667585598,MDEyOklzc3VlQ29tbWVudDY2NzU4NTU5OA==,9599,simonw,2020-08-01T20:51:28Z,2020-08-01T20:51:28Z,OWNER,CLI documentation: https://github.com/simonw/sqlite-utils/commit/57e4eb8e5564af5d97f892b3be8342451ee177a2?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371,Support tokenize option for FTS,
https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667585561,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667585561,MDEyOklzc3VlQ29tbWVudDY2NzU4NTU2MQ==,9599,simonw,2020-08-01T20:50:59Z,2020-08-01T20:50:59Z,OWNER,Turns out it works for FTS4 as well: https://www.sqlite.org/fts3.html#tokenizer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371,Support tokenize option for FTS,
https://github.com/simonw/sqlite-utils/issues/130#issuecomment-667584567,https://api.github.com/repos/simonw/sqlite-utils/issues/130,667584567,MDEyOklzc3VlQ29tbWVudDY2NzU4NDU2Nw==,9599,simonw,2020-08-01T20:41:09Z,2020-08-01T20:41:09Z,OWNER,API documentation here: https://github.com/simonw/sqlite-utils/commit/617e6f070c85be66ea04c80b78dafd08c875f8c8?short_path=e156262#diff-e1562629b8def6da772d9b0903faf703,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671130371,Support tokenize option for FTS,
https://github.com/simonw/datasette/issues/900#issuecomment-667431123,https://api.github.com/repos/simonw/datasette/issues/900,667431123,MDEyOklzc3VlQ29tbWVudDY2NzQzMTEyMw==,9599,simonw,2020-07-31T23:56:33Z,2020-07-31T23:56:33Z,OWNER,I think this is the same issue as #865. I'll look at these together!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url,
https://github.com/simonw/datasette/issues/899#issuecomment-667430790,https://api.github.com/repos/simonw/datasette/issues/899,667430790,MDEyOklzc3VlQ29tbWVudDY2NzQzMDc5MA==,9599,simonw,2020-07-31T23:54:40Z,2020-07-31T23:54:40Z,OWNER,"There's no mechanism that can do this at the moment.

You could absolutely support this with a plugin, probably using the `asgi_wrapper` plugin hook. There's an existing package at https://pypi.org/project/asgi-ratelimit/ which may be usable for this - it may even be possible to configure that using https://github.com/simonw/datasette-configure-asgi rather than using it to write a custom plugin.

Using a separate revers proxy would also be a good way to solve this. It depends which option would work best in your environment.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",660827546,How to setup a request limit per user,
https://github.com/simonw/datasette/issues/913#issuecomment-667430352,https://api.github.com/repos/simonw/datasette/issues/913,667430352,MDEyOklzc3VlQ29tbWVudDY2NzQzMDM1Mg==,9599,simonw,2020-07-31T23:52:10Z,2020-07-31T23:52:10Z,OWNER,The bigger question here is when this mechanism should be used in place of `metadata.json` or `metadata.yml`. Especially since I'm already considering renaming or reworking that mechanism since plugin configuration has nothing to do with database metadata: #493,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/913#issuecomment-667429616,https://api.github.com/repos/simonw/datasette/issues/913,667429616,MDEyOklzc3VlQ29tbWVudDY2NzQyOTYxNg==,9599,simonw,2020-07-31T23:48:25Z,2020-07-31T23:49:59Z,OWNER,"I could let plugins add additional options to `datasette serve` - but what if two plugins both try to register an option with the same name?

A better solution could be to use the existing `--config` option - and allow plugins to register their own, namespaced config options. So you could do things like:

    datasette my.db --config datasette-insert:unsafe:1

Maybe even drop the `datasette-` prefix?

    datasette my.db --config insert:unsafe:1

I think I prefer keeping the prefix to be honest - it makes it more obvious that this is a setting which comes from a specific named plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/913#issuecomment-667429690,https://api.github.com/repos/simonw/datasette/issues/913,667429690,MDEyOklzc3VlQ29tbWVudDY2NzQyOTY5MA==,9599,simonw,2020-07-31T23:48:48Z,2020-07-31T23:48:48Z,OWNER,"Here's the code in Datasette that parses `--config` options at the moment:

https://github.com/simonw/datasette/blob/7ca8c0521ac1ea48a3cd8d0fe9275d1316e54b43/datasette/cli.py#L25-L40","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/849#issuecomment-667424128,https://api.github.com/repos/simonw/datasette/issues/849,667424128,MDEyOklzc3VlQ29tbWVudDY2NzQyNDEyOA==,9599,simonw,2020-07-31T23:21:56Z,2020-07-31T23:23:24Z,OWNER,"I'm going to change the default branch on the GitHub repository. If something breaks I can always change it back again.

Done that! Default is now `main`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-667424020,https://api.github.com/repos/simonw/datasette/issues/849,667424020,MDEyOklzc3VlQ29tbWVudDY2NzQyNDAyMA==,9599,simonw,2020-07-31T23:21:30Z,2020-07-31T23:21:30Z,OWNER,https://github.com/simonw/datasette/tree/main branch now exists and will automatically mirror master (and vice-versa).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/datasette/issues/849#issuecomment-667295759,https://api.github.com/repos/simonw/datasette/issues/849,667295759,MDEyOklzc3VlQ29tbWVudDY2NzI5NTc1OQ==,9599,simonw,2020-07-31T18:45:35Z,2020-07-31T18:45:35Z,OWNER,"Watch out for places in the documentation that might link to `master` - e.g. here:

https://github.com/simonw/datasette/blob/2d7fa8b9058dfbf9c7c371cdeec115d32a177dc9/docs/custom_templates.rst#L247","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main,
https://github.com/simonw/sqlite-utils/issues/124#issuecomment-664105302,https://api.github.com/repos/simonw/sqlite-utils/issues/124,664105302,MDEyOklzc3VlQ29tbWVudDY2NDEwNTMwMg==,9599,simonw,2020-07-27T03:54:24Z,2020-07-30T22:57:51Z,OWNER,"Documentation: https://github.com/simonw/sqlite-utils/commit/814d4a7f90991be865d38aac45ff12e36df1c67d?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40

> You can pass named parameters to the query using -p:
>
>     $ sqlite-utils query dogs.db ""select :num * :num2"" -p num 5 -p num2 6
>     [{"":num * :num2"": 30}]
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665802405,sqlite-utils query should support named parameters,
https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666752039,https://api.github.com/repos/simonw/sqlite-utils/issues/129,666752039,MDEyOklzc3VlQ29tbWVudDY2Njc1MjAzOQ==,9599,simonw,2020-07-30T22:40:55Z,2020-07-30T22:40:55Z,OWNER,"This should be a separate command from `insert-files`. SQLite Archives should use a table with this schema:
```sql
CREATE TABLE sqlar(
  name TEXT PRIMARY KEY,  -- name of the file
  mode INT,               -- access permissions
  mtime INT,              -- last modification time
  sz INT,                 -- original file size
  data BLOB               -- compressed content
);
```
`insert-files` currently treats the table name as a required argument - but it's not necessary for this table. Also there shouldn't be any support for the `--column` option.

So if I write this command it should be this instead:

    sqlite-utils sqlar files.db file.txt file2.txt

But at that point, why bother? Users can use `sqlite3 files.db -Ac *.txt` instead.

So I'm not going to bother implementing this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",668308777,"""insert-files --sqlar"" for creating SQLite archives",
https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666063689,https://api.github.com/repos/simonw/sqlite-utils/issues/127,666063689,MDEyOklzc3VlQ29tbWVudDY2NjA2MzY4OQ==,9599,simonw,2020-07-30T03:08:51Z,2020-07-30T03:08:51Z,OWNER,Documentation at the bottom of this section: https://github.com/simonw/sqlite-utils/blob/8fe1e6d1be021aeeb8f08b0f77f03b75a83b6f75/docs/cli.rst#inserting-binary-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390,Ability to insert files piped to insert-files stdin,
https://github.com/simonw/sqlite-utils/issues/127#issuecomment-666047928,https://api.github.com/repos/simonw/sqlite-utils/issues/127,666047928,MDEyOklzc3VlQ29tbWVudDY2NjA0NzkyOA==,9599,simonw,2020-07-30T02:31:05Z,2020-07-30T02:31:05Z,OWNER,"Maybe could do this using an improved version of this lambda? Could teach it to look for `-` and read from `sys.stdin` if it sees it.

https://github.com/simonw/sqlite-utils/blob/710454d72aed5094573e642344fd075a0ef5372c/sqlite_utils/cli.py#L839","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390,Ability to insert files piped to insert-files stdin,
https://github.com/simonw/sqlite-utils/issues/129#issuecomment-666046819,https://api.github.com/repos/simonw/sqlite-utils/issues/129,666046819,MDEyOklzc3VlQ29tbWVudDY2NjA0NjgxOQ==,9599,simonw,2020-07-30T02:28:34Z,2020-07-30T02:28:34Z,OWNER,This code looks useful as inspiration: https://github.com/j4mie/sqlsite/blob/f2dadb8db5ed7880f8872b6591d8cb1487f777ea/sqlsite/sqlar.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",668308777,"""insert-files --sqlar"" for creating SQLite archives",
https://github.com/simonw/datasette/issues/909#issuecomment-666010395,https://api.github.com/repos/simonw/datasette/issues/909,666010395,MDEyOklzc3VlQ29tbWVudDY2NjAxMDM5NQ==,9599,simonw,2020-07-30T00:56:17Z,2020-07-30T00:56:17Z,OWNER,"```
$ curl -I https://latest.datasette.io/fixtures.db
HTTP/1.1 200 OK
content-disposition: attachment; filename=""fixtures.db""
content-type: application/octet-stream
Date: Thu, 30 Jul 2020 00:56:05 GMT
Server: Google Frontend
Transfer-Encoding: chunked
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",667467128,AsgiFileDownload: filename not correctly passed,
https://github.com/simonw/datasette/issues/909#issuecomment-665854704,https://api.github.com/repos/simonw/datasette/issues/909,665854704,MDEyOklzc3VlQ29tbWVudDY2NTg1NDcwNA==,9599,simonw,2020-07-29T19:22:31Z,2020-07-29T19:22:31Z,OWNER,"I think this results in a bug where the ""download database"" link doesn't include the correct filename: https://github.com/simonw/datasette/blob/549b1c2063db48c4622ee5c7b478a1e3cbc1ac07/datasette/views/database.py#L110-L131","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",667467128,AsgiFileDownload: filename not correctly passed,
https://github.com/simonw/sqlite-utils/issues/128#issuecomment-664683608,https://api.github.com/repos/simonw/sqlite-utils/issues/128,664683608,MDEyOklzc3VlQ29tbWVudDY2NDY4MzYwOA==,9599,simonw,2020-07-27T23:09:22Z,2020-07-27T23:09:22Z,OWNER,"This seems to work, but needs more tests:
```diff
diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..ee26433 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -7,6 +7,7 @@ import itertools
 import json
 import os
 import pathlib
+import uuid
 
 SQLITE_MAX_VARS = 999
 
@@ -40,11 +41,13 @@ COLUMN_TYPE_MAPPING = {
     str: ""TEXT"",
     bytes.__class__: ""BLOB"",
     bytes: ""BLOB"",
+    memoryview: ""BLOB"",
     datetime.datetime: ""TEXT"",
     datetime.date: ""TEXT"",
     datetime.time: ""TEXT"",
     decimal.Decimal: ""FLOAT"",
     None.__class__: ""TEXT"",
+    uuid.UUID: ""TEXT"",
     # SQLite explicit types
     ""TEXT"": ""TEXT"",
     ""INTEGER"": ""INTEGER"",
@@ -1336,6 +1339,8 @@ def jsonify_if_needed(value):
         return json.dumps(value, default=repr)
     elif isinstance(value, (datetime.time, datetime.date, datetime.datetime)):
         return value.isoformat()
+    elif isinstance(value, uuid.UUID):
+        return str(value)
     else:
         return value
 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666639051,Support UUID and memoryview types,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664163524,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664163524,MDEyOklzc3VlQ29tbWVudDY2NDE2MzUyNA==,9599,simonw,2020-07-27T07:10:41Z,2020-07-27T07:10:41Z,OWNER,Docs: https://github.com/simonw/sqlite-utils/blob/ebc802f7ff0e640b6ae11ea525290fea0115228c/docs/cli.rst#inserting-binary-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/127#issuecomment-664163206,https://api.github.com/repos/simonw/sqlite-utils/issues/127,664163206,MDEyOklzc3VlQ29tbWVudDY2NDE2MzIwNg==,9599,simonw,2020-07-27T07:10:05Z,2020-07-27T07:10:05Z,OWNER,I tried to get this working but it was a bit tricky because `-` doesn't behave like a regular `pathlib.Path` - needs a bit more thought on how the implementation would work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",666040390,Ability to insert files piped to insert-files stdin,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664128071,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664128071,MDEyOklzc3VlQ29tbWVudDY2NDEyODA3MQ==,9599,simonw,2020-07-27T05:30:54Z,2020-07-27T05:30:54Z,OWNER,"Inserting files by piping them in should work - but since a filename cannot be derived this will need a `--name blah.gif` option.

    cat blah.gif | sqlite-utils insert-files files.db files - --name=blah.gif","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664127741,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664127741,MDEyOklzc3VlQ29tbWVudDY2NDEyNzc0MQ==,9599,simonw,2020-07-27T05:29:48Z,2020-07-27T05:29:48Z,OWNER,"Test command:
```
sqlite-utils insert-files gifs.db *.gif \
    -c filename:filename \
    -c filepath:filepath \
    -c absolutepath:absolutepath \
    -c sha256:sha256 \
    -c md5:md5 \
    -c content:content \
    -c mtime:mtime \
    -c ctime:ctime \
    -c mtime_iso:mtime_iso \
    -c ctime_iso:ctime_iso \
    -c size:size \
    --pk absolutepath
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931279,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931279,MDEyOklzc3VlQ29tbWVudDY2MzkzMTI3OQ==,9599,simonw,2020-07-26T03:33:23Z,2020-07-27T04:30:49Z,OWNER,"One idea: `sqlite-utils insert-files`

It could work something like this:

    sqlite-utils insert-files files.db /tmp/blah.jpg /tmp/foo.gif \
      --table files \
      -c key:filename -c hash:sha256 -c body:content \
      --pk key

This would insert those two image files into the database in a table called `files` with a schema that looks something like this:

```sql
CREATE TABLE files (
    key text primary key,
    hash text,
    body blob
);
```

The `-c key:filename` options here are the most interesting: they let you create the table with a specific layout. The bit before the `:` is the column name. The bit after the `:` can be a range of different things:

- `filename` - just the filename
- `filepath` - the full filepath (provided on the command-line)
- `absolutepath` - the filepath expanded to start with `/home/...` or whatever
- `sha256` - the SHA256 of the contents
- `md5` - the MD5
- `content` - the binary content itself
- `mtime` - the mtime (floating point timestamp)
- `ctime` - the ctime (floating point timestamp)
- `mtime_iso` - the mtime as an ISO datetime
- `ctime_iso` - the mtime as an ISO datetime
- `size` - the size of the file in bytes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-664106621,https://api.github.com/repos/simonw/sqlite-utils/issues/114,664106621,MDEyOklzc3VlQ29tbWVudDY2NDEwNjYyMQ==,9599,simonw,2020-07-27T04:01:13Z,2020-07-27T04:01:13Z,OWNER,Work in progress in `transform` branch here: https://github.com/simonw/sqlite-utils/tree/transform,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664106405,https://api.github.com/repos/simonw/sqlite-utils/issues/126,664106405,MDEyOklzc3VlQ29tbWVudDY2NDEwNjQwNQ==,9599,simonw,2020-07-27T04:00:08Z,2020-07-27T04:00:33Z,OWNER,"```
$ echo '[
    {
        ""name"": ""transparent.gif"",
        ""content"": {
            ""$base64"": true,
            ""encoded"": ""R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7""
        }
    }
]' | sqlite-utils insert trans.db files - --pk=name
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665819048,Ability to insert binary data on the CLI using JSON,
https://github.com/simonw/sqlite-utils/issues/126#issuecomment-664065597,https://api.github.com/repos/simonw/sqlite-utils/issues/126,664065597,MDEyOklzc3VlQ29tbWVudDY2NDA2NTU5Nw==,9599,simonw,2020-07-27T00:51:11Z,2020-07-27T00:51:11Z,OWNER,"I'm going to implement this as the reverse of #125 - binary columns in JSON are now output like this:
```json
  {
    ""name"": ""lorem.txt"",
    ""mode"": 33188,
    ""mtime"": 1595805965,
    ""sz"": 16984,
    ""data"": {
      ""$base64"": true,
      ""encoded"": ""eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9""
    }
  }
]
```
So the `sqlite-utils insert` command should learn to spot `{""$base64"": true...}` values and base64 decode them before inserting them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665819048,Ability to insert binary data on the CLI using JSON,
https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664065341,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664065341,MDEyOklzc3VlQ29tbWVudDY2NDA2NTM0MQ==,9599,simonw,2020-07-27T00:49:41Z,2020-07-27T00:49:41Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/commit/20e543e9a492f2e764caae73c38e87f18eaec444?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570,"Output binary columns in ""sqlite-utils query"" JSON",
https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664062546,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664062546,MDEyOklzc3VlQ29tbWVudDY2NDA2MjU0Ng==,9599,simonw,2020-07-27T00:33:03Z,2020-07-27T00:33:03Z,OWNER,"I'm going to imitate how Datasette solves this problem:
```json
[
  {
    ""name"": ""lorem.txt"",
    ""mode"": 33188,
    ""mtime"": 1595805965,
    ""sz"": 16984,
    ""data"": {
      ""$base64"": true,
      ""encoded"": ""eJzt0c1xAyEMBeC7q1ABHleR3HxNAQrIjmb4M0gelx+RTY7p4N2WBYT0vmufUknH8kq5lz5pqRFXsTOl3pYkE/NJnHXoStruJEVjc0mOCyTqq/ZMJnXEZW1Js2ZvRm5U+DPKk9hRWqjyvTFx0YfzhT6MpGmN2lR1fzxjyfVMD9dFrS+bnkleMpMam/ZGXgrX1I/K+5Au3S/9lNQRh0k4Gq/RUz8GiKfsQm+7JLsJ6fTo5JhVG00ZU76kZZkxePx49uIjnpNoJyYlWUsoaSl/CcVATje/Kxu13RANnrHweaH3V5Jh4jvGyKCnxJLiXPKhmW3fiCnG7Jql7RR3UvFo8jJ4z039dtOkTFmWzL1be9lt8A5II471m6vXy+l0BR/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984AMf+MAHPvCBD3zgAx/4wAc+8IEPfOADH/jABz7wgQ984PuP7xubBoN9""
    }
  }
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570,"Output binary columns in ""sqlite-utils query"" JSON",
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048720,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664048720,MDEyOklzc3VlQ29tbWVudDY2NDA0ODcyMA==,9599,simonw,2020-07-26T22:32:50Z,2020-07-26T22:33:20Z,OWNER,"This seems to work in creating a SQLite archive containing all `.gif` files in the current directory:

    /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -c *.gif

Then listing files like this:
```
$ /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 archive.db -A -t
copyable.gif
debug-allow.gif
flash.gif
table-md.gif
```
Here's the schema:
```
$ sqlite3 archive.db .schema                                  
CREATE TABLE sqlar(
  name TEXT PRIMARY KEY,  -- name of the file
  mode INT,               -- access permissions
  mtime INT,              -- last modification time
  sz INT,                 -- original file size
  data BLOB               -- compressed content
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664048432,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664048432,MDEyOklzc3VlQ29tbWVudDY2NDA0ODQzMg==,9599,simonw,2020-07-26T22:29:31Z,2020-07-26T22:29:31Z,OWNER,"I'm trying to play with `sqlite3 -A` on my Mac.

`sqlite3 -A` tells me that it's an unknown option - but I used `brew info sqlite` to find my homebrew installed version and it turns out this works:

```
 % /usr/local/Cellar/sqlite/3.32.1/bin/sqlite3 -A
Wrong number of arguments.  Usage:
.archive ...             Manage SQL archives
   Each command must have exactly one of the following options:
     -c, --create               Create a new archive
     -u, --update               Add or update files with changed mtime
     -i, --insert               Like -u but always add even if unchanged
     -t, --list                 List contents of archive
     -x, --extract              Extract files from archive
   Optional arguments:
     -v, --verbose              Print each filename as it is processed
     -f FILE, --file FILE       Use archive FILE (default is current db)
     -a FILE, --append FILE     Open FILE using the apndvfs VFS
     -C DIR, --directory DIR    Read/extract files from directory DIR
     -n, --dryrun               Show the SQL that would have occurred
   Examples:
     .ar -cf ARCHIVE foo bar  # Create ARCHIVE from files foo and bar
     .ar -tf ARCHIVE          # List members of ARCHIVE
     .ar -xvf ARCHIVE         # Verbosely extract files from ARCHIVE
   See also:
      http://sqlite.org/cli.html#sqlar_archive_support
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-664013338,https://api.github.com/repos/simonw/sqlite-utils/issues/122,664013338,MDEyOklzc3VlQ29tbWVudDY2NDAxMzMzOA==,9599,simonw,2020-07-26T16:57:35Z,2020-07-26T16:57:35Z,OWNER,"I should consider easy compatibility with https://www.sqlite.org/sqlar.html

> An SQLite Archive is an ordinary SQLite database file that contains the following table as part of its schema:
> ```
> CREATE TABLE sqlar(
>   name TEXT PRIMARY KEY,  -- name of the file
>   mode INT,               -- access permissions
>   mtime INT,              -- last modification time
>   sz INT,                 -- original file size
>   data BLOB               -- compressed content
> );
> ```
> Each row of the SQLAR table holds the content of a single file. The filename (the full pathname relative to the root of the archive) is in the ""name"" field. The ""mode"" field is an integer which is the unix-style access permissions for the file. ""mtime"" is the modification time of the file in seconds since 1970. ""sz"" is the original uncompressed size of the file. The ""data"" field contains the file content. The content is usually compressed using [Deflate](http://zlib.net/), though not always. If the ""sz"" field is equal to the size of the ""data"" field, then the content is stored uncompressed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012247,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664012247,MDEyOklzc3VlQ29tbWVudDY2NDAxMjI0Nw==,9599,simonw,2020-07-26T16:48:46Z,2020-07-26T16:48:46Z,OWNER,"I could solve round tripping (at least a bit) by allowing insert to be run with a flag that says ""these columns are base64 encoded, store the decoded data in a BLOB"".

That would solve inserting binary data using JSON too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570,"Output binary columns in ""sqlite-utils query"" JSON",
https://github.com/simonw/sqlite-utils/issues/125#issuecomment-664012148,https://api.github.com/repos/simonw/sqlite-utils/issues/125,664012148,MDEyOklzc3VlQ29tbWVudDY2NDAxMjE0OA==,9599,simonw,2020-07-26T16:47:51Z,2020-07-26T16:47:51Z,OWNER,Best solution I can think of is to return the data as base64. It's a bit nasty since it means you can't round trip it back again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665817570,"Output binary columns in ""sqlite-utils query"" JSON",
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931426,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931426,MDEyOklzc3VlQ29tbWVudDY2MzkzMTQyNg==,9599,simonw,2020-07-26T03:35:58Z,2020-07-26T16:44:33Z,OWNER,Related: #123 (`--raw` option),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931662,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931662,MDEyOklzc3VlQ29tbWVudDY2MzkzMTY2Mg==,9599,simonw,2020-07-26T03:40:29Z,2020-07-26T03:40:29Z,OWNER,Maybe support `--replace` for replacing images with an existing primary key.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/sqlite-utils/issues/122#issuecomment-663931317,https://api.github.com/repos/simonw/sqlite-utils/issues/122,663931317,MDEyOklzc3VlQ29tbWVudDY2MzkzMTMxNw==,9599,simonw,2020-07-26T03:33:54Z,2020-07-26T03:33:54Z,OWNER,"The command also accepts one or more directories, in which case it will recursively scan them for all files that they contain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665700495,CLI utility for inserting binary files into SQLite,
https://github.com/simonw/datasette/issues/906#issuecomment-663779460,https://api.github.com/repos/simonw/datasette/issues/906,663779460,MDEyOklzc3VlQ29tbWVudDY2Mzc3OTQ2MA==,9599,simonw,2020-07-25T00:07:10Z,2020-07-25T00:07:10Z,OWNER,"Demo of the new functionality:

* https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=false
* https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22root%22%0D%0A%7D&allow=true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665400224,"""allow"": true for anyone, ""allow"": false for nobody",
https://github.com/simonw/datasette/issues/908#issuecomment-663779179,https://api.github.com/repos/simonw/datasette/issues/908,663779179,MDEyOklzc3VlQ29tbWVudDY2Mzc3OTE3OQ==,9599,simonw,2020-07-25T00:05:48Z,2020-07-25T00:06:15Z,OWNER,"The documentation section here now has a bunch of different links to live demos illustrating different ""allow"" block syntax: https://github.com/simonw/datasette/blob/092874202c8748d6e0d4800eaf707c0145d95ffe/docs/authentication.rst#defining-permissions-with-allow-blocks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665407663,"Interactive debugging tool for ""allow"" blocks",
https://github.com/simonw/datasette/issues/906#issuecomment-663767678,https://api.github.com/repos/simonw/datasette/issues/906,663767678,MDEyOklzc3VlQ29tbWVudDY2Mzc2NzY3OA==,9599,simonw,2020-07-24T23:07:22Z,2020-07-24T23:07:22Z,OWNER,"Illustration of current system:

https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow=null - `null` allows

https://latest.datasette.io/-/allow-debug?actor=%7B%0D%0A++++%22id%22%3A+%22terry%22%0D%0A%7D&allow={} - `{}` denies","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665400224,"""allow"": true for anyone, ""allow"": false for nobody",
https://github.com/simonw/datasette/issues/908#issuecomment-663765308,https://api.github.com/repos/simonw/datasette/issues/908,663765308,MDEyOklzc3VlQ29tbWVudDY2Mzc2NTMwOA==,9599,simonw,2020-07-24T22:57:15Z,2020-07-24T22:57:15Z,OWNER,Tool lives at https://latest.datasette.io/-/allow-debug,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665407663,"Interactive debugging tool for ""allow"" blocks",
https://github.com/simonw/datasette/issues/907#issuecomment-663764203,https://api.github.com/repos/simonw/datasette/issues/907,663764203,MDEyOklzc3VlQ29tbWVudDY2Mzc2NDIwMw==,9599,simonw,2020-07-24T22:53:07Z,2020-07-24T22:53:07Z,OWNER,"Actually that is already covered here: 
https://github.com/simonw/datasette/blob/6be5654ffab282e8cf39cc138ba2d4496ebc7407/docs/authentication.rst#L158","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665403403,Allow documentation doesn't explain what happens with multiple allow keys,
https://github.com/simonw/datasette/issues/908#issuecomment-663726318,https://api.github.com/repos/simonw/datasette/issues/908,663726318,MDEyOklzc3VlQ29tbWVudDY2MzcyNjMxOA==,9599,simonw,2020-07-24T20:43:57Z,2020-07-24T20:45:38Z,OWNER,"I can implement this as a plugin. Or it could ship as part of Datasette, somewhere under the `/-/` namespace like the `PermissionsDebugView` and `MessagesDebugView` tools.

I'm going to ship it in Datasette core, to further reinforce the philosophy that debugging tools are important.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665407663,"Interactive debugging tool for ""allow"" blocks",
https://github.com/simonw/datasette/issues/907#issuecomment-663726146,https://api.github.com/repos/simonw/datasette/issues/907,663726146,MDEyOklzc3VlQ29tbWVudDY2MzcyNjE0Ng==,9599,simonw,2020-07-24T20:43:27Z,2020-07-24T20:43:27Z,OWNER,"It might be good to have a little interactive tool which helps debug these things, since there are quite a few edge-cases and the damage caused if people use them incorrectly is substantial.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665403403,Allow documentation doesn't explain what happens with multiple allow keys,
https://github.com/simonw/datasette/issues/456#issuecomment-663724675,https://api.github.com/repos/simonw/datasette/issues/456,663724675,MDEyOklzc3VlQ29tbWVudDY2MzcyNDY3NQ==,9599,simonw,2020-07-24T20:39:17Z,2020-07-24T20:39:17Z,OWNER,Yes this is still a bug!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",442327592,Installing installs the tests package,
https://github.com/simonw/datasette/pull/902#issuecomment-663724425,https://api.github.com/repos/simonw/datasette/issues/902,663724425,MDEyOklzc3VlQ29tbWVudDY2MzcyNDQyNQ==,9599,simonw,2020-07-24T20:38:42Z,2020-07-24T20:38:42Z,OWNER,Thanks for spotting this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",662439034,Don't install tests package,
https://github.com/simonw/datasette/issues/906#issuecomment-663720907,https://api.github.com/repos/simonw/datasette/issues/906,663720907,MDEyOklzc3VlQ29tbWVudDY2MzcyMDkwNw==,9599,simonw,2020-07-24T20:29:24Z,2020-07-24T20:29:24Z,OWNER,"Here are the existing test cases:
https://github.com/simonw/datasette/blob/2115d7e3457b48b3cf9c81551b9fed2d0e9cd111/tests/test_utils.py#L468-L505","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",665400224,"""allow"": true for anyone, ""allow"": false for nobody",
https://github.com/simonw/datasette/issues/905#issuecomment-662241702,https://api.github.com/repos/simonw/datasette/issues/905,662241702,MDEyOklzc3VlQ29tbWVudDY2MjI0MTcwMg==,9599,simonw,2020-07-22T04:59:46Z,2020-07-22T04:59:46Z,OWNER,"Deployed and working:
```
% curl -I 'https://fivethirtyeight.datasettes.com/fivethirtyeight.db'
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2020 04:59:23 GMT
Content-Type: application/octet-stream
Content-Length: 281845760
Connection: keep-alive
Set-Cookie: __cfduid=d550b15c99aa59144e49557ced64fc48a1595393963; expires=Fri, 21-Aug-20 04:59:23 GMT; path=/; domain=.datasettes.com; HttpOnly; SameSite=Lax
Via: 1.1 vegur
Cache-Control: max-age=14400
CF-Cache-Status: MISS
Accept-Ranges: bytes
cf-request-id: 04167d0c7100000540f98e8200000001
Expect-CT: max-age=604800, report-uri=""https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct""
Server: cloudflare
CF-RAY: 5b6a978d89b30540-LAX
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663317875,/database.db download should include content-length header,
https://github.com/simonw/datasette/issues/905#issuecomment-662114881,https://api.github.com/repos/simonw/datasette/issues/905,662114881,MDEyOklzc3VlQ29tbWVudDY2MjExNDg4MQ==,9599,simonw,2020-07-21T21:25:37Z,2020-07-21T21:25:37Z,OWNER,I can use `aiofiles.os.stat` for this: https://github.com/Tinche/aiofiles/blob/master/aiofiles/os.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663317875,/database.db download should include content-length header,
https://github.com/simonw/datasette/issues/898#issuecomment-660419792,https://api.github.com/repos/simonw/datasette/issues/898,660419792,MDEyOklzc3VlQ29tbWVudDY2MDQxOTc5Mg==,9599,simonw,2020-07-18T03:57:46Z,2020-07-18T03:57:46Z,OWNER,"This requires some thought. There are various testing utilities that don't exist yet that plugins might benefit from - off the top of my head:

- `assert_permissions_checked`
- `assert_template_rendered`

I should resist the temptation to provide a reusable version of `make_app_client` that provides a fully configured Datasette instance because I need to be able to change the design of the Datasette `fixtures.db` test database without accidentally breaking any plugins that depend on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",659873662,datasette.utils.testing module,
https://github.com/simonw/datasette/issues/898#issuecomment-660419499,https://api.github.com/repos/simonw/datasette/issues/898,660419499,MDEyOklzc3VlQ29tbWVudDY2MDQxOTQ5OQ==,9599,simonw,2020-07-18T03:55:13Z,2020-07-18T03:55:13Z,OWNER,Maybe I should make `httpx` a testing dependency of Datasette itself. It's usage is already encouraged in plugins by https://datasette.readthedocs.io/en/stable/testing_plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",659873662,datasette.utils.testing module,
https://github.com/simonw/datasette/issues/897#issuecomment-660318063,https://api.github.com/repos/simonw/datasette/issues/897,660318063,MDEyOklzc3VlQ29tbWVudDY2MDMxODA2Mw==,9599,simonw,2020-07-17T20:16:02Z,2020-07-17T20:16:02Z,OWNER,Documentation here: https://datasette.readthedocs.io/en/latest/internals.html#request-object,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",659580487,Request method for retrieving the unparsed request body,
https://github.com/simonw/datasette/issues/896#issuecomment-659773897,https://api.github.com/repos/simonw/datasette/issues/896,659773897,MDEyOklzc3VlQ29tbWVudDY1OTc3Mzg5Nw==,9599,simonw,2020-07-17T01:26:08Z,2020-07-17T01:26:08Z,OWNER,I manually tested it with those plugins and it seems to interoperate just fine - since both of those use `<pre>` tags for the cases that I care about so they're already expecting white-space to be pre wrapped in some way.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659734703,https://api.github.com/repos/simonw/datasette/issues/896,659734703,MDEyOklzc3VlQ29tbWVudDY1OTczNDcwMw==,9599,simonw,2020-07-16T23:34:57Z,2020-07-16T23:34:57Z,OWNER,"I'm worried about how this will interact with some of the plugins:

* https://github.com/simonw/datasette-json-html
* https://github.com/simonw/datasette-pretty-json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659615034,https://api.github.com/repos/simonw/datasette/issues/896,659615034,MDEyOklzc3VlQ29tbWVudDY1OTYxNTAzNA==,9599,simonw,2020-07-16T19:14:07Z,2020-07-16T19:14:07Z,OWNER,"Demo: https://srccon-2020.datasette.io/srccon?sql=select+id%2C+day%2C+time%2C+event_name%2C+event_description%2C+facilitators+from+sessions+order+by+event_dtstart+limit+101

I really like this:

<img width=""1396"" alt=""srccon__select_id__day__time__event_name__event_description__facilitators_from_sessions_order_by_event_dtstart_limit_101"" src=""https://user-images.githubusercontent.com/9599/87712630-d228da00-c75d-11ea-8f44-69886e599a07.png"">


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/896#issuecomment-659610687,https://api.github.com/repos/simonw/datasette/issues/896,659610687,MDEyOklzc3VlQ29tbWVudDY1OTYxMDY4Nw==,9599,simonw,2020-07-16T19:05:43Z,2020-07-16T19:05:43Z,OWNER,I'm going to give this a go - if it turns out to be a bad idea I can revert it back out again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",658476055,Use white-space: pre-wrap on ALL table cell contents,
https://github.com/simonw/datasette/issues/895#issuecomment-659085528,https://api.github.com/repos/simonw/datasette/issues/895,659085528,MDEyOklzc3VlQ29tbWVudDY1OTA4NTUyOA==,9599,simonw,2020-07-16T00:32:47Z,2020-07-16T00:32:47Z,OWNER,"This was added in https://github.com/simonw/datasette/commit/504196341c49840270bd75ea1a1871ef386ba7ea - here's the relevant code (which only applies on the table page, not the query page):

https://github.com/simonw/datasette/blob/d6e03b04302a0852e7133dc030eab50177c37be7/datasette/views/table.py#L196-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657747959,SQL query output should show numeric values in a different colour,
https://github.com/simonw/datasette/issues/892#issuecomment-657268433,https://api.github.com/repos/simonw/datasette/issues/892,657268433,MDEyOklzc3VlQ29tbWVudDY1NzI2ODQzMw==,9599,simonw,2020-07-12T20:02:17Z,2020-07-12T20:02:35Z,OWNER,"Fixed https://datasette.readthedocs.io/en/latest/

<img width=""805"" alt=""Datasette_—_Datasette_documentation_and_How_to_set_a_custom_theme__·_Issue__638_·_readthedocs_readthedocs_org"" src=""https://user-images.githubusercontent.com/9599/87255480-f32cba80-c43f-11ea-8928-01528859929f.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",655465863,"""latest"" in new documentation navbar is invisible",
https://github.com/simonw/datasette/issues/892#issuecomment-657268051,https://api.github.com/repos/simonw/datasette/issues/892,657268051,MDEyOklzc3VlQ29tbWVudDY1NzI2ODA1MQ==,9599,simonw,2020-07-12T19:58:24Z,2020-07-12T19:58:24Z,OWNER,"```css
.wy-side-nav-search > div.version {
    margin-top: -.4045em;
    margin-bottom: .809em;
    font-weight: normal;
    color: rgba(255,255,255,0.3);
}
```
Fix can go here: https://github.com/simonw/datasette/blob/ee0ef016523a765b6ef6eaa43cad9ad568f78ae4/docs/_static/css/custom.css#L1-L3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",655465863,"""latest"" in new documentation navbar is invisible",
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-656363548,https://api.github.com/repos/simonw/sqlite-utils/issues/114,656363548,MDEyOklzc3VlQ29tbWVudDY1NjM2MzU0OA==,9599,simonw,2020-07-09T21:37:28Z,2020-07-09T21:37:28Z,OWNER,"I'm going to add a second method `.transform_table_sql(...)` - which returns the SQL that would have been executed but does NOT execute it.

Advanced callers can use this to include their own additional steps in the same transaction - e.g. recreating views or triggers.

More importantly it gives me a useful hook for writing some unit tests against the generated SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655786374,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655786374,MDEyOklzc3VlQ29tbWVudDY1NTc4NjM3NA==,9599,simonw,2020-07-08T22:16:54Z,2020-07-08T22:16:54Z,OWNER,"According to https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes the hardest bits to consider are how to deal with existing foreign key relationships, triggers and views.

I'm OK leaving views as an exercise for the caller - many of these transformations may not need any view changes at all.

Foreign key relationships are important: it should handle these automatically as effectively as possible.

Likewise trigger changes: need to think about what this means.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655785396,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655785396,MDEyOklzc3VlQ29tbWVudDY1NTc4NTM5Ng==,9599,simonw,2020-07-08T22:14:10Z,2020-07-08T22:14:10Z,OWNER,"Work in progress: not quite right yet, I need smarter logic for how renamed columns are reflected in the generated `INSERT INTO ... SELECT ...` query:
```python
    def transform_table(
        self,
        columns=None,
        rename=None,
        change_type=None,
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):
        assert self.exists(), ""Cannot transform a table that doesn't exist yet""
        columns = columns or self.columns_dict
        if rename is not None or change_type is not None:
            columns = {rename.get(key, key): change_type.get(key, value) for key, value in columns.items()}
        new_table_name = ""{}_new_{}"".format(self.name, os.urandom(6).hex())
        previous_columns = set(self.columns_dict.keys())
        with self.db.conn:
            columns = {name: value for (name, value) in columns.items()}
            new_table = self.db.create_table(
                new_table_name,
                columns,
                pk=pk,
                foreign_keys=foreign_keys,
                column_order=column_order,
                not_null=not_null,
                defaults=defaults,
                hash_id=hash_id,
                extracts=extracts,
            )
            # Copy across data - but only for columns that exist in both
            new_columns = set(columns.keys())
            columns_to_copy = new_columns.intersection(previous_columns)
            copy_sql = ""INSERT INTO [{new_table}] ({new_cols}) SELECT {old_cols} FROM [{old_table}]"".format(
                new_table=new_table_name,
                old_table=self.name,
                old_cols="", "".join(""[{}]"".format(col) for col in columns_to_copy),
                new_cols="", "".join(""[{}]"".format(rename.get(col, col)) for col in columns_to_copy),
            )
            self.db.conn.execute(copy_sql)
            # Drop the old table
            self.db.conn.execute(""DROP TABLE [{}]"".format(self.name))
            # Rename the new one
            self.db.conn.execute(
                ""ALTER TABLE [{}] RENAME TO [{}]"".format(new_table_name, self.name)
            )
        return self
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655783875,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655783875,MDEyOklzc3VlQ29tbWVudDY1NTc4Mzg3NQ==,9599,simonw,2020-07-08T22:09:51Z,2020-07-08T22:10:16Z,OWNER,I can have a convenient `change_type={...}` parameter for changing column types too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655782477,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655782477,MDEyOklzc3VlQ29tbWVudDY1NTc4MjQ3Nw==,9599,simonw,2020-07-08T22:06:23Z,2020-07-08T22:06:23Z,OWNER,"Thinking about the method signature:
```python
    def transform_table(
        self,
        columns,
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):
```
This requires the caller to provide the exact set of columns for the new table.

It would be useful if this was optional - if you could omit the columns and have it automatically use the previous columns. This would let you change things like the primary key or the column order using the other arguments.

Even better: allow column renaming using an optional `rename={...}` argument:

```python
db[""dogs""].transform_table(rename={""name"": ""dog_name""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655778058,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655778058,MDEyOklzc3VlQ29tbWVudDY1NTc3ODA1OA==,9599,simonw,2020-07-08T21:54:30Z,2020-07-08T21:54:30Z,OWNER,"Don't forget this step:
>  If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677909,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677909,MDEyOklzc3VlQ29tbWVudDY1NTY3NzkwOQ==,9599,simonw,2020-07-08T18:16:39Z,2020-07-08T18:16:39Z,OWNER,"Since neither the term ""transform"" or ""migrate"" are used in the codebase at the moment, I think I'll go with `.transform_table()` - that leaves the term ""migrate"" available for any future database migrations system (similar to Django's).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677396,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677396,MDEyOklzc3VlQ29tbWVudDY1NTY3NzM5Ng==,9599,simonw,2020-07-08T18:15:39Z,2020-07-08T18:15:39Z,OWNER,"Alternative possible names:
- `.transform_table()`
- `.migrate()`
- `.transform()`

I'm torn between `.migrate_table()` and `.transform_table()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655677099,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655677099,MDEyOklzc3VlQ29tbWVudDY1NTY3NzA5OQ==,9599,simonw,2020-07-08T18:15:02Z,2020-07-08T18:15:02Z,OWNER,"I'm not so keen on that chained API - it's pretty complicated.

Here's an idea for a much simpler interface. Essentially it lets you say ""take table X and migrate its contents to a new table with this structure - then atomically rename the tables to switch them"":
```python
db[""mytable""].migrate_table({""id"": int, ""name"": str""}, pk=""id"")
```
The `migrate_table()` method would take the same exact signature as the `table.create()` method: https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L615-L625","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/issues/119#issuecomment-655674910,https://api.github.com/repos/simonw/sqlite-utils/issues/119,655674910,MDEyOklzc3VlQ29tbWVudDY1NTY3NDkxMA==,9599,simonw,2020-07-08T18:10:18Z,2020-07-08T18:10:18Z,OWNER,"This will work similar to how `.add_foreign_keys()` works: turn on `writable_schema` and rewrite the `sql` for that table in the `sqlite_master` table.

Here's that code today - it could be adapted to include removal of foreign keys that we no longer want:

https://github.com/simonw/sqlite-utils/blob/a236a6bc771a5a6a9d7e814f1986d461afc422d2/sqlite_utils/db.py#L391-L401","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770,Ability to remove a foreign key,
https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655673896,https://api.github.com/repos/simonw/sqlite-utils/issues/121,655673896,MDEyOklzc3VlQ29tbWVudDY1NTY3Mzg5Ng==,9599,simonw,2020-07-08T18:08:11Z,2020-07-08T18:08:11Z,OWNER,"I'm with you on most of this. Completely agreed that the CLI should do everything in a transaction.

The one thing I'm not keen on is forcing calling code to explicitly start a transaction, for a couple of reasons:

1. It will break all of the existing code out there
2. It doesn't match to how I most commonly use this library - as an interactive tool in a Jupyter notebook, where I'm generally working against a brand new scratch database and any errors don't actually matter

So... how about this: IF you wrap your code in a `with db:` block then the `.insert()` and suchlike methods expect you to manage transactions yourself. But if you don't use the context manager they behave like they do at the moment (or maybe a bit more sensibly).

That way existing code works as it does today, lazy people like me can call `.insert()` without thinking about transactions, but people writing actual production code (as opposed to Jupyter hacks) have a sensible way to take control of the transactions themselves.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652961907,Improved (and better documented) support for transactions,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655653292,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655653292,MDEyOklzc3VlQ29tbWVudDY1NTY1MzI5Mg==,9599,simonw,2020-07-08T17:26:02Z,2020-07-08T17:26:02Z,OWNER,"Awesome, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/issues/114#issuecomment-655290625,https://api.github.com/repos/simonw/sqlite-utils/issues/114,655290625,MDEyOklzc3VlQ29tbWVudDY1NTI5MDYyNQ==,9599,simonw,2020-07-08T05:15:45Z,2020-07-08T05:15:45Z,OWNER,"Ideally this would all happen in a single transaction, such that other processes talking to the database would not see any inconsistent state while the table copy was taking place. Need to confirm that this is possible. Also refs transactions thoughts in #121.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table,
https://github.com/simonw/sqlite-utils/pull/120#issuecomment-655289686,https://api.github.com/repos/simonw/sqlite-utils/issues/120,655289686,MDEyOklzc3VlQ29tbWVudDY1NTI4OTY4Ng==,9599,simonw,2020-07-08T05:13:11Z,2020-07-08T05:13:11Z,OWNER,"This is an excellent fix, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652816158,Fix query command's support for DML,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655286864,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655286864,MDEyOklzc3VlQ29tbWVudDY1NTI4Njg2NA==,9599,simonw,2020-07-08T05:05:27Z,2020-07-08T05:05:36Z,OWNER,"The only thing missing from this PR is updates to the documentation. Those need to go in two places:

- In the Python API docs. I suggest adding a note to this section about bulk inserts: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/python-api.rst#bulk-inserts
- In the CLI docs, in this section: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/docs/cli.rst#inserting-json-data

Here's an example of a previous commit that includes updates to both CLI and API documentation: https://github.com/simonw/sqlite-utils/commit/f9473ace14878212c1fa968b7bd2f51e4f064dba#diff-e3e2a9bfd88566b05001b02a3f51d286","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284168,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655284168,MDEyOklzc3VlQ29tbWVudDY1NTI4NDE2OA==,9599,simonw,2020-07-08T04:58:00Z,2020-07-08T04:58:00Z,OWNER,"Oops didn't mean to click ""close"" there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655284054,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655284054,MDEyOklzc3VlQ29tbWVudDY1NTI4NDA1NA==,9599,simonw,2020-07-08T04:57:38Z,2020-07-08T04:57:38Z,OWNER,Thoughts on transactions would be much appreciated in #121 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655283393,https://api.github.com/repos/simonw/sqlite-utils/issues/118,655283393,MDEyOklzc3VlQ29tbWVudDY1NTI4MzM5Mw==,9599,simonw,2020-07-08T04:55:18Z,2020-07-08T04:55:18Z,OWNER,"This is a really good idea - and thank you for the detailed discussion in the pull request.

I'm keen to discuss how transactions can work better. I tend to use this pattern in my own code:

    with db.conn:
        db[""table""].insert(...)

But it's not documented and I've not though very hard about it!

I like having inserts that handle 10,000+ rows commit on every chunk so I can watch their progress from another process, but the library should absolutely support people who want to commit all of the rows in a single transaction - or combine changes with DML.

Lots to discuss here. I'll start a new issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",651844316,Add insert --truncate option,
https://github.com/simonw/datasette/issues/784#issuecomment-654424704,https://api.github.com/repos/simonw/datasette/issues/784,654424704,MDEyOklzc3VlQ29tbWVudDY1NDQyNDcwNA==,9599,simonw,2020-07-06T19:31:53Z,2020-07-06T19:31:53Z,OWNER,Documentation: https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",628003707,Ability to sign in to Datasette as a root account,
https://github.com/simonw/datasette/pull/890#issuecomment-653314465,https://api.github.com/repos/simonw/datasette/issues/890,653314465,MDEyOklzc3VlQ29tbWVudDY1MzMxNDQ2NQ==,9599,simonw,2020-07-03T03:07:41Z,2020-07-03T03:07:41Z,OWNER,"This is an excellent fix. Thanks!

Not sure why codecov is complaining. I'm going to merge it as-is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",650305298,Load only python files from plugins-dir.,
https://github.com/simonw/datasette/issues/886#issuecomment-652732460,https://api.github.com/repos/simonw/datasette/issues/886,652732460,MDEyOklzc3VlQ29tbWVudDY1MjczMjQ2MA==,9599,simonw,2020-07-02T01:52:02Z,2020-07-02T01:52:02Z,OWNER,In investigating this I'm not convinced 500 errors are being correctly raised by errors in canned writable queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649429772,Reconsider how _actor_X magic parameter deals with missing values,
https://github.com/simonw/datasette/issues/886#issuecomment-652731459,https://api.github.com/repos/simonw/datasette/issues/886,652731459,MDEyOklzc3VlQ29tbWVudDY1MjczMTQ1OQ==,9599,simonw,2020-07-02T01:48:08Z,2020-07-02T01:48:08Z,OWNER,"A common error with this (and other) magic parameters is for the database query to result in the following:

    You did not supply a value for binding 3.

This is a pretty crufty error. I'm inclined to say that ANY missing or invalid magic parameter should be treated as a `None` value instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649429772,Reconsider how _actor_X magic parameter deals with missing values,
https://github.com/simonw/datasette/issues/887#issuecomment-652711822,https://api.github.com/repos/simonw/datasette/issues/887,652711822,MDEyOklzc3VlQ29tbWVudDY1MjcxMTgyMg==,9599,simonw,2020-07-02T00:31:33Z,2020-07-02T00:31:33Z,OWNER,"If a canned query has a title defined that will be used instead: https://latest.datasette.io/fixtures/neighborhood_search

<img width=""821"" alt=""fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_"" src=""https://user-images.githubusercontent.com/9599/86303455-b26fae80-bbc0-11ea-9054-0cc6c086f143.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/issues/887#issuecomment-652711562,https://api.github.com/repos/simonw/datasette/issues/887,652711562,MDEyOklzc3VlQ29tbWVudDY1MjcxMTU2Mg==,9599,simonw,2020-07-02T00:30:43Z,2020-07-02T00:30:43Z,OWNER,"Demo has updated: https://latest.datasette.io/fixtures/magic_parameters

<img width=""900"" alt=""fixtures__select___header_user_agent_as_user_agent____now_datetime_utc_as_datetime"" src=""https://user-images.githubusercontent.com/9599/86303410-95d37680-bbc0-11ea-892c-0ef4744c261a.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/pull/883#issuecomment-652710178,https://api.github.com/repos/simonw/datasette/issues/883,652710178,MDEyOklzc3VlQ29tbWVudDY1MjcxMDE3OA==,9599,simonw,2020-07-02T00:25:44Z,2020-07-02T00:25:44Z,OWNER,This is a great idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648749062,Skip counting hidden tables,
https://github.com/simonw/datasette/issues/887#issuecomment-652709199,https://api.github.com/repos/simonw/datasette/issues/887,652709199,MDEyOklzc3VlQ29tbWVudDY1MjcwOTE5OQ==,9599,simonw,2020-07-02T00:21:54Z,2020-07-02T00:21:54Z,OWNER,"Example in the live demo: https://latest.datasette.io/fixtures/magic_parameters

<img width=""858"" alt=""fixtures__select___header_user_agent_as_user_agent____now_datetime_utc_as_datetime"" src=""https://user-images.githubusercontent.com/9599/86303008-58bab480-bbbf-11ea-9c99-aec7ed1f4f56.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649437530,Canned query page should show the name of the canned query,
https://github.com/simonw/datasette/issues/885#issuecomment-652681996,https://api.github.com/repos/simonw/datasette/issues/885,652681996,MDEyOklzc3VlQ29tbWVudDY1MjY4MTk5Ng==,9599,simonw,2020-07-01T22:44:47Z,2020-07-01T22:44:47Z,OWNER,https://simonwillison.net/2020/Jul/1/datasette-045/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",649373451,Blog entry about the release,
https://github.com/simonw/datasette/issues/882#issuecomment-652663177,https://api.github.com/repos/simonw/datasette/issues/882,652663177,MDEyOklzc3VlQ29tbWVudDY1MjY2MzE3Nw==,9599,simonw,2020-07-01T21:48:08Z,2020-07-01T21:48:08Z,OWNER,https://datasette.readthedocs.io/en/latest/changelog.html#v0-45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648673556,Release notes for 0.45,
https://github.com/simonw/datasette/issues/880#issuecomment-652646487,https://api.github.com/repos/simonw/datasette/issues/880,652646487,MDEyOklzc3VlQ29tbWVudDY1MjY0NjQ4Nw==,9599,simonw,2020-07-01T21:05:48Z,2020-07-01T21:05:48Z,OWNER,"I've been testing the WIP using this in the console:
```javascript
fetch('/data/add_name.json', {
  method: 'POST',
  body: 'name=XXXfetch',
  credentials: 'omit',
  headers: {'Content-Type': 'application/x-www-form-urlencoded'}
})
.then(response => console.log(response))
```
Against a canned query configured like this:
```yaml
databases:
  data:
    queries:
      add_name:
        sql: insert into names (name) values (:name)
        write: true
```
I haven't got it to work yet. Latest error is this one:
```
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
Traceback (most recent call last):
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 975, in route_path
    await response.asgi_send(send)
AttributeError: 'tuple' object has no attribute 'asgi_send'
INFO:     127.0.0.1:49938 - ""POST /data/add_name.json HTTP/1.1"" 500 Internal Server Error
```
It looks like I'm going to have to rethink how the `BaseView` code around tables, formats and hashes is structured in order to fix this. That's a big refactoring! I'm moving this to a new milestone for Datasette 0.46.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/882#issuecomment-652604569,https://api.github.com/repos/simonw/datasette/issues/882,652604569,MDEyOklzc3VlQ29tbWVudDY1MjYwNDU2OQ==,9599,simonw,2020-07-01T19:27:17Z,2020-07-01T19:27:17Z,OWNER,Don't forget to update the news in the README.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648673556,Release notes for 0.45,
https://github.com/simonw/datasette/issues/877#issuecomment-652597975,https://api.github.com/repos/simonw/datasette/issues/877,652597975,MDEyOklzc3VlQ29tbWVudDY1MjU5Nzk3NQ==,9599,simonw,2020-07-01T19:12:15Z,2020-07-01T19:12:15Z,OWNER,The latest release of https://github.com/simonw/datasette-auth-tokens (0.2) now supports SQL configuration of tokens.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652520496,https://api.github.com/repos/simonw/datasette/issues/877,652520496,MDEyOklzc3VlQ29tbWVudDY1MjUyMDQ5Ng==,9599,simonw,2020-07-01T16:26:52Z,2020-07-01T16:26:52Z,OWNER,Tokens get verified by plugins. So far there's only one: https://github.com/simonw/datasette-auth-tokens - which has you hard-coding plugins in a configuration file. I have a issue there to add support for database-backed tokens too: https://github.com/simonw/datasette-auth-tokens/issues/1,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-652182990,https://api.github.com/repos/simonw/datasette/issues/877,652182990,MDEyOklzc3VlQ29tbWVudDY1MjE4Mjk5MA==,9599,simonw,2020-07-01T04:29:38Z,2020-07-01T04:42:59Z,OWNER,"Have you tried the method described here? https://datasette.readthedocs.io/en/latest/internals.html#csrf-protection - I'm happy to bulk out that section of the documentation if that doesn't help solve your problem.

I just closed #835 which should make CSRF protection easier to work with - it won't interfere with requests without cookies or requests with `Authentication: Bearer token` tokens. See also https://github.com/simonw/asgi-csrf/issues/11

You can try out `pip install datasette==0.45a5` to get those features. Hopefully releasing a full 0.45 tomorrow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/812#issuecomment-652165709,https://api.github.com/repos/simonw/datasette/issues/812,652165709,MDEyOklzc3VlQ29tbWVudDY1MjE2NTcwOQ==,9599,simonw,2020-07-01T03:26:35Z,2020-07-01T03:26:35Z,OWNER,"This case may not be covered without extra work:
https://github.com/simonw/datasette/blob/3ec5b1abf6afa2d22a3378092809a1a8c0249d26/datasette/views/database.py#L122-L123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634112607,Ability to customize what happens when a view permission fails,
https://github.com/simonw/datasette/issues/812#issuecomment-652163450,https://api.github.com/repos/simonw/datasette/issues/812,652163450,MDEyOklzc3VlQ29tbWVudDY1MjE2MzQ1MA==,9599,simonw,2020-07-01T03:18:51Z,2020-07-01T03:20:28Z,OWNER,"This can be a plugin hook:

```python
@hookspec
def forbidden(datasette, request, message, send):
    ""Custom response for a 403 forbidden error""
```
If the hook returns a `Response` object, it will be returned to the user. Plugins are likely to want to return a redirect response.

Maybe the hook can instead use the `send` argument to respond to the request and return `True` which means ""I've responded to this""?

I'm going to leave `send` off for the moment - I can add that in the future if it turns out it would have been a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634112607,Ability to customize what happens when a view permission fails,
https://github.com/simonw/datasette/issues/880#issuecomment-652162722,https://api.github.com/repos/simonw/datasette/issues/880,652162722,MDEyOklzc3VlQ29tbWVudDY1MjE2MjcyMg==,9599,simonw,2020-07-01T03:16:07Z,2020-07-01T03:16:07Z,OWNER,The response from this will never be a 302 - it will always be a 200 if the response worked or a 400 for bad parameters or a 500 for errors. The body returned will always be in JSON format.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648637666,POST to /db/canned-query that returns JSON should be supported (for API clients),
https://github.com/simonw/datasette/issues/835#issuecomment-652159398,https://api.github.com/repos/simonw/datasette/issues/835,652159398,MDEyOklzc3VlQ29tbWVudDY1MjE1OTM5OA==,9599,simonw,2020-07-01T03:03:51Z,2020-07-01T03:03:51Z,OWNER,I'm going to add some tests for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/876#issuecomment-652106227,https://api.github.com/repos/simonw/datasette/issues/876,652106227,MDEyOklzc3VlQ29tbWVudDY1MjEwNjIyNw==,9599,simonw,2020-06-30T23:49:55Z,2020-06-30T23:50:04Z,OWNER,"Done: https://latest.datasette.io/-/patterns

<img width=""1004"" alt=""Datasette__Pattern_Portfolio"" src=""https://user-images.githubusercontent.com/9599/86187862-b680cb00-baf1-11ea-8905-8c257dd67665.png"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647879783,Add log out link to the pattern portfolio,
https://github.com/simonw/datasette/issues/879#issuecomment-652105722,https://api.github.com/repos/simonw/datasette/issues/879,652105722,MDEyOklzc3VlQ29tbWVudDY1MjEwNTcyMg==,9599,simonw,2020-06-30T23:48:06Z,2020-06-30T23:48:06Z,OWNER,Updated documentation: https://datasette.readthedocs.io/en/latest/pages.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648569227,Database page documentation still talks about hashes in URLs,
https://github.com/simonw/datasette/issues/832#issuecomment-652103895,https://api.github.com/repos/simonw/datasette/issues/832,652103895,MDEyOklzc3VlQ29tbWVudDY1MjEwMzg5NQ==,9599,simonw,2020-06-30T23:41:22Z,2020-06-30T23:41:22Z,OWNER,I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651999516,https://api.github.com/repos/simonw/datasette/issues/832,651999516,MDEyOklzc3VlQ29tbWVudDY1MTk5OTUxNg==,9599,simonw,2020-06-30T19:33:49Z,2020-06-30T21:34:59Z,OWNER,"Tests needed for this:

- If a user has view table but NOT view database / view instance, can they view the table page?
- If a user has view canned query but NOT view database / view instance, can they view the canned query page?
- If a user has view database but NOT view instance, can they view the database page?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651995453,https://api.github.com/repos/simonw/datasette/issues/832,651995453,MDEyOklzc3VlQ29tbWVudDY1MTk5NTQ1Mw==,9599,simonw,2020-06-30T19:25:13Z,2020-06-30T19:25:26Z,OWNER,I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651994978,https://api.github.com/repos/simonw/datasette/issues/832,651994978,MDEyOklzc3VlQ29tbWVudDY1MTk5NDk3OA==,9599,simonw,2020-06-30T19:24:12Z,2020-06-30T19:24:12Z,OWNER,"Hah... but check_permission` is a method on `BaseView`. Here are the various permission methods at the moment:

https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14

And on BaseView:

https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651993977,https://api.github.com/repos/simonw/datasette/issues/832,651993977,MDEyOklzc3VlQ29tbWVudDY1MTk5Mzk3Nw==,9599,simonw,2020-06-30T19:22:06Z,2020-06-30T19:22:06Z,OWNER,`permission_allowed` is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on `datasette.`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651993537,https://api.github.com/repos/simonw/datasette/issues/832,651993537,MDEyOklzc3VlQ29tbWVudDY1MTk5MzUzNw==,9599,simonw,2020-06-30T19:21:15Z,2020-06-30T19:21:15Z,OWNER,"I could rename `permission_allowed()` to `check_permission()` and have a complementary `check_permissions()` method.

This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary `permission_allowed()` alias which prints a deprecation warning to the console, then remove that at 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/832#issuecomment-651992737,https://api.github.com/repos/simonw/datasette/issues/832,651992737,MDEyOklzc3VlQ29tbWVudDY1MTk5MjczNw==,9599,simonw,2020-06-30T19:19:33Z,2020-06-30T19:20:02Z,OWNER,"I already have this method on Datasette:
```python
async def permission_allowed(self, actor, action, resource=None, default=False):
```
What would be a good method name that complements that and indicates ""check a list of permissions in order""? Should it even run against the request or should you have to hand it `request.actor`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636722501,Having view-table permission but NOT view-database should still grant access to /db/table,
https://github.com/simonw/datasette/issues/877#issuecomment-651984989,https://api.github.com/repos/simonw/datasette/issues/877,651984989,MDEyOklzc3VlQ29tbWVudDY1MTk4NDk4OQ==,9599,simonw,2020-06-30T19:03:25Z,2020-06-30T19:03:25Z,OWNER,Relevant: #835,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/877#issuecomment-651984355,https://api.github.com/repos/simonw/datasette/issues/877,651984355,MDEyOklzc3VlQ29tbWVudDY1MTk4NDM1NQ==,9599,simonw,2020-06-30T19:02:15Z,2020-06-30T19:02:15Z,OWNER,"https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf

> Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form.

Sounds like regular CSRF protection to me.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648421105,Consider dropping explicit CSRF protection entirely?,
https://github.com/simonw/datasette/issues/805#issuecomment-651302221,https://api.github.com/repos/simonw/datasette/issues/805,651302221,MDEyOklzc3VlQ29tbWVudDY1MTMwMjIyMQ==,9599,simonw,2020-06-29T19:02:45Z,2020-06-29T19:05:26Z,OWNER,"No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way.

So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good.

Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/805#issuecomment-651301202,https://api.github.com/repos/simonw/datasette/issues/805,651301202,MDEyOklzc3VlQ29tbWVudDY1MTMwMTIwMg==,9599,simonw,2020-06-29T19:00:37Z,2020-06-29T19:00:37Z,OWNER,"How about a blog? Pre-configured canned queries that are only available to `""root""`, plus datasette-template-sql and default templates for the index page and blog entry pages.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/875#issuecomment-651293559,https://api.github.com/repos/simonw/datasette/issues/875,651293559,MDEyOklzc3VlQ29tbWVudDY1MTI5MzU1OQ==,9599,simonw,2020-06-29T18:43:50Z,2020-06-29T18:43:50Z,OWNER,"<img width=""470"" alt=""_memory_"" src=""https://user-images.githubusercontent.com/9599/86043599-ca4b0500-b9fd-11ea-9378-a053e96cfde7.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/873#issuecomment-651203178,https://api.github.com/repos/simonw/datasette/issues/873,651203178,MDEyOklzc3VlQ29tbWVudDY1MTIwMzE3OA==,9599,simonw,2020-06-29T15:44:38Z,2020-06-29T15:44:54Z,OWNER,I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact `-p 0 --root` combination which isn't going to affect many people at all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-651193594,https://api.github.com/repos/simonw/datasette/issues/873,651193594,MDEyOklzc3VlQ29tbWVudDY1MTE5MzU5NA==,9599,simonw,2020-06-29T15:27:46Z,2020-06-29T15:27:46Z,OWNER,Uninstalling `datasette-debug-asgi` caused the server to startup correctly again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-651193131,https://api.github.com/repos/simonw/datasette/issues/873,651193131,MDEyOklzc3VlQ29tbWVudDY1MTE5MzEzMQ==,9599,simonw,2020-06-29T15:27:00Z,2020-06-29T15:27:00Z,OWNER,"Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486

Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding `lifespan=""on""` to the call to `uvicorn.run()`.

When I did that here:

`uvicorn.run(ds.app(), host=host, port=port, log_level=""info"", lifespan=""on"")`

The server failed to start with this error:

```
INFO:     Started server process [68849]
INFO:     Waiting for application startup.
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File "".../uvicorn/lifespan/on.py"", line 48, in main
    await app(scope, self.receive, self.send)
  File "".../uvicorn/middleware/proxy_headers.py"", line 45, in __call__
    return await self.app(scope, receive, send)
  File "".../datasette_debug_asgi.py"", line 9, in wrapped_app
    if scope[""path""] == ""/-/asgi-scope"":
KeyError: 'path'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650910137,https://api.github.com/repos/simonw/datasette/issues/873,650910137,MDEyOklzc3VlQ29tbWVudDY1MDkxMDEzNw==,9599,simonw,2020-06-29T05:16:32Z,2020-06-29T05:16:32Z,OWNER,I'm not convinced that function is ever actually being called - I added a `print()` statement to it and it's not executing. I don't think the tests cover it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650909476,https://api.github.com/repos/simonw/datasette/issues/873,650909476,MDEyOklzc3VlQ29tbWVudDY1MDkwOTQ3Ng==,9599,simonw,2020-06-29T05:14:08Z,2020-06-29T05:14:08Z,OWNER,"I already have a `AsgiLifespan` class:
https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905

It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894

Could that startup function also output the `--root` login URL, if needed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650909136,https://api.github.com/repos/simonw/datasette/issues/873,650909136,MDEyOklzc3VlQ29tbWVudDY1MDkwOTEzNg==,9599,simonw,2020-06-29T05:12:58Z,2020-06-29T05:12:58Z,OWNER,"On startup Datasette currently outputs:
```
INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
```
So the ASGI lifespan protocol is almost certainly the right way to solve this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650908854,https://api.github.com/repos/simonw/datasette/issues/873,650908854,MDEyOklzc3VlQ29tbWVudDY1MDkwODg1NA==,9599,simonw,2020-06-29T05:12:04Z,2020-06-29T05:12:04Z,OWNER,Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic `--root` link a second after the server starts up somehow.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650908534,https://api.github.com/repos/simonw/datasette/issues/873,650908534,MDEyOklzc3VlQ29tbWVudDY1MDkwODUzNA==,9599,simonw,2020-06-29T05:11:06Z,2020-06-29T05:11:06Z,OWNER,"Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650907323,https://api.github.com/repos/simonw/datasette/issues/873,650907323,MDEyOklzc3VlQ29tbWVudDY1MDkwNzMyMw==,9599,simonw,2020-06-29T05:07:16Z,2020-06-29T05:07:16Z,OWNER,"This line is interesting: is this a hook I can attach to somehow?
```python
        await self.lifespan.startup()
```
From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650906533,https://api.github.com/repos/simonw/datasette/issues/873,650906533,MDEyOklzc3VlQ29tbWVudDY1MDkwNjUzMw==,9599,simonw,2020-06-29T05:04:44Z,2020-06-29T05:04:44Z,OWNER,The challenge is... can we run our own custom code after that line has executed that has access to `server` and can hence access `server.servers[0].sockets[0].getsockname()[1]` to find the port?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650906318,https://api.github.com/repos/simonw/datasette/issues/873,650906318,MDEyOklzc3VlQ29tbWVudDY1MDkwNjMxOA==,9599,simonw,2020-06-29T05:04:04Z,2020-06-29T05:04:12Z,OWNER,"Within uvicorn it does this:
```python
            if port == 0:
                port = server.sockets[0].getsockname()[1]
```
That `server` variable is later stashed here:
```
self.servers = [server]
```
Where `self` is the instance of `class Server` - which is the class that Uvicorn instantiates and calls `.run()` on when we do `uvicorn.run()` here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/873#issuecomment-650905399,https://api.github.com/repos/simonw/datasette/issues/873,650905399,MDEyOklzc3VlQ29tbWVudDY1MDkwNTM5OQ==,9599,simonw,2020-06-29T05:01:03Z,2020-06-29T05:01:03Z,OWNER,This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647095487,"""datasette -p 0 --root"" gives the wrong URL",
https://github.com/simonw/datasette/issues/875#issuecomment-650899265,https://api.github.com/repos/simonw/datasette/issues/875,650899265,MDEyOklzc3VlQ29tbWVudDY1MDg5OTI2NQ==,9599,simonw,2020-06-29T04:34:32Z,2020-06-29T04:34:32Z,OWNER,"From https://github.com/simonw/datasette/issues/840#issuecomment-643454625
> Another problem: what to display in the ""you are logged in as"", since we don't dictate an actor design.
> 
> I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.
> 
> The default will look for the first available of the following keys:
> 
>     * display
>     * name
>     * username
>     * login
>     * id","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/875#issuecomment-650898808,https://api.github.com/repos/simonw/datasette/issues/875,650898808,MDEyOklzc3VlQ29tbWVudDY1MDg5ODgwOA==,9599,simonw,2020-06-29T04:32:31Z,2020-06-29T04:33:30Z,OWNER,"I could borrow the implementation for this from `datasette-auth-github`
https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html
```html+jinja
{% extends ""default:base.html"" %}

{% block extra_head %}
<style type=""text/css"">
.hd .logout {
    float: right;
    text-align: right;
    padding-left: 1em;
}
</style>
{% endblock %}

{% block nav %}
    {{ super() }}
    {% if auth and auth.username %}
        <p class=""logout"">
            <strong>{{ auth.username }}</strong> &middot; <a href=""/-/logout"">Log out</a>
        </p>
    {% endif %}
{% endblock %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",647103735,"""Logged in as: XXX - logout"" navigation item",
https://github.com/simonw/datasette/issues/840#issuecomment-650895874,https://api.github.com/repos/simonw/datasette/issues/840,650895874,MDEyOklzc3VlQ29tbWVudDY1MDg5NTg3NA==,9599,simonw,2020-06-29T04:18:59Z,2020-06-29T04:19:11Z,OWNER,"Now just need the ""Logged in as: XXX &lt;logout&gt;"" navigation item.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/simonw/datasette/issues/840#issuecomment-650891502,https://api.github.com/repos/simonw/datasette/issues/840,650891502,MDEyOklzc3VlQ29tbWVudDY1MDg5MTUwMg==,9599,simonw,2020-06-29T03:58:08Z,2020-06-29T03:58:08Z,OWNER,"Step one: a ""logout"" page at `/-/logout` - which shows you a single CSRF-protected ""logout"" button if you do a GET against it and logs you out if you do a POST against it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637966833,Log out mechanism for clearing ds_actor cookie,
https://github.com/simonw/datasette/issues/805#issuecomment-650891257,https://api.github.com/repos/simonw/datasette/issues/805,650891257,MDEyOklzc3VlQ29tbWVudDY1MDg5MTI1Nw==,9599,simonw,2020-06-29T03:56:48Z,2020-06-29T03:56:48Z,OWNER,Using `datasette-glitch` and the new https://github.com/simonw/datasette-write - currently running on `datasette==0.45a4` - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the `/-/write` interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/864#issuecomment-650847013,https://api.github.com/repos/simonw/datasette/issues/864,650847013,MDEyOklzc3VlQ29tbWVudDY1MDg0NzAxMw==,9599,simonw,2020-06-29T00:41:55Z,2020-06-29T00:41:55Z,OWNER,To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650846625,https://api.github.com/repos/simonw/datasette/issues/864,650846625,MDEyOklzc3VlQ29tbWVudDY1MDg0NjYyNQ==,9599,simonw,2020-06-29T00:39:47Z,2020-06-29T00:39:47Z,OWNER,"I think the fix is to move the `""show_messages""` variable to here:

https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650846473,https://api.github.com/repos/simonw/datasette/issues/864,650846473,MDEyOklzc3VlQ29tbWVudDY1MDg0NjQ3Mw==,9599,simonw,2020-06-29T00:39:04Z,2020-06-29T00:39:04Z,OWNER,"Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends `base.html`. For example, this code in a plugin:
```python
        return Response.html(
            await datasette.render_template(
                ""write.html"",
                {""databases"": databases, ""sql"": request.args.get(""sql"") or """"},
                request=request,
            )
        )
```
This won't display messages. The reason is that the messages are made available to the template context in the `BaseView.render()` method here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/864#issuecomment-650842514,https://api.github.com/repos/simonw/datasette/issues/864,650842514,MDEyOklzc3VlQ29tbWVudDY1MDg0MjUxNA==,9599,simonw,2020-06-29T00:12:59Z,2020-06-29T00:12:59Z,OWNER,"> I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
https://github.com/simonw/datasette/issues/870#issuecomment-650842381","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644309017,datasette.add_message() doesn't work inside plugins,
https://github.com/simonw/datasette/issues/870#issuecomment-650842381,https://api.github.com/repos/simonw/datasette/issues/870,650842381,MDEyOklzc3VlQ29tbWVudDY1MDg0MjM4MQ==,9599,simonw,2020-06-29T00:12:07Z,2020-06-29T00:12:07Z,OWNER,I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650838972,https://api.github.com/repos/simonw/datasette/issues/870,650838972,MDEyOklzc3VlQ29tbWVudDY1MDgzODk3Mg==,9599,simonw,2020-06-28T23:46:40Z,2020-06-28T23:46:40Z,OWNER,I'm going to create the single `Request()` instance in the `DatasetteRouter` class - at the beginning of the `route_path` method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650838691,https://api.github.com/repos/simonw/datasette/issues/870,650838691,MDEyOklzc3VlQ29tbWVudDY1MDgzODY5MQ==,9599,simonw,2020-06-28T23:44:12Z,2020-06-28T23:44:25Z,OWNER,"This code is interesting:

https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955

I want to change the signature of that `return await view(new_scope, receive, send)` method to instead take `(request, send)` - so I can have a single shared request object that's created just once per HTTP request.

The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared `Request` instance? Should its `.scope` be replaced with alternative scopes as it travels through the codebase?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650834666,https://api.github.com/repos/simonw/datasette/issues/870,650834666,MDEyOklzc3VlQ29tbWVudDY1MDgzNDY2Ng==,9599,simonw,2020-06-28T23:07:19Z,2020-06-28T23:07:19Z,OWNER,So now the problem is simpler: I need to get `BaseView` to a state where it can accept a shared `request` object and it can be used in conjunction with `register_routes()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650834251,https://api.github.com/repos/simonw/datasette/issues/870,650834251,MDEyOklzc3VlQ29tbWVudDY1MDgzNDI1MQ==,9599,simonw,2020-06-28T23:03:28Z,2020-06-28T23:03:28Z,OWNER,"I'm going to ditch that `AsgiView` class too, by combining it into `BaseView`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650820068,https://api.github.com/repos/simonw/datasette/issues/870,650820068,MDEyOklzc3VlQ29tbWVudDY1MDgyMDA2OA==,9599,simonw,2020-06-28T20:52:09Z,2020-06-28T20:53:00Z,OWNER,"Maybe I could add a `as_request_view` method as an alternative to `as_asgi`:

https://github.com/simonw/datasette/blob/a8bcafc1775c8a8655b365ae22a3d64f6361c74a/datasette/utils/asgi.py#L150-L174

Or I could teach the `Router` to spot the `dispatch_request` method and call it directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/847#issuecomment-650819895,https://api.github.com/repos/simonw/datasette/issues/847,650819895,MDEyOklzc3VlQ29tbWVudDY1MDgxOTg5NQ==,9599,simonw,2020-06-28T20:50:21Z,2020-06-28T20:50:21Z,OWNER,I'm happy enough with https://codecov.io/gh/simonw/datasette that I'm not going to spend any more time on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638259643,Take advantage of .coverage being a SQLite database,
https://github.com/simonw/datasette/issues/870#issuecomment-650818309,https://api.github.com/repos/simonw/datasette/issues/870,650818309,MDEyOklzc3VlQ29tbWVudDY1MDgxODMwOQ==,9599,simonw,2020-06-28T20:36:28Z,2020-06-28T20:36:52Z,OWNER,"Since `AsgiRouter` is only used as the super-class of the `DatasetteRouter` class maybe I should get rid of `AsgiRouter` entirely - no point in having a Datasette-specific subclass of it if the parent class isn't ever used by anything else.

I could also rename it to just `Router` which is a nicer name than `DatasetteRouter`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650818086,https://api.github.com/repos/simonw/datasette/issues/870,650818086,MDEyOklzc3VlQ29tbWVudDY1MDgxODA4Ng==,9599,simonw,2020-06-28T20:34:33Z,2020-06-28T20:34:33Z,OWNER,"The key to all of this may be the `DatasetteRouter` class. It deals with `scope` right now but if it internally dealt with `request` that could be enough to fix #864 by adding logic needed by the `.add_message()` mechanism.

https://github.com/simonw/datasette/blob/0991ea75cc7b265389aa8362414a305ba532d31a/datasette/app.py#L904-L938","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/870#issuecomment-650815278,https://api.github.com/repos/simonw/datasette/issues/870,650815278,MDEyOklzc3VlQ29tbWVudDY1MDgxNTI3OA==,9599,simonw,2020-06-28T20:09:07Z,2020-06-28T20:11:21Z,OWNER,"There's a lot of complex logic in the `DataView` class, which handles conditionally returning content as `.json` or as HTML or as `.csv`.

That view subclasses `AsgiView` which is itself request-aware, so maybe I don't need to reconsider how those classes work - just figure out how to hook them up with `register_routes`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646737558,Refactor default views to use register_routes,
https://github.com/simonw/datasette/issues/871#issuecomment-650812444,https://api.github.com/repos/simonw/datasette/issues/871,650812444,MDEyOklzc3VlQ29tbWVudDY1MDgxMjQ0NA==,9599,simonw,2020-06-28T19:43:27Z,2020-06-28T19:43:27Z,OWNER,"Currently:

> `_timestamp_epoch`
> 
> The number of seconds since the Unix epoch.
> 
> `_timestamp_date_utc`
> 
> The date in UTC, e.g. `2020-06-01`
> 
> `_timestamp_datetime_utc`
> 
> The ISO 8601 datetime in UTC, e.g. `2020-06-24T18:01:07Z`

I'm going to rename them to:

- `_now_epoch`
- `_now_date_utc`
- `_now_datetime_utc`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646840273,Rename the _timestamp magic parameters to _now,
https://github.com/simonw/datasette/issues/834#issuecomment-650811919,https://api.github.com/repos/simonw/datasette/issues/834,650811919,MDEyOklzc3VlQ29tbWVudDY1MDgxMTkxOQ==,9599,simonw,2020-06-28T19:38:50Z,2020-06-28T19:38:50Z,OWNER,"I have two plugins in progress that use this hook now:

- https://github.com/simonw/datasette-init creates tables and views on startup
- https://github.com/simonw/datasette-glitch outputs the login-as-root secret link on Glitch","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/805#issuecomment-650784162,https://api.github.com/repos/simonw/datasette/issues/805,650784162,MDEyOklzc3VlQ29tbWVudDY1MDc4NDE2Mg==,9599,simonw,2020-06-28T15:48:32Z,2020-06-28T15:48:32Z,OWNER,https://github.com/simonw/datasette-glitch is my new plugin that outputs the root login link on Glitch when the server starts.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/834#issuecomment-643657067,https://api.github.com/repos/simonw/datasette/issues/834,643657067,MDEyOklzc3VlQ29tbWVudDY0MzY1NzA2Nw==,9599,simonw,2020-06-13T17:59:42Z,2020-06-28T04:01:52Z,OWNER,Documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#startup-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637342551,startup() plugin hook,
https://github.com/simonw/datasette/issues/842#issuecomment-650684635,https://api.github.com/repos/simonw/datasette/issues/842,650684635,MDEyOklzc3VlQ29tbWVudDY1MDY4NDYzNQ==,9599,simonw,2020-06-28T03:30:31Z,2020-06-28T03:30:31Z,OWNER,Live demo: https://latest.datasette.io/fixtures/magic_parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/805#issuecomment-650681496,https://api.github.com/repos/simonw/datasette/issues/805,650681496,MDEyOklzc3VlQ29tbWVudDY1MDY4MTQ5Ng==,9599,simonw,2020-06-28T03:11:51Z,2020-06-28T03:11:51Z,OWNER,I can use magic parameters from #842 in this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",632724154,Writable canned queries live demo on Glitch,
https://github.com/simonw/datasette/issues/842#issuecomment-650679100,https://api.github.com/repos/simonw/datasette/issues/842,650679100,MDEyOklzc3VlQ29tbWVudDY1MDY3OTEwMA==,9599,simonw,2020-06-28T03:00:44Z,2020-06-28T03:00:44Z,OWNER,I'm going to add some canned queries to the `metadata.json` used by the live demo that illustrate this feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650678951,https://api.github.com/repos/simonw/datasette/issues/842,650678951,MDEyOklzc3VlQ29tbWVudDY1MDY3ODk1MQ==,9599,simonw,2020-06-28T02:59:52Z,2020-06-28T02:59:52Z,OWNER,"Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters

Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650648434,https://api.github.com/repos/simonw/datasette/issues/842,650648434,MDEyOklzc3VlQ29tbWVudDY1MDY0ODQzNA==,9599,simonw,2020-06-27T23:27:35Z,2020-06-27T23:37:38Z,OWNER,I'm going to rename `_request_X` to `_header_X` as that better reflects what it now does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/pull/868#issuecomment-650600606,https://api.github.com/repos/simonw/datasette/issues/868,650600606,MDEyOklzc3VlQ29tbWVudDY1MDYwMDYwNg==,9599,simonw,2020-06-27T18:44:28Z,2020-06-27T18:44:28Z,OWNER,"This is really exciting! Thanks so much for looking into this.

I'm interested in moving CI for this repo over to GitHub Actions, so I'd be fine with you getting this to work as an Action rather than through Travis. If you can get it working in Travis though I'll happily land that and figure out how to convert that to GitHub Actions later on.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646448486,initial windows ci setup,
https://github.com/simonw/datasette/issues/835#issuecomment-650598710,https://api.github.com/repos/simonw/datasette/issues/835,650598710,MDEyOklzc3VlQ29tbWVudDY1MDU5ODcxMA==,9599,simonw,2020-06-27T18:32:22Z,2020-06-27T18:32:22Z,OWNER,"Skipping CSRF on `Authorization: Bearer xxx` headers also makes sense for JWT applications, which tend to send JWTs using that form of header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637363686,Mechanism for skipping CSRF checks on API posts,
https://github.com/simonw/datasette/issues/842#issuecomment-650593122,https://api.github.com/repos/simonw/datasette/issues/842,650593122,MDEyOklzc3VlQ29tbWVudDY1MDU5MzEyMg==,9599,simonw,2020-06-27T18:03:02Z,2020-06-27T18:03:10Z,OWNER,"> Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.

This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.

https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650458857,https://api.github.com/repos/simonw/datasette/issues/842,650458857,MDEyOklzc3VlQ29tbWVudDY1MDQ1ODg1Nw==,9599,simonw,2020-06-27T00:11:04Z,2020-06-27T00:11:04Z,OWNER,Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650455793,https://api.github.com/repos/simonw/datasette/issues/842,650455793,MDEyOklzc3VlQ29tbWVudDY1MDQ1NTc5Mw==,9599,simonw,2020-06-26T23:57:30Z,2020-06-27T00:00:16Z,OWNER,"Maybe I should ship a default `_scope_headers_...` parameter instead, which reads from a dictionary of `scope[""headers""]` - https://asgi-scope.now.sh/ shows what those look like.

```
{'client': ('148.64.98.14', 0),
 'headers': [[b'host', b'asgi-scope.now.sh'],
             [b'x-forwarded-for', b'148.64.98.14'],
             [b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'],
             [b'x-forwarded-host', b'asgi-scope.now.sh'],
             [b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'],
             [b'x-real-ip', b'148.64.98.14'],
             [b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'],
             [b'upgrade-insecure-requests', b'1'],
             [b'x-vercel-trace', b'sfo1'],
             [b'x-forwarded-proto', b'https'],
             [b'accept-language', b'en-US,en;q=0.5'],
             [b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko'
              b'/20100101 Firefox/77.0'],
             [b'x-vercel-forwarded-for', b'148.64.98.14'],
             [b'accept-encoding', b'gzip, deflate, br'],
             [b'dnt', b'1'],
             [b'te', b'trailers']],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/',
 'query_string': b'',
 'raw_path': b'/',
 'root_path': '',
 'scheme': 'https',
 'server': ('asgi-scope.now.sh', 80),
 'type': 'http'}
```

I'm going to have `_request_X` actually mean ""find the first value for X in `scope[""headers""`]"" - with underscores converted to hyphens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-650455353,https://api.github.com/repos/simonw/datasette/issues/842,650455353,MDEyOklzc3VlQ29tbWVudDY1MDQ1NTM1Mw==,9599,simonw,2020-06-26T23:55:40Z,2020-06-26T23:55:40Z,OWNER,"`_request_ip` is actually quite hard to implement - should it take into account things like the `x-forwarded-for` header?

It probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment.

As such I think I'll leave that for a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/867#issuecomment-649931714,https://api.github.com/repos/simonw/datasette/issues/867,649931714,MDEyOklzc3VlQ29tbWVudDY0OTkzMTcxNA==,9599,simonw,2020-06-26T03:12:51Z,2020-06-26T03:12:51Z,OWNER,"Here's the relevant code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/datasette/app.py#L1057-L1070
And the relevant test code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/test_plugins.py#L567-L573

https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/plugins/my_plugin.py#L162-L196","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",645975649,register_routes() should support non-async view functions too,
https://github.com/simonw/datasette/issues/842#issuecomment-649014757,https://api.github.com/repos/simonw/datasette/issues/842,649014757,MDEyOklzc3VlQ29tbWVudDY0OTAxNDc1Nw==,9599,simonw,2020-06-24T19:15:46Z,2020-06-24T19:31:52Z,OWNER,I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646271834,https://api.github.com/repos/simonw/datasette/issues/842,646271834,MDEyOklzc3VlQ29tbWVudDY0NjI3MTgzNA==,9599,simonw,2020-06-18T19:49:41Z,2020-06-24T18:49:22Z,OWNER,"But then what kind of magic parameters might plugins want to add?

Here's a crazy idea: `_scrapedcontent_url` - it would look for the `url` column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function `scrapedcontent()` would need to optionally be sent the full row dictionary being inserted too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-646270702,https://api.github.com/repos/simonw/datasette/issues/842,646270702,MDEyOklzc3VlQ29tbWVudDY0NjI3MDcwMg==,9599,simonw,2020-06-18T19:47:19Z,2020-06-24T18:48:48Z,OWNER,"Brainstorming more potential magic parameters:

* `_actor_id`
* `_actor_name`
* `_request_ip`
* `_request_user_agent`
* `_cookie_cookiename`
* `_signedcookie_cookiename` - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, `_signedcookie_cookiename_mynamespace`. Might not need special signed cookie support since `actor` is already usually from a signed cookie.
* `_timestamp_unix` (not happy with these names yet)
* `_timestamp_localtime`
* `_timestamp_datetime`
* `_timestamp_utc`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/842#issuecomment-649000075,https://api.github.com/repos/simonw/datasette/issues/842,649000075,MDEyOklzc3VlQ29tbWVudDY0OTAwMDA3NQ==,9599,simonw,2020-06-24T18:46:36Z,2020-06-24T18:47:37Z,OWNER,"Another magic parameter that would be useful would be `_random`. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.

Maybe `_random_chars_128` to create a 128 character long random string (using `os.urandom(64).hex()`).

This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately `_random_chars:128` wouldn't work because these parameters are used in a SQLite query where `:` has special meaning: `insert into blah (secret) values (:_random_chars:128)` wouldn't make sense.

Actually this is already supported by the proposed design - `_random_chars_128` would become `random(""chars_128"")` so the `random()` function could split off the 128 itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638212085,Magic parameters for canned queries,
https://github.com/simonw/datasette/issues/865#issuecomment-648998264,https://api.github.com/repos/simonw/datasette/issues/865,648998264,MDEyOklzc3VlQ29tbWVudDY0ODk5ODI2NA==,9599,simonw,2020-06-24T18:43:02Z,2020-06-24T18:43:02Z,OWNER,Thanks for the bug report. Yes I think #838 may be the same issue. Will investigate.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""",
https://github.com/simonw/datasette/issues/858#issuecomment-648997857,https://api.github.com/repos/simonw/datasette/issues/858,648997857,MDEyOklzc3VlQ29tbWVudDY0ODk5Nzg1Nw==,9599,simonw,2020-06-24T18:42:10Z,2020-06-24T18:42:10Z,OWNER,I really need to get myself a Windows 10 development environment working so I can dig into this kind of bug properly. I have a gaming PC lying around that I could re-task for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,