html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1016#issuecomment-706788010,https://api.github.com/repos/simonw/datasette/issues/1016,706788010,MDEyOklzc3VlQ29tbWVudDcwNjc4ODAxMA==,9599,2020-10-11T23:50:39Z,2020-10-11T23:50:39Z,OWNER,"For consistency can reuse the icon used on selected facets:
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718953669,
https://github.com/simonw/datasette/issues/1015#issuecomment-706756879,https://api.github.com/repos/simonw/datasette/issues/1015,706756879,MDEyOklzc3VlQ29tbWVudDcwNjc1Njg3OQ==,9599,2020-10-11T19:35:03Z,2020-10-11T19:35:03Z,OWNER,"Since plugins are installed via pip this would require Datasette to be restarted. This StackOverflow thread looks relevant to that: https://stackoverflow.com/questions/11329917/restart-python-script-from-within-itself
This recipe looks promising:
```python
import os
import sys
import psutil
import logging
def restart_program():
""""""Restarts the current program, with file objects and descriptors
cleanup
""""""
try:
p = psutil.Process(os.getpid())
for handler in p.get_open_files() + p.connections():
os.close(handler.fd)
except Exception, e:
logging.error(e)
python = sys.executable
os.execl(python, python, *sys.argv)
```
https://docs.python.org/3/library/os.html#os.execl says about `os.execl`:
> These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718910318,
https://github.com/simonw/datasette/issues/782#issuecomment-706745236,https://api.github.com/repos/simonw/datasette/issues/782,706745236,MDEyOklzc3VlQ29tbWVudDcwNjc0NTIzNg==,9599,2020-10-11T18:16:05Z,2020-10-11T18:16:05Z,OWNER,Here's the `datasette-json-preview` plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-706740250,https://api.github.com/repos/simonw/datasette/issues/782,706740250,MDEyOklzc3VlQ29tbWVudDcwNjc0MDI1MA==,9599,2020-10-11T17:40:48Z,2020-10-11T17:43:07Z,OWNER,"Building this plugin reminded me of an oddity of the `register_output_renderer()` plugin hook: one of the arguments that can be passed to it is `data`, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface.
That's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis.
So maybe that kind of stuff is re-implemented as ""extras"" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution.
To illustrate the problem (in this case the need to access `next_url`) here's my first prototype of the plugin:
```python
from datasette import hookimpl
from datasette.utils.asgi import Response
@hookimpl
def register_output_renderer(datasette):
return {
""extension"": ""json-preview"",
""render"": json_preview,
}
def json_preview(data, columns, rows):
next_url = data.get(""next_url"")
headers = {}
if next_url:
headers[""link""] = '<{}>; rel=""next""'.format(next_url)
return Response.json([dict(zip(columns, row)) for row in rows], headers=headers)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-706738020,https://api.github.com/repos/simonw/datasette/issues/782,706738020,MDEyOklzc3VlQ29tbWVudDcwNjczODAyMA==,9599,2020-10-11T17:23:18Z,2020-10-11T17:23:48Z,OWNER,"I'm going to prototype what it would look like if the default shape was a list of objects and `?_extra=` turns that into an object with a `rows` key, in a plugin. As a separate extension (maybe `.json-preview`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-706735341,https://api.github.com/repos/simonw/datasette/issues/782,706735341,MDEyOklzc3VlQ29tbWVudDcwNjczNTM0MQ==,9599,2020-10-11T17:03:29Z,2020-10-11T17:15:34Z,OWNER,"Maybe `.jsonfull` becomes a new renderer that returns ALL of the defined `?_extra=` blocks.
Or... `?_extra=all` turns on ALL of the available information blocks (some of which can come from plugins).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-706735200,https://api.github.com/repos/simonw/datasette/issues/782,706735200,MDEyOklzc3VlQ29tbWVudDcwNjczNTIwMA==,9599,2020-10-11T17:02:11Z,2020-10-11T17:14:51Z,OWNER,"Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe `?_extra=count`.
Based on that, the default JSON shape could look something like this:
```json
{
""rows"": [{""id"": 1}, {""id"": 2}],
""next"": ""2"",
""next_url"": ""/db/table?_next=2""
}
```
And with `?_extra=count`:
```json
{
""rows"": [{""id"": 1}, {""id"": 2}],
""next"": ""2"",
""next_url"": ""/db/table?_next=2"",
""count"": 31
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-706736541,https://api.github.com/repos/simonw/datasette/issues/782,706736541,MDEyOklzc3VlQ29tbWVudDcwNjczNjU0MQ==,9599,2020-10-11T17:12:27Z,2020-10-11T17:12:27Z,OWNER,"The core issue that I keep reconsidering is whether the default `.json` representation should be an object or a list.
Arguments in favour of a list:
- It's what I always want. Almost all of the code that I've written against the API myself uses `?_shape=array`.
- It's really easy to use. You can pipe it to e.g. `sqlite-utils insert`, you can load it into JavaScript without thinking about it.
Arguments against:
- Nowhere to put pagination or total counts. I added pagination to the `link:` HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like `""truncated"": true` to indicate that more than 1000 results were returned and they were truncated.
- An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format.
But maybe that last point is a positive? It ensures the default `.json` format remains completely predictable forever.
If `.json` DID default to an array of objects, the `?_shape=` argument could still be used to get back alternative formats.
Maybe `.json?_extra=total` changes the shape of that default to be this instead:
```json
{
""rows"": [{""id"": 1}, {""id"": 2}],
""total"": 104
}
```
The thing I care about most though is `next_url`. That could be provided like so:
`.json?_extra=total&_extra=next` - alternative syntax `.json?_extra=total,next`:
```json
{
""rows"": [{""id"": 1}, {""id"": 2}],
""total"": 104,
""next"": ""2"",
""next_url"": ""/db/table.json?_extra=total&_extra=next&_next=2""
}
```
This is feeling a bit verbose for a common combination though.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/782#issuecomment-706735280,https://api.github.com/repos/simonw/datasette/issues/782,706735280,MDEyOklzc3VlQ29tbWVudDcwNjczNTI4MA==,9599,2020-10-11T17:03:01Z,2020-10-11T17:03:01Z,OWNER,"Should that default also include `""columns""` as a list of strings? That would be duplicate data of the keys in the `""rows""` list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/1014#issuecomment-706631006,https://api.github.com/repos/simonw/datasette/issues/1014,706631006,MDEyOklzc3VlQ29tbWVudDcwNjYzMTAwNg==,9599,2020-10-11T00:36:43Z,2020-10-11T00:36:43Z,OWNER,"Demo using [paginate-json](https://github.com/simonw/paginate-json):
```
% paginate-json 'https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array' | jq '. | length'
https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cd%2Cv
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ch%2Cr
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cl%2Cn
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cp%2Cj
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ct%2Cf
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cx%2Cb
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ca%2Cx
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ce%2Ct
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ci%2Cp
http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Cm%2Cl
1001
```
New documentation: https://docs.datasette.io/en/latest/json_api.html#pagination","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718723543,