issue_comments

Here's the datasette-json-preview plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview

Building this plugin reminded me of an oddity of the register_output_renderer() plugin hook: one of the arguments that can be passed to it is data, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface.

That's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis.

So maybe that kind of stuff is re-implemented as "extras" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution.

To illustrate the problem (in this case the need to access next_url) here's my first prototype of the plugin: ```python from datasette import hookimpl from datasette.utils.asgi import Response

@hookimpl def register_output_renderer(datasette): return { "extension": "json-preview", "render": json_preview, }

def json_preview(data, columns, rows): next_url = data.get("next_url") headers = {} if next_url: headers["link"] = '<{}>; rel="next"'.format(next_url) return Response.json([dict(zip(columns, row)) for row in rows], headers=headers) ```

I'm going to prototype what it would look like if the default shape was a list of objects and ?_extra= turns that into an object with a rows key, in a plugin. As a separate extension (maybe .json-preview).

Maybe .jsonfull becomes a new renderer that returns ALL of the defined ?_extra= blocks.

Or... ?_extra=all turns on ALL of the available information blocks (some of which can come from plugins).

Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe ?_extra=count.

Based on that, the default JSON shape could look something like this:

json { "rows": [{"id": 1}, {"id": 2}], "next": "2", "next_url": "/db/table?_next=2" } And with ?_extra=count: json { "rows": [{"id": 1}, {"id": 2}], "next": "2", "next_url": "/db/table?_next=2", "count": 31 }

The core issue that I keep reconsidering is whether the default .json representation should be an object or a list.

Arguments in favour of a list:

• It's what I always want. Almost all of the code that I've written against the API myself uses ?_shape=array.
• It's really easy to use. You can pipe it to e.g. sqlite-utils insert, you can load it into JavaScript without thinking about it.

Arguments against:

• Nowhere to put pagination or total counts. I added pagination to the link: HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like "truncated": true to indicate that more than 1000 results were returned and they were truncated.
• An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format.

But maybe that last point is a positive? It ensures the default .json format remains completely predictable forever.

If .json DID default to an array of objects, the ?_shape= argument could still be used to get back alternative formats.

Maybe .json?_extra=total changes the shape of that default to be this instead:

json { "rows": [{"id": 1}, {"id": 2}], "total": 104 }

The thing I care about most though is next_url. That could be provided like so:

.json?_extra=total&_extra=next - alternative syntax .json?_extra=total,next:

json { "rows": [{"id": 1}, {"id": 2}], "total": 104, "next": "2", "next_url": "/db/table.json?_extra=total&_extra=next&_next=2" } This is feeling a bit verbose for a common combination though.

Should that default also include "columns" as a list of strings? That would be duplicate data of the keys in the "rows" list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.