{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706745236", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706745236, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc0NTIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T18:16:05Z", "updated_at": "2020-10-11T18:16:05Z", "author_association": "OWNER", "body": "Here's the `datasette-json-preview` plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706740250", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706740250, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc0MDI1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:40:48Z", "updated_at": "2020-10-11T17:43:07Z", "author_association": "OWNER", "body": "Building this plugin reminded me of an oddity of the `register_output_renderer()` plugin hook: one of the arguments that can be passed to it is `data`, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface.\r\n\r\nThat's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis.\r\n\r\nSo maybe that kind of stuff is re-implemented as \"extras\" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution.\r\n\r\nTo illustrate the problem (in this case the need to access `next_url`) here's my first prototype of the plugin:\r\n```python\r\nfrom datasette import hookimpl\r\nfrom datasette.utils.asgi import Response\r\n\r\n\r\n@hookimpl\r\ndef register_output_renderer(datasette):\r\n    return {\r\n        \"extension\": \"json-preview\",\r\n        \"render\": json_preview,\r\n    }\r\n\r\n\r\ndef json_preview(data, columns, rows):\r\n    next_url = data.get(\"next_url\")\r\n    headers = {}\r\n    if next_url:\r\n        headers[\"link\"] = '<{}>; rel=\"next\"'.format(next_url)\r\n    return Response.json([dict(zip(columns, row)) for row in rows], headers=headers)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706738020", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706738020, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczODAyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:23:18Z", "updated_at": "2020-10-11T17:23:48Z", "author_association": "OWNER", "body": "I'm going to prototype what it would look like if the default shape was a list of objects and `?_extra=` turns that into an object with a `rows` key, in a plugin. As a separate extension (maybe `.json-preview`).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706735341", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706735341, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczNTM0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:03:29Z", "updated_at": "2020-10-11T17:15:34Z", "author_association": "OWNER", "body": "Maybe `.jsonfull` becomes a new renderer that returns ALL of the defined `?_extra=` blocks.\r\n\r\nOr... `?_extra=all` turns on ALL of the available information blocks (some of which can come from plugins).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706735200", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706735200, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczNTIwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:02:11Z", "updated_at": "2020-10-11T17:14:51Z", "author_association": "OWNER", "body": "Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe `?_extra=count`.\r\n\r\nBased on that, the default JSON shape could look something like this:\r\n\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1}, {\"id\": 2}],\r\n  \"next\": \"2\",\r\n  \"next_url\": \"/db/table?_next=2\"\r\n}\r\n```\r\nAnd with `?_extra=count`:\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1}, {\"id\": 2}],\r\n  \"next\": \"2\",\r\n  \"next_url\": \"/db/table?_next=2\",\r\n  \"count\": 31\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706736541", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706736541, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczNjU0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:12:27Z", "updated_at": "2020-10-11T17:12:27Z", "author_association": "OWNER", "body": "The core issue that I keep reconsidering is whether the default `.json` representation should be an object or a list.\r\n\r\nArguments in favour of a list:\r\n\r\n- It's what I always want. Almost all of the code that I've written against the API myself uses `?_shape=array`.\r\n- It's really easy to use. You can pipe it to e.g. `sqlite-utils insert`, you can load it into JavaScript without thinking about it.\r\n\r\nArguments against:\r\n\r\n- Nowhere to put pagination or total counts. I added pagination to the `link:` HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like `\"truncated\": true` to indicate that more than 1000 results were returned and they were truncated.\r\n- An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format.\r\n\r\nBut maybe that last point is a positive? It ensures the default `.json` format remains completely predictable forever.\r\n\r\nIf `.json` DID default to an array of objects, the `?_shape=` argument could still be used to get back alternative formats.\r\n\r\nMaybe `.json?_extra=total` changes the shape of that default to be this instead:\r\n\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1}, {\"id\": 2}],\r\n  \"total\": 104\r\n}\r\n```\r\n\r\nThe thing I care about most though is `next_url`. That could be provided like so:\r\n\r\n`.json?_extra=total&_extra=next` - alternative syntax `.json?_extra=total,next`:\r\n\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1}, {\"id\": 2}],\r\n  \"total\": 104,\r\n  \"next\": \"2\",\r\n  \"next_url\": \"/db/table.json?_extra=total&_extra=next&_next=2\"\r\n}\r\n```\r\nThis is feeling a bit verbose for a common combination though.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706735280", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706735280, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczNTI4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:03:01Z", "updated_at": "2020-10-11T17:03:01Z", "author_association": "OWNER", "body": "Should that default also include `\"columns\"` as a list of strings? That would be duplicate data of the keys in the `\"rows\"` list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}