{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-755484384", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 755484384, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTQ4NDM4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T18:31:14Z", "updated_at": "2021-01-06T18:31:57Z", "author_association": "OWNER", "body": "In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3:\r\n```python\r\nimport pandas\r\ndf = pandas.read_json(\r\n  \"https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array\"\r\n)\r\n```\r\nAnd:\r\n```javascript\r\nd3 = require(\"d3@5\")\r\nrows = d3.json(\r\n    \"https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array\"\r\n)\r\n```\r\nOnce again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both `pandas.read_json()` and `d3.json()` - but it leaves nowhere to put the extra information like pagination and suchlike!\r\n\r\nEven given this I still think the correct default is an object with `\"rows\"`, `\"total\"` and `\"next_url\"` keys. I should commit to that and implement it - this thought exercise has been running for far too long.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-754958610", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 754958610, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NDk1ODYxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-05T23:15:24Z", "updated_at": "2021-01-05T23:15:24Z", "author_association": "OWNER", "body": "https://latest-with-plugins.datasette.io/fixtures/roadside_attraction_characteristics/1.json-preview returns a 500 error at the moment - a KeyError on 'filtered_table_rows_count'.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-720028476", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 720028476, "node_id": "MDEyOklzc3VlQ29tbWVudDcyMDAyODQ3Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-01T05:00:05Z", "updated_at": "2020-11-01T05:00:05Z", "author_association": "OWNER", "body": "This should be the key focus for Datasette 0.52.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-712986115", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 712986115, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMjk4NjExNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-20T16:28:46Z", "updated_at": "2020-10-20T16:29:51Z", "author_association": "OWNER", "body": "I think this all comes down to how the `?_extras=` mechanism works (see #262), as first hinted at in a30c5b220c15360d575e94b0e67f3255e120b916 (see commit message) when I added this long-forgotten undocumented feature: https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables\r\n\r\nExtras need to be able to execute additional SQL, since that would solve the problem we have now where the expensive \"suggested facets\" code runs on all `.json` output even when its results are not being shown.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-712590398", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 712590398, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMjU5MDM5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-20T05:03:46Z", "updated_at": "2020-10-20T05:04:09Z", "author_association": "OWNER", "body": "OK, https://latest-with-plugins.datasette.io/ is running that now - e.g. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions.json-preview or https://latest-with-plugins.datasette.io/fixtures/compound_three_primary_keys.json-preview\r\n\r\n```json\r\n{\r\n  \"rows\": [\r\n    {\r\n      \"pk\": 1,\r\n      \"name\": \"The Mystery Spot\",\r\n      \"address\": \"465 Mystery Spot Road, Santa Cruz, CA 95065\",\r\n      \"latitude\": 37.0167,\r\n      \"longitude\": -122.0024\r\n    },\r\n    {\r\n      \"pk\": 2,\r\n      \"name\": \"Winchester Mystery House\",\r\n      \"address\": \"525 South Winchester Boulevard, San Jose, CA 95128\",\r\n      \"latitude\": 37.3184,\r\n      \"longitude\": -121.9511\r\n    },\r\n    {\r\n      \"pk\": 3,\r\n      \"name\": \"Burlingame Museum of PEZ Memorabilia\",\r\n      \"address\": \"214 California Drive, Burlingame, CA 94010\",\r\n      \"latitude\": 37.5793,\r\n      \"longitude\": -122.3442\r\n    },\r\n    {\r\n      \"pk\": 4,\r\n      \"name\": \"Bigfoot Discovery Museum\",\r\n      \"address\": \"5497 Highway 9, Felton, CA 95018\",\r\n      \"latitude\": 37.0414,\r\n      \"longitude\": -122.0725\r\n    }\r\n  ],\r\n  \"total\": 4,\r\n  \"next_url\": null\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-712585921", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 712585921, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMjU4NTkyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-20T04:48:01Z", "updated_at": "2020-10-20T04:48:01Z", "author_association": "OWNER", "body": "I'll update `datasette-json-preview` with that now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-712585687", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 712585687, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMjU4NTY4Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-20T04:47:02Z", "updated_at": "2020-10-20T04:47:12Z", "author_association": "OWNER", "body": "Great point about CORS, I hadn't considered that.\r\n\r\nI think I'm going to keep the `Link:` header (added in #1014) because I quite enjoy using it with GitHub and WordPress, but I'm not going to have it be the default way of doing pagination. For the default shape I'm now leaning towards this:\r\n\r\n```json\r\n{\r\n  \"total\": 36,\r\n  \"rows\": [{\"id\": 1, \"name\": \"Cleo\"}],\r\n  \"next_url\": \"https://latest-with-plugins.datasette.io/fixtures/facetable.json?_next=5\"\r\n}\r\n```\r\n\r\nSo three keys: `total`, `rows` and `next_url`. Then extra keys can be added using `?_extra=` with various named bundles.", "reactions": "{\"total_count\": 3, \"+1\": 3, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-712569695", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 712569695, "node_id": "MDEyOklzc3VlQ29tbWVudDcxMjU2OTY5NQ==", "user": {"value": 222245, "label": "carlmjohnson"}, "created_at": "2020-10-20T03:45:48Z", "updated_at": "2020-10-20T03:46:14Z", "author_association": "NONE", "body": "I vote against headers. It has a lot of strikes against it: poor discoverability, new developers often don\u2019t know how to use them, makes CORS harder, makes it hard to use eg with JQ, needs ad hoc specification for each bit of metadata, etc. \r\n\r\nThe only advantage of headers is that you don\u2019t need to do .rows, but that\u2019s actually good as a data validation step anyway\u2014if .rows is missing assume there\u2019s an error and do your error handling path instead of parsing the rest.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706745236", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706745236, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc0NTIzNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T18:16:05Z", "updated_at": "2020-10-11T18:16:05Z", "author_association": "OWNER", "body": "Here's the `datasette-json-preview` plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706740250", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706740250, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjc0MDI1MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:40:48Z", "updated_at": "2020-10-11T17:43:07Z", "author_association": "OWNER", "body": "Building this plugin reminded me of an oddity of the `register_output_renderer()` plugin hook: one of the arguments that can be passed to it is `data`, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface.\r\n\r\nThat's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis.\r\n\r\nSo maybe that kind of stuff is re-implemented as \"extras\" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution.\r\n\r\nTo illustrate the problem (in this case the need to access `next_url`) here's my first prototype of the plugin:\r\n```python\r\nfrom datasette import hookimpl\r\nfrom datasette.utils.asgi import Response\r\n\r\n\r\n@hookimpl\r\ndef register_output_renderer(datasette):\r\n    return {\r\n        \"extension\": \"json-preview\",\r\n        \"render\": json_preview,\r\n    }\r\n\r\n\r\ndef json_preview(data, columns, rows):\r\n    next_url = data.get(\"next_url\")\r\n    headers = {}\r\n    if next_url:\r\n        headers[\"link\"] = '<{}>; rel=\"next\"'.format(next_url)\r\n    return Response.json([dict(zip(columns, row)) for row in rows], headers=headers)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706738020", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706738020, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczODAyMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:23:18Z", "updated_at": "2020-10-11T17:23:48Z", "author_association": "OWNER", "body": "I'm going to prototype what it would look like if the default shape was a list of objects and `?_extra=` turns that into an object with a `rows` key, in a plugin. As a separate extension (maybe `.json-preview`).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706735341", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706735341, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczNTM0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:03:29Z", "updated_at": "2020-10-11T17:15:34Z", "author_association": "OWNER", "body": "Maybe `.jsonfull` becomes a new renderer that returns ALL of the defined `?_extra=` blocks.\r\n\r\nOr... `?_extra=all` turns on ALL of the available information blocks (some of which can come from plugins).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706735200", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706735200, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczNTIwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:02:11Z", "updated_at": "2020-10-11T17:14:51Z", "author_association": "OWNER", "body": "Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe `?_extra=count`.\r\n\r\nBased on that, the default JSON shape could look something like this:\r\n\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1}, {\"id\": 2}],\r\n  \"next\": \"2\",\r\n  \"next_url\": \"/db/table?_next=2\"\r\n}\r\n```\r\nAnd with `?_extra=count`:\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1}, {\"id\": 2}],\r\n  \"next\": \"2\",\r\n  \"next_url\": \"/db/table?_next=2\",\r\n  \"count\": 31\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706736541", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706736541, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczNjU0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:12:27Z", "updated_at": "2020-10-11T17:12:27Z", "author_association": "OWNER", "body": "The core issue that I keep reconsidering is whether the default `.json` representation should be an object or a list.\r\n\r\nArguments in favour of a list:\r\n\r\n- It's what I always want. Almost all of the code that I've written against the API myself uses `?_shape=array`.\r\n- It's really easy to use. You can pipe it to e.g. `sqlite-utils insert`, you can load it into JavaScript without thinking about it.\r\n\r\nArguments against:\r\n\r\n- Nowhere to put pagination or total counts. I added pagination to the `link:` HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like `\"truncated\": true` to indicate that more than 1000 results were returned and they were truncated.\r\n- An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format.\r\n\r\nBut maybe that last point is a positive? It ensures the default `.json` format remains completely predictable forever.\r\n\r\nIf `.json` DID default to an array of objects, the `?_shape=` argument could still be used to get back alternative formats.\r\n\r\nMaybe `.json?_extra=total` changes the shape of that default to be this instead:\r\n\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1}, {\"id\": 2}],\r\n  \"total\": 104\r\n}\r\n```\r\n\r\nThe thing I care about most though is `next_url`. That could be provided like so:\r\n\r\n`.json?_extra=total&_extra=next` - alternative syntax `.json?_extra=total,next`:\r\n\r\n```json\r\n{\r\n  \"rows\": [{\"id\": 1}, {\"id\": 2}],\r\n  \"total\": 104,\r\n  \"next\": \"2\",\r\n  \"next_url\": \"/db/table.json?_extra=total&_extra=next&_next=2\"\r\n}\r\n```\r\nThis is feeling a bit verbose for a common combination though.\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-706735280", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 706735280, "node_id": "MDEyOklzc3VlQ29tbWVudDcwNjczNTI4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-11T17:03:01Z", "updated_at": "2020-10-11T17:03:01Z", "author_association": "OWNER", "body": "Should that default also include `\"columns\"` as a list of strings? That would be duplicate data of the keys in the `\"rows\"` list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-691554088", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 691554088, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTU1NDA4OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-12T21:39:03Z", "updated_at": "2020-09-12T21:39:03Z", "author_association": "OWNER", "body": "Plan: release a new release of Datasette (probably 0.49) with the new JSON API design, but provide a plugin called something like `datasette-api-0-48` which runs as ASGI wrapping middleware and internally rewrites incoming requests to e.g. `/db/table.json` to behave if they have the `?_extra=` params on them necessary to produce the 0.48 version of the JSON.\r\n\r\nAnyone who has built applications against 0.48 can install that plugin.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-691526878", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 691526878, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTUyNjg3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-12T18:21:41Z", "updated_at": "2020-09-12T18:22:20Z", "author_association": "OWNER", "body": "Would it be so bad if the default format had a `\"rows\"` key containing the array of rows? Maybe it wouldn't. The reason I always use `?_shape=array` is because I want an array of objects, rather than an array of arrays that I have to match up again with their columns.\r\n\r\nA default format that's an object rather than array also gives something for the `?_extra=` parameter to add its extras to.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-691526762", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 691526762, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTUyNjc2Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-12T18:20:19Z", "updated_at": "2020-09-12T18:20:19Z", "author_association": "OWNER", "body": "I'd like to revisit the idea of using `?_extra=x` to opt-in to extra blocks of JSON, from #262", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-691526489", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 691526489, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTUyNjQ4OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-12T18:17:16Z", "updated_at": "2020-09-12T18:17:16Z", "author_association": "OWNER", "body": "(I think I may have been over-thinking the details of this is for a couple of years now.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-691526416", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 691526416, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTUyNjQxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-12T18:16:36Z", "updated_at": "2020-09-12T18:16:36Z", "author_association": "OWNER", "body": "I'm going to hack together a preview of this in a branch and deploy it somewhere so people can see what I've got planned. Much easier to evaluate a working prototype than static examples.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-691323302", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 691323302, "node_id": "MDEyOklzc3VlQ29tbWVudDY5MTMyMzMwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-11T21:38:27Z", "updated_at": "2020-09-11T21:40:04Z", "author_association": "OWNER", "body": "Another idea: the default output could be the list of dicts:\r\n```json\r\n[\r\n  {\r\n    \"pk1\": \"a\",\r\n    \"pk2\": \"a\",\r\n    \"pk3\": \"a\",\r\n    \"content\": \"a-a-a\"\r\n  },\r\n  ...\r\n]\r\n```\r\nBUT... I could include pagination information in the HTTP headers - as seen in the WordPress REST API or the GitHub API:\r\n\r\n```\r\n~ % curl -s -i 'https://api.github.com/repos/simonw/datasette/commits' | head -n 40\r\nHTTP/1.1 200 OK\r\nserver: GitHub.com\r\ndate: Fri, 11 Sep 2020 21:37:46 GMT\r\ncontent-type: application/json; charset=utf-8\r\nstatus: 200 OK\r\ncache-control: public, max-age=60, s-maxage=60\r\nvary: Accept, Accept-Encoding, Accept, X-Requested-With\r\netag: W/\"71c99379743513394e880c6306b66bf9\"\r\nlast-modified: Fri, 11 Sep 2020 21:32:54 GMT\r\nx-github-media-type: github.v3; format=json\r\nlink: <https://api.github.com/repositories/107914493/commits?page=2>; rel=\"next\", <https://api.github.com/repositories/107914493/commits?page=44>; rel=\"last\"\r\naccess-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset\r\naccess-control-allow-origin: *\r\nstrict-transport-security: max-age=31536000; includeSubdomains; preload\r\nx-frame-options: deny\r\nx-content-type-options: nosniff\r\nx-xss-protection: 1; mode=block\r\nreferrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin\r\ncontent-security-policy: default-src 'none'\r\nX-Ratelimit-Limit: 60\r\nX-Ratelimit-Remaining: 55\r\nX-Ratelimit-Reset: 1599863850\r\nX-Ratelimit-Used: 5\r\nAccept-Ranges: bytes\r\nContent-Length: 118240\r\nX-GitHub-Request-Id: EC76:0EAD:313F40:5291A4:5F5BEE37\r\n\r\n[\r\n  {\r\n    \"sha\": \"d02f6151dae073135a22d0123e8abdc6cbef7c50\",\r\n    \"node_id\": \"MDY6Q29tbWl0MTA3OTE0NDkzOmQwMmY2MTUxZGFlMDczMTM1YTIyZDAxMjNlOGFiZGM2Y2JlZjdjNTA=\",\r\n    \"commit\": {\r\n```\r\nAlternative shapes would provide the pagination information (and other extensions) in the JSON, e.g.:\r\n\r\n`/squirrels/squirrels.json?_shape=paginated`\r\n```json\r\n{\r\n  \"rows\": [\r\n  {\r\n    \"pk1\": \"a\",\r\n    \"pk2\": \"a\",\r\n    \"pk3\": \"a\",\r\n    \"content\": \"a-a-a\"\r\n  }\r\n  ],\r\n  \"pagination\": {\r\n    \"next\": \"234\",\r\n    \"count\": 442\r\n  }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-636370064", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 636370064, "node_id": "MDEyOklzc3VlQ29tbWVudDYzNjM3MDA2NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-30T18:51:19Z", "updated_at": "2020-05-30T18:51:19Z", "author_association": "OWNER", "body": "https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_shape=array returns this:\r\n```json\r\n[\r\n  {\r\n    \"pk1\": \"a\",\r\n    \"pk2\": \"a\",\r\n    \"pk3\": \"a\",\r\n    \"content\": \"a-a-a\"\r\n  },\r\n  {\r\n    \"pk1\": \"a\",\r\n    \"pk2\": \"a\",\r\n    \"pk3\": \"b\",\r\n    \"content\": \"a-a-b\"\r\n  }\r\n]\r\n```\r\nThere's one big problem with this format: it doesn't provide any space for pagination information.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/782#issuecomment-636369978", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/782", "id": 636369978, "node_id": "MDEyOklzc3VlQ29tbWVudDYzNjM2OTk3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-05-30T18:50:31Z", "updated_at": "2020-05-30T18:50:31Z", "author_association": "OWNER", "body": "Here's the default JSON at the moment: https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2\r\n\r\n```json\r\n{\r\n  \"database\": \"fixtures\",\r\n  \"table\": \"compound_three_primary_keys\",\r\n  \"is_view\": false,\r\n  \"human_description_en\": \"\",\r\n  \"rows\": [\r\n    [\r\n      \"a\",\r\n      \"a\",\r\n      \"a\",\r\n      \"a-a-a\"\r\n    ],\r\n    [\r\n      \"a\",\r\n      \"a\",\r\n      \"b\",\r\n      \"a-a-b\"\r\n    ]\r\n  ],\r\n  \"truncated\": false,\r\n  \"filtered_table_rows_count\": 1001,\r\n  \"expanded_columns\": [],\r\n  \"expandable_columns\": [],\r\n  \"columns\": [\r\n    \"pk1\",\r\n    \"pk2\",\r\n    \"pk3\",\r\n    \"content\"\r\n  ],\r\n  \"primary_keys\": [\r\n    \"pk1\",\r\n    \"pk2\",\r\n    \"pk3\"\r\n  ],\r\n  \"units\": {},\r\n  \"query\": {\r\n    \"sql\": \"select pk1, pk2, pk3, content from compound_three_primary_keys order by pk1, pk2, pk3 limit 3\",\r\n    \"params\": {}\r\n  },\r\n  \"facet_results\": {},\r\n  \"suggested_facets\": [\r\n    {\r\n      \"name\": \"pk1\",\r\n      \"toggle_url\": \"http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk1\"\r\n    },\r\n    {\r\n      \"name\": \"pk2\",\r\n      \"toggle_url\": \"http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk2\"\r\n    },\r\n    {\r\n      \"name\": \"pk3\",\r\n      \"toggle_url\": \"http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_facet=pk3\"\r\n    }\r\n  ],\r\n  \"next\": \"a,a,b\",\r\n  \"next_url\": \"http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_next=a%2Ca%2Cb\",\r\n  \"query_ms\": 17.56119728088379,\r\n  \"source\": \"tests/fixtures.py\",\r\n  \"source_url\": \"https://github.com/simonw/datasette/blob/master/tests/fixtures.py\",\r\n  \"license\": \"Apache License 2.0\",\r\n  \"license_url\": \"https://github.com/simonw/datasette/blob/master/LICENSE\"\r\n}\r\n```\r\nThere's a lot of stuff in there. This increases the risk that future minor changes might break existing API consumers.\r\n\r\nIt returns rows as a list of lists of values, and expects you to correlate these with the list of columns. I originally designed it like this because I thought this was a more efficient representation than repeating the column names in a dictionary for every row. With hindsight this was a bad optimization - I _always_ use `?shape=array` because it's more convenient, and gzip encoding of the response means there's no bandwidth saving. Users who want that efficiency should request it using a custom `?_shape=`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 627794879, "label": "Redesign default .json format"}, "performed_via_github_app": null}