{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-755134771", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 755134771, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTEzNDc3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T07:28:01Z", "updated_at": "2021-01-06T07:28:01Z", "author_association": "OWNER", "body": "With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the `stream_rows()` method could output `[` first, then each row followed by a comma, then `]` after the very last row.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-755133937", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 755133937, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTEzMzkzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T07:25:48Z", "updated_at": "2021-01-06T07:26:43Z", "author_association": "OWNER", "body": "Idea: instead of returning a dictionary, `register_output_renderer` could return an object. The object could have the following properties:\r\n\r\n- `.extension` - the extension to use\r\n- `.can_render(...)` - says if it can render this\r\n- `.can_stream(...)` - says if streaming is supported\r\n- `async .stream_rows(rows_iterator, send)` - method that loops through all rows and uses `send` to send them to the response in the correct format\r\n\r\nI can then deprecate the existing `dict` return type for 1.0.", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-755128038", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 755128038, "node_id": "MDEyOklzc3VlQ29tbWVudDc1NTEyODAzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-01-06T07:10:22Z", "updated_at": "2021-01-06T07:10:22Z", "author_association": "OWNER", "body": "Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:\r\n\r\n    pandas.read_json(\"https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on\", lines=True)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-732544590", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 732544590, "node_id": "MDEyOklzc3VlQ29tbWVudDczMjU0NDU5MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-24T02:22:55Z", "updated_at": "2020-11-24T02:22:55Z", "author_association": "OWNER", "body": "The trick I'm using here is to follow the `next_url` in order to paginate through all of the matching results. The loop calls the `data()` method multiple times, once for each page of results: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L304-L307", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1101#issuecomment-732543700", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1101", "id": 732543700, "node_id": "MDEyOklzc3VlQ29tbWVudDczMjU0MzcwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-11-24T02:20:30Z", "updated_at": "2020-11-24T02:20:30Z", "author_association": "OWNER", "body": "Current design: https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette\r\n\r\n```python\r\n@hookimpl\r\ndef register_output_renderer(datasette):\r\n    return {\r\n        \"extension\": \"test\",\r\n        \"render\": render_demo,\r\n        \"can_render\": can_render_demo,  # Optional\r\n    }\r\n```\r\nWhere `render_demo` looks something like this:\r\n```python\r\nasync def render_demo(datasette, columns, rows):\r\n    db = datasette.get_database()\r\n    result = await db.execute(\"select sqlite_version()\")\r\n    first_row = \" | \".join(columns)\r\n    lines = [first_row]\r\n    lines.append(\"=\" * len(first_row))\r\n    for row in rows:\r\n        lines.append(\" | \".join(row))\r\n    return Response(\r\n        \"\\n\".join(lines),\r\n        content_type=\"text/plain; charset=utf-8\",\r\n        headers={\"x-sqlite-version\": result.first()[0]}\r\n    )\r\n```\r\nMeanwhile here's where the CSV streaming mode is implemented: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L297-L380", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 749283032, "label": "register_output_renderer() should support streaming data"}, "performed_via_github_app": null}