{"html_url": "https://github.com/simonw/datasette/issues/1355#issuecomment-1073362979", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1355", "id": 1073362979, "node_id": "IC_kwDOBm6k_c4_-jgj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-20T22:38:53Z", "updated_at": "2022-03-20T22:38:53Z", "author_association": "OWNER", "body": "Built a research prototype:\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 5c8101a..5cd3e63 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -1,6 +1,7 @@\r\n import asyncio\r\n import asgi_csrf\r\n import collections\r\n+import contextlib\r\n import datetime\r\n import functools\r\n import glob\r\n@@ -1490,3 +1491,11 @@ class DatasetteClient:\r\n             return await client.request(\r\n                 method, self._fix(path, avoid_path_rewrites), **kwargs\r\n             )\r\n+\r\n+    @contextlib.asynccontextmanager\r\n+    async def stream(self, method, path, **kwargs):\r\n+        async with httpx.AsyncClient(app=self.app) as client:\r\n+            print(\"async with as client\")\r\n+            async with client.stream(method, self._fix(path), **kwargs) as response:\r\n+                print(\"async with client.stream about to yield response\")\r\n+                yield response\r\ndiff --git a/datasette/cli.py b/datasette/cli.py\r\nindex 3c6e1b2..3025ead 100644\r\n--- a/datasette/cli.py\r\n+++ b/datasette/cli.py\r\n@@ -585,11 +585,19 @@ def serve(\r\n     asyncio.get_event_loop().run_until_complete(check_databases(ds))\r\n \r\n     if get:\r\n-        client = TestClient(ds)\r\n-        response = client.get(get)\r\n-        click.echo(response.text)\r\n-        exit_code = 0 if response.status == 200 else 1\r\n-        sys.exit(exit_code)\r\n+\r\n+        async def _run_get():\r\n+            print(\"_run_get\")\r\n+            async with ds.client.stream(\"GET\", get) as response:\r\n+                print(\"Got response:\", response)\r\n+                async for chunk in response.aiter_bytes(chunk_size=1024):\r\n+                    print(\"   chunk\")\r\n+                    sys.stdout.buffer.write(chunk)\r\n+                    sys.stdout.buffer.flush()\r\n+                exit_code = 0 if response.status_code == 200 else 1\r\n+                sys.exit(exit_code)\r\n+\r\n+        asyncio.get_event_loop().run_until_complete(_run_get())\r\n         return\r\n \r\n     # Start the server\r\n```\r\nBut for some reason it didn't appear to stream out the response - it would print this out:\r\n```\r\n% datasette covid.db --get '/covid/ny_times_us_counties.csv?_size=10&_stream=on'\r\n_run_get\r\nasync with as client\r\n```\r\nAnd then hang. I would expect it to start printing out chunks of CSV data here, but instead it looks like it waited for everything to be generated before returning anything to the console.\r\n\r\nNo idea why. I dropped this for the moment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 910088936, "label": "datasette --get should efficiently handle streaming CSV"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1355#issuecomment-853557439", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1355", "id": 853557439, "node_id": "MDEyOklzc3VlQ29tbWVudDg1MzU1NzQzOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-06-03T04:43:14Z", "updated_at": "2021-06-03T04:43:14Z", "author_association": "OWNER", "body": "It's using `TestClient` at the moment which is a wrapper around `httpx` (as of ) that uses the `@async_to_sync` decorator to hide the async nature.\r\n\r\nhttps://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/utils/testing.py#L102-L156\r\n\r\nMaybe the fix here is to switch the `--get` implementation to using `httpx` directly with https://www.python-httpx.org/async/#streaming-responses", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 910088936, "label": "datasette --get should efficiently handle streaming CSV"}, "performed_via_github_app": null}