github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/issues/1723#issuecomment-1110330554 | https://api.github.com/repos/simonw/datasette/issues/1723 | 1110330554 | IC_kwDOBm6k_c5CLky6 | 9599 | 2022-04-26T23:06:20Z | 2022-04-26T23:06:20Z | OWNER | Deployed here: https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_trace=1&_facet=committer | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1216508080 | |
https://github.com/simonw/datasette/issues/1723#issuecomment-1110305790 | https://api.github.com/repos/simonw/datasette/issues/1723 | 1110305790 | IC_kwDOBm6k_c5CLev- | 9599 | 2022-04-26T22:19:04Z | 2022-04-26T22:19:04Z | OWNER | I realized that seeing the total time in queries wasn't enough to understand this, because if the queries were executed in serial or parallel it should still sum up to the same amount of SQL time (roughly). Instead I need to know how long the page took to render. But that's hard to display on the page since you can't measure it until rendering has finished! So I built an ASGI plugin to handle that measurement: https://github.com/simonw/datasette-total-page-time And with that plugin installed, `http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel2&_facet=other_fuel1&_parallel=1` (the parallel version) takes 377ms: <img width="543" alt="CleanShot 2022-04-26 at 15 17 38@2x" src="https://user-images.githubusercontent.com/9599/165401856-d592ed7a-0240-4514-b9d8-fb9e7d8c9629.png"> While `http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel2&_facet=other_fuel1` (the serial version) takes 762ms: <img width="543" alt="image" src="https://user-images.githubusercontent.com/9599/165401933-6d647014-4cab-4fbd-b9aa-958fc24ff435.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1216508080 | |
https://github.com/simonw/datasette/issues/1723#issuecomment-1110279869 | https://api.github.com/repos/simonw/datasette/issues/1723 | 1110279869 | IC_kwDOBm6k_c5CLYa9 | 9599 | 2022-04-26T21:45:39Z | 2022-04-26T21:45:39Z | OWNER | Getting some nice traces out of this: <img width="1384" alt="CleanShot 2022-04-26 at 14 45 21@2x" src="https://user-images.githubusercontent.com/9599/165397745-e8bfbe0a-306f-45bd-81f1-f5f6fc6422b9.png"> | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1216508080 | |
https://github.com/simonw/datasette/issues/1723#issuecomment-1110278577 | https://api.github.com/repos/simonw/datasette/issues/1723 | 1110278577 | IC_kwDOBm6k_c5CLYGx | 9599 | 2022-04-26T21:44:04Z | 2022-04-26T21:44:04Z | OWNER | And some simple benchmarks with `ab` - using the `?_parallel=1` hack to try it with and without a parallel `asyncio.gather()`: ``` ~ % ab -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done Server Software: uvicorn Server Hostname: 127.0.0.1 Server Port: 8001 Document Path: /global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2 Document Length: 314187 bytes Concurrency Level: 1 Time taken for tests: 68.279 seconds Complete requests: 100 Failed requests: 13 (Connect: 0, Receive: 0, Length: 13, Exceptions: 0) Total transferred: 31454937 bytes HTML transferred: 31418437 bytes Requests per second: 1.46 [#/sec] (mean) Time per request: 682.787 [ms] (mean) Time per request: 682.787 [ms] (mean, across all concurrent requests) Transfer rate: 449.89 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 621 683 68.0 658 993 Waiting: 620 682 68.0 657 992 Total: 621 683 68.0 658 993 Percentage of the requests served within a certain time (ms) 50% 658 66% 678 75% 687 80% 711 90% 763 95% 879 98% 926 99% 993 100% 993 (longest request) ---- In parallel: ~ % ab -n 100 'http://127.0.0.1:8001/global-power-plants/global-power-plants?_facet=primary_fuel&_facet=other_fuel1&_facet=other_fuel3&_facet=other_fuel2&_parallel=1' This is ApacheBench, Version 2.3 <$Revision: 1879490 $> Copyright 1… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1216508080 | |
https://github.com/simonw/datasette/issues/1723#issuecomment-1110278182 | https://api.github.com/repos/simonw/datasette/issues/1723 | 1110278182 | IC_kwDOBm6k_c5CLYAm | 9599 | 2022-04-26T21:43:34Z | 2022-04-26T21:43:34Z | OWNER | Here's the diff I'm using: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index d66adb8..f15ef1e 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1,3 +1,4 @@ +import asyncio import itertools import json @@ -5,6 +6,7 @@ import markupsafe from datasette.plugins import pm from datasette.database import QueryInterrupted +from datasette import tracer from datasette.utils import ( await_me_maybe, CustomRow, @@ -150,6 +152,16 @@ class TableView(DataView): default_labels=False, _next=None, _size=None, + ): + with tracer.trace_child_tasks(): + return await self._data_traced(request, default_labels, _next, _size) + + async def _data_traced( + self, + request, + default_labels=False, + _next=None, + _size=None, ): database_route = tilde_decode(request.url_vars["database"]) table_name = tilde_decode(request.url_vars["table"]) @@ -159,6 +171,20 @@ class TableView(DataView): raise NotFound("Database not found: {}".format(database_route)) database_name = db.name + # For performance profiling purposes, ?_parallel=1 turns on asyncio.gather + async def _gather_parallel(*args): + return await asyncio.gather(*args) + + async def _gather_sequential(*args): + results = [] + for fn in args: + results.append(await fn) + return results + + gather = ( + _gather_parallel if request.args.get("_parallel") else _gather_sequential + ) + # If this is a canned query, not a table, then dispatch to QueryView instead canned_query = await self.ds.get_canned_query( database_name, table_name, request.actor @@ -174,8 +200,12 @@ class TableView(DataView): write=bool(canned_query.get("write")), ) - is_view = bool(await db.ge… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1216508080 |