{"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-991827468", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 991827468, "node_id": "IC_kwDOBm6k_c47HhYM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-12T03:15:00Z", "updated_at": "2021-12-12T03:15:00Z", "author_association": "OWNER", "body": " I don't think this code is necessary any more: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399\r\n\r\nThat dates back from when Datasette was built on top of Sanic and Sanic didn't preserve those query parameters the way I needed it to:\r\n\r\nhttps://github.com/simonw/datasette/blob/1f69269fe93e4cd42e56890126cc0dbcf719c6cb/datasette/views/table.py#L202-L206", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-981172801", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 981172801, "node_id": "IC_kwDOBm6k_c46e4JB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-28T23:23:51Z", "updated_at": "2021-11-28T23:23:51Z", "author_association": "OWNER", "body": "(I could experiment with merging the two tables by adding a temporary undocumented `?_sql=` parameter to the in-progress table view that sets an alternative query instead of `select cols from table` - added bonus, this will force me to use introspection against the returned columns rather than mixing in the known columns for the specified table)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-993794247", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 993794247, "node_id": "IC_kwDOBm6k_c47PBjH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-14T17:09:40Z", "updated_at": "2021-12-14T17:09:40Z", "author_association": "OWNER", "body": "- `table_actions` should be an extra.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973700549", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973700549, "node_id": "IC_kwDOBm6k_c46CX3F", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T03:31:20Z", "updated_at": "2021-11-19T03:31:26Z", "author_association": "OWNER", "body": "... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs:\r\n- #1510.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-999870282", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 999870282, "node_id": "IC_kwDOBm6k_c47mM9K", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-22T20:45:56Z", "updated_at": "2021-12-22T20:46:08Z", "author_association": "OWNER", "body": "> New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.\r\n\r\nI wrote code to execute those in parallel using `asyncio.gather()` - which seems to work but causes the SQL run inside the parallel `async def` functions not to show up in the trace graph at all.\r\n\r\n```diff\r\ndiff --git a/datasette/views/table.py b/datasette/views/table.py\r\nindex 9808fd2..ec9db64 100644\r\n--- a/datasette/views/table.py\r\n+++ b/datasette/views/table.py\r\n@@ -1,3 +1,4 @@\r\n+import asyncio\r\n import urllib\r\n import itertools\r\n import json\r\n@@ -615,44 +616,37 @@ class TableView(RowTableShared):\r\n if request.args.get(\"_timelimit\"):\r\n extra_args[\"custom_time_limit\"] = int(request.args.get(\"_timelimit\"))\r\n \r\n- # Execute the main query!\r\n- results = await db.execute(sql, params, truncate=True, **extra_args)\r\n-\r\n- # Calculate the total count for this query\r\n- filtered_table_rows_count = None\r\n- if (\r\n- not db.is_mutable\r\n- and self.ds.inspect_data\r\n- and count_sql == f\"select count(*) from {table} \"\r\n- ):\r\n- # We can use a previously cached table row count\r\n- try:\r\n- filtered_table_rows_count = self.ds.inspect_data[database][\"tables\"][\r\n- table\r\n- ][\"count\"]\r\n- except KeyError:\r\n- pass\r\n-\r\n- # Otherwise run a select count(*) ...\r\n- if count_sql and filtered_table_rows_count is None and not nocount:\r\n- try:\r\n- count_rows = list(await db.execute(count_sql, from_sql_params))\r\n- filtered_table_rows_count = count_rows[0][0]\r\n- except QueryInterrupted:\r\n- pass\r\n-\r\n- # Faceting\r\n- if not self.ds.setting(\"allow_facet\") and any(\r\n- arg.startswith(\"_facet\") for arg in request.args\r\n- ):\r\n- raise BadRequest(\"_facet= is not allowed\")\r\n+ async def execute_count():\r\n+ # Calculate the total count for this query\r\n+ filtered_table_rows_count = None\r\n+ if (\r\n+ not db.is_mutable\r\n+ and self.ds.inspect_data\r\n+ and count_sql == f\"select count(*) from {table} \"\r\n+ ):\r\n+ # We can use a previously cached table row count\r\n+ try:\r\n+ filtered_table_rows_count = self.ds.inspect_data[database][\r\n+ \"tables\"\r\n+ ][table][\"count\"]\r\n+ except KeyError:\r\n+ pass\r\n+\r\n+ if count_sql and filtered_table_rows_count is None and not nocount:\r\n+ try:\r\n+ count_rows = list(await db.execute(count_sql, from_sql_params))\r\n+ filtered_table_rows_count = count_rows[0][0]\r\n+ except QueryInterrupted:\r\n+ pass\r\n+\r\n+ return filtered_table_rows_count\r\n+\r\n+ filtered_table_rows_count = await execute_count()\r\n \r\n # pylint: disable=no-member\r\n facet_classes = list(\r\n itertools.chain.from_iterable(pm.hook.register_facet_classes())\r\n )\r\n- facet_results = {}\r\n- facets_timed_out = []\r\n facet_instances = []\r\n for klass in facet_classes:\r\n facet_instances.append(\r\n@@ -668,33 +662,58 @@ class TableView(RowTableShared):\r\n )\r\n )\r\n \r\n- if not nofacet:\r\n- for facet in facet_instances:\r\n- (\r\n- instance_facet_results,\r\n- instance_facets_timed_out,\r\n- ) = await facet.facet_results()\r\n- for facet_info in instance_facet_results:\r\n- base_key = facet_info[\"name\"]\r\n- key = base_key\r\n- i = 1\r\n- while key in facet_results:\r\n- i += 1\r\n- key = f\"{base_key}_{i}\"\r\n- facet_results[key] = facet_info\r\n- facets_timed_out.extend(instance_facets_timed_out)\r\n-\r\n- # Calculate suggested facets\r\n- suggested_facets = []\r\n- if (\r\n- self.ds.setting(\"suggest_facets\")\r\n- and self.ds.setting(\"allow_facet\")\r\n- and not _next\r\n- and not nofacet\r\n- and not nosuggest\r\n- ):\r\n- for facet in facet_instances:\r\n- suggested_facets.extend(await facet.suggest())\r\n+ async def execute_suggested_facets():\r\n+ # Calculate suggested facets\r\n+ suggested_facets = []\r\n+ if (\r\n+ self.ds.setting(\"suggest_facets\")\r\n+ and self.ds.setting(\"allow_facet\")\r\n+ and not _next\r\n+ and not nofacet\r\n+ and not nosuggest\r\n+ ):\r\n+ for facet in facet_instances:\r\n+ suggested_facets.extend(await facet.suggest())\r\n+ return suggested_facets\r\n+\r\n+ async def execute_facets():\r\n+ facet_results = {}\r\n+ facets_timed_out = []\r\n+ if not self.ds.setting(\"allow_facet\") and any(\r\n+ arg.startswith(\"_facet\") for arg in request.args\r\n+ ):\r\n+ raise BadRequest(\"_facet= is not allowed\")\r\n+\r\n+ if not nofacet:\r\n+ for facet in facet_instances:\r\n+ (\r\n+ instance_facet_results,\r\n+ instance_facets_timed_out,\r\n+ ) = await facet.facet_results()\r\n+ for facet_info in instance_facet_results:\r\n+ base_key = facet_info[\"name\"]\r\n+ key = base_key\r\n+ i = 1\r\n+ while key in facet_results:\r\n+ i += 1\r\n+ key = f\"{base_key}_{i}\"\r\n+ facet_results[key] = facet_info\r\n+ facets_timed_out.extend(instance_facets_timed_out)\r\n+\r\n+ return facet_results, facets_timed_out\r\n+\r\n+ # Execute the main query, facets and facet suggestions in parallel:\r\n+ (\r\n+ results,\r\n+ suggested_facets,\r\n+ (facet_results, facets_timed_out),\r\n+ ) = await asyncio.gather(\r\n+ db.execute(sql, params, truncate=True, **extra_args),\r\n+ execute_suggested_facets(),\r\n+ execute_facets(),\r\n+ )\r\n+\r\n+ results = await db.execute(sql, params, truncate=True, **extra_args)\r\n \r\n # Figure out columns and rows for the query\r\n columns = [r[0] for r in results.description]\r\n```\r\nHere's the trace for `http://127.0.0.1:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2` with the missing facet and facet suggestion queries:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-974300823", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 974300823, "node_id": "IC_kwDOBm6k_c46EqaX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T18:18:32Z", "updated_at": "2021-11-19T18:18:32Z", "author_association": "OWNER", "body": "> This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.\r\n\r\nI can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973698917", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973698917, "node_id": "IC_kwDOBm6k_c46CXdl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T03:26:18Z", "updated_at": "2021-11-19T03:29:03Z", "author_association": "OWNER", "body": "A (likely incomplete) list of features on the table page:\r\n\r\n- [ ] Display table/database/instance metadata\r\n- [ ] Show count of all results\r\n- [ ] Display table of results\r\n - [ ] Special table display treatment for URLs, numbers\r\n - [ ] Allow plugins to modify table cells\r\n - [ ] Respect `?_col=` and `?_nocol=`\r\n- [ ] Show interface for filtering by columns and operations\r\n- [ ] Show search box, support executing FTS searches\r\n- [ ] Sort table by specified column\r\n- [ ] Paginate table\r\n- [ ] Show facet results\r\n- [ ] Show suggested facets\r\n- [ ] Link to available exports\r\n- [ ] Display schema for table\r\n - [ ] Maybe it should show the SQL for the query too?\r\n- [ ] Handle various non-obvious querystring options, like `?_where=` and `?_through=`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996225235", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996225235, "node_id": "IC_kwDOBm6k_c47YTDT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T21:58:24Z", "updated_at": "2021-12-16T21:58:41Z", "author_association": "OWNER", "body": "A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters.\r\n\r\nThe human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says:\r\n\r\n> 1 row where search matches \"dog\" and text2 = \"sara weasel\" sorted by pk\r\n\r\n(Also used in the page `