{"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-999870993", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 999870993, "node_id": "IC_kwDOBm6k_c47mNIR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-22T20:47:18Z", "updated_at": "2021-12-22T20:50:24Z", "author_association": "OWNER", "body": "The reason they aren't showing up in the traces is that traces are stored just for the currently executing `asyncio` task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25\r\n\r\nThis is so traces for other incoming requests don't end up mixed together. But there's no current mechanism to track async tasks that are effectively \"child tasks\" of the current request, and hence should be tracked the same.\r\n\r\nhttps://stackoverflow.com/a/69349501/6083 suggests that you pass the task ID as an argument to the child tasks that are executed using `asyncio.gather()` to work around this kind of problem.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-999870282", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 999870282, "node_id": "IC_kwDOBm6k_c47mM9K", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-22T20:45:56Z", "updated_at": "2021-12-22T20:46:08Z", "author_association": "OWNER", "body": "> New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.\r\n\r\nI wrote code to execute those in parallel using `asyncio.gather()` - which seems to work but causes the SQL run inside the parallel `async def` functions not to show up in the trace graph at all.\r\n\r\n```diff\r\ndiff --git a/datasette/views/table.py b/datasette/views/table.py\r\nindex 9808fd2..ec9db64 100644\r\n--- a/datasette/views/table.py\r\n+++ b/datasette/views/table.py\r\n@@ -1,3 +1,4 @@\r\n+import asyncio\r\n import urllib\r\n import itertools\r\n import json\r\n@@ -615,44 +616,37 @@ class TableView(RowTableShared):\r\n if request.args.get(\"_timelimit\"):\r\n extra_args[\"custom_time_limit\"] = int(request.args.get(\"_timelimit\"))\r\n \r\n- # Execute the main query!\r\n- results = await db.execute(sql, params, truncate=True, **extra_args)\r\n-\r\n- # Calculate the total count for this query\r\n- filtered_table_rows_count = None\r\n- if (\r\n- not db.is_mutable\r\n- and self.ds.inspect_data\r\n- and count_sql == f\"select count(*) from {table} \"\r\n- ):\r\n- # We can use a previously cached table row count\r\n- try:\r\n- filtered_table_rows_count = self.ds.inspect_data[database][\"tables\"][\r\n- table\r\n- ][\"count\"]\r\n- except KeyError:\r\n- pass\r\n-\r\n- # Otherwise run a select count(*) ...\r\n- if count_sql and filtered_table_rows_count is None and not nocount:\r\n- try:\r\n- count_rows = list(await db.execute(count_sql, from_sql_params))\r\n- filtered_table_rows_count = count_rows[0][0]\r\n- except QueryInterrupted:\r\n- pass\r\n-\r\n- # Faceting\r\n- if not self.ds.setting(\"allow_facet\") and any(\r\n- arg.startswith(\"_facet\") for arg in request.args\r\n- ):\r\n- raise BadRequest(\"_facet= is not allowed\")\r\n+ async def execute_count():\r\n+ # Calculate the total count for this query\r\n+ filtered_table_rows_count = None\r\n+ if (\r\n+ not db.is_mutable\r\n+ and self.ds.inspect_data\r\n+ and count_sql == f\"select count(*) from {table} \"\r\n+ ):\r\n+ # We can use a previously cached table row count\r\n+ try:\r\n+ filtered_table_rows_count = self.ds.inspect_data[database][\r\n+ \"tables\"\r\n+ ][table][\"count\"]\r\n+ except KeyError:\r\n+ pass\r\n+\r\n+ if count_sql and filtered_table_rows_count is None and not nocount:\r\n+ try:\r\n+ count_rows = list(await db.execute(count_sql, from_sql_params))\r\n+ filtered_table_rows_count = count_rows[0][0]\r\n+ except QueryInterrupted:\r\n+ pass\r\n+\r\n+ return filtered_table_rows_count\r\n+\r\n+ filtered_table_rows_count = await execute_count()\r\n \r\n # pylint: disable=no-member\r\n facet_classes = list(\r\n itertools.chain.from_iterable(pm.hook.register_facet_classes())\r\n )\r\n- facet_results = {}\r\n- facets_timed_out = []\r\n facet_instances = []\r\n for klass in facet_classes:\r\n facet_instances.append(\r\n@@ -668,33 +662,58 @@ class TableView(RowTableShared):\r\n )\r\n )\r\n \r\n- if not nofacet:\r\n- for facet in facet_instances:\r\n- (\r\n- instance_facet_results,\r\n- instance_facets_timed_out,\r\n- ) = await facet.facet_results()\r\n- for facet_info in instance_facet_results:\r\n- base_key = facet_info[\"name\"]\r\n- key = base_key\r\n- i = 1\r\n- while key in facet_results:\r\n- i += 1\r\n- key = f\"{base_key}_{i}\"\r\n- facet_results[key] = facet_info\r\n- facets_timed_out.extend(instance_facets_timed_out)\r\n-\r\n- # Calculate suggested facets\r\n- suggested_facets = []\r\n- if (\r\n- self.ds.setting(\"suggest_facets\")\r\n- and self.ds.setting(\"allow_facet\")\r\n- and not _next\r\n- and not nofacet\r\n- and not nosuggest\r\n- ):\r\n- for facet in facet_instances:\r\n- suggested_facets.extend(await facet.suggest())\r\n+ async def execute_suggested_facets():\r\n+ # Calculate suggested facets\r\n+ suggested_facets = []\r\n+ if (\r\n+ self.ds.setting(\"suggest_facets\")\r\n+ and self.ds.setting(\"allow_facet\")\r\n+ and not _next\r\n+ and not nofacet\r\n+ and not nosuggest\r\n+ ):\r\n+ for facet in facet_instances:\r\n+ suggested_facets.extend(await facet.suggest())\r\n+ return suggested_facets\r\n+\r\n+ async def execute_facets():\r\n+ facet_results = {}\r\n+ facets_timed_out = []\r\n+ if not self.ds.setting(\"allow_facet\") and any(\r\n+ arg.startswith(\"_facet\") for arg in request.args\r\n+ ):\r\n+ raise BadRequest(\"_facet= is not allowed\")\r\n+\r\n+ if not nofacet:\r\n+ for facet in facet_instances:\r\n+ (\r\n+ instance_facet_results,\r\n+ instance_facets_timed_out,\r\n+ ) = await facet.facet_results()\r\n+ for facet_info in instance_facet_results:\r\n+ base_key = facet_info[\"name\"]\r\n+ key = base_key\r\n+ i = 1\r\n+ while key in facet_results:\r\n+ i += 1\r\n+ key = f\"{base_key}_{i}\"\r\n+ facet_results[key] = facet_info\r\n+ facets_timed_out.extend(instance_facets_timed_out)\r\n+\r\n+ return facet_results, facets_timed_out\r\n+\r\n+ # Execute the main query, facets and facet suggestions in parallel:\r\n+ (\r\n+ results,\r\n+ suggested_facets,\r\n+ (facet_results, facets_timed_out),\r\n+ ) = await asyncio.gather(\r\n+ db.execute(sql, params, truncate=True, **extra_args),\r\n+ execute_suggested_facets(),\r\n+ execute_facets(),\r\n+ )\r\n+\r\n+ results = await db.execute(sql, params, truncate=True, **extra_args)\r\n \r\n # Figure out columns and rows for the query\r\n columns = [r[0] for r in results.description]\r\n```\r\nHere's the trace for `http://127.0.0.1:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2` with the missing facet and facet suggestion queries:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-999863269", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 999863269, "node_id": "IC_kwDOBm6k_c47mLPl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-22T20:35:41Z", "updated_at": "2021-12-22T20:37:13Z", "author_association": "OWNER", "body": "It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660-L671\r\n\r\nIt's used in facet suggestion logic here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/facets.py#L172-L178", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-999850191", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 999850191, "node_id": "IC_kwDOBm6k_c47mIDP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-22T20:29:38Z", "updated_at": "2021-12-22T20:29:38Z", "author_association": "OWNER", "body": "New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using `datasette-pretty-traces`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-999837569", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 999837569, "node_id": "IC_kwDOBm6k_c47mE-B", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-22T20:15:45Z", "updated_at": "2021-12-22T20:15:45Z", "author_association": "OWNER", "body": "Also the whole `special_args` v.s. `request.args` thing is pretty confusing, I think that might be an older code pattern back from when I was using Sanic.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-999837220", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 999837220, "node_id": "IC_kwDOBm6k_c47mE4k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-22T20:15:04Z", "updated_at": "2021-12-22T20:15:04Z", "author_association": "OWNER", "body": "I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-999831967", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 999831967, "node_id": "IC_kwDOBm6k_c47mDmf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-22T20:04:47Z", "updated_at": "2021-12-22T20:10:11Z", "author_association": "OWNER", "body": "I think I might be able to clean up a lot of the stuff in here using the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L87-L89\r\n\r\nThe catch with that hook - https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette - is that it gets called for every single cell. I don't want the overhead of looking up the foreign key relationships etc once for every value in a specific column.\r\n\r\nBut maybe I could extend the hook to include a shared cache that gets used for all of the cells in a specific table? Something like this:\r\n```python\r\nrender_cell(value, column, table, database, datasette, cache)\r\n```\r\n`cache` is a dictionary - and the same dictionary is passed to every call to that hook while rendering a specific page.\r\n\r\nIt's a bit of a gross hack though, and would it ever be useful for plugins outside of the default plugin in Datasette which does the foreign key stuff?\r\n\r\nIf I can think of one other potential application for this `cache` then I might implement it.\r\n\r\nNo, this optimization doesn't make sense: the most complex cell enrichment logic is the stuff that does a `select * from categories where id in (2, 5, 6)` query, using just the distinct set of IDs that are rendered on the current page. That's not going to fit in the `render_cell` hook no matter how hard I try to warp it into the right shape, because it needs full visibility of all of the results that are being rendered in order to collect those unique ID values.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-997472214", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 997472214, "node_id": "IC_kwDOBm6k_c47dDfW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-19T22:22:08Z", "updated_at": "2021-12-19T22:22:08Z", "author_association": "OWNER", "body": "I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was:\r\n\r\n```python\r\nclass TableQuery:\r\n def __init__(self, table, columns, pks, is_view=False, prev=None):\r\n self.table = table\r\n self.columns = columns\r\n self.pks = pks\r\n self.is_view = is_view\r\n self.prev = prev\r\n \r\n # These can be changed for different instances in the chain:\r\n self._where_clauses = None\r\n self._order_by = None\r\n self._page_size = None\r\n self._offset = None\r\n self._select_columns = None\r\n\r\n self.select_all_columns = '*'\r\n self.select_specified_columns = '*'\r\n\r\n @property\r\n def where_clauses(self):\r\n wheres = []\r\n current = self\r\n while current:\r\n if current._where_clauses is not None:\r\n wheres.extend(current._where_clauses)\r\n current = current.prev\r\n return list(reversed(wheres))\r\n\r\n def where(self, where):\r\n new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self)\r\n new_cls._where_clauses = [where]\r\n return new_cls\r\n \r\n @classmethod\r\n async def introspect(cls, db, table):\r\n return cls(\r\n table,\r\n columns = await db.table_columns(table),\r\n pks = await db.primary_keys(table),\r\n is_view = bool(await db.get_view_definition(table))\r\n )\r\n \r\n @property\r\n def sql_from(self):\r\n return f\"from {self.table}{self.sql_where}\"\r\n\r\n @property\r\n def sql_where(self):\r\n if not self.where_clauses:\r\n return \"\"\r\n else:\r\n return f\" where {' and '.join(self.where_clauses)}\"\r\n\r\n @property\r\n def sql_no_order_no_limit(self):\r\n return f\"select {self.select_all_columns} from {self.table}{self.sql_where}\"\r\n\r\n @property\r\n def sql(self):\r\n return f\"select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}\"\r\n\r\n @property\r\n def sql_count(self):\r\n return f\"select count(*) {self.sql_from}\"\r\n\r\n\r\n def __repr__(self):\r\n return f\"\"\r\n```\r\nUsage:\r\n```python\r\nfrom datasette.app import Datasette\r\nds = Datasette(memory=True, files=[\"/Users/simon/Dropbox/Development/datasette/fixtures.db\"])\r\ndb = ds.get_database(\"fixtures\")\r\nquery = await TableQuery.introspect(db, \"facetable\")\r\nprint(query.where(\"foo = bar\").where(\"baz = 1\").sql_count)\r\n# 'select count(*) from facetable where foo = bar and baz = 1'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-981153060", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 981153060, "node_id": "IC_kwDOBm6k_c46ezUk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-28T21:13:09Z", "updated_at": "2021-12-17T23:37:08Z", "author_association": "OWNER", "body": "Two new requirements inspired by work on the `datasette-table` (and `datasette-notebook`) projects:\r\n\r\n- #1533\r\n- #1534", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-997082845", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 997082845, "node_id": "IC_kwDOBm6k_c47bkbd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-17T23:10:09Z", "updated_at": "2021-12-17T23:10:17Z", "author_association": "OWNER", "body": "These changes so far are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996286104", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996286104, "node_id": "IC_kwDOBm6k_c47Yh6Y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-17T00:00:07Z", "updated_at": "2021-12-17T00:00:07Z", "author_association": "OWNER", "body": "Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996272906", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996272906, "node_id": "IC_kwDOBm6k_c47YesK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T23:27:42Z", "updated_at": "2021-12-16T23:27:42Z", "author_association": "OWNER", "body": "Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996264617", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996264617, "node_id": "IC_kwDOBm6k_c47Ycqp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T23:11:12Z", "updated_at": "2021-12-16T23:11:12Z", "author_association": "OWNER", "body": "I managed to extract both `_search=` and `_where=` out using a prototype of that hook. I wonder if it could extract the complex code for `?_next` too?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996250585", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996250585, "node_id": "IC_kwDOBm6k_c47YZPZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T22:43:37Z", "updated_at": "2021-12-16T22:45:07Z", "author_association": "OWNER", "body": "Ran into a problem prototyping that hook up for handling `?_where=` - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the `extra_wheres_for_ui` variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463\r\n\r\nMaybe change to this?\r\n\r\n```python\r\nclass FilterArguments(NamedTuple):\r\n where_clauses: List[str]\r\n params: Dict[str, Union[str, int, float]]\r\n human_descriptions: List[str]\r\n extra_context: Dict[str, Any]\r\n```\r\nThat might be necessary for `_search` too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996248713", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996248713, "node_id": "IC_kwDOBm6k_c47YYyJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T22:39:47Z", "updated_at": "2021-12-16T22:39:47Z", "author_association": "OWNER", "body": "The hook could return a named tuple like this one:\r\n```python\r\nfrom typing import NamedTuple, List, Optional, Union, Dict\r\n\r\nclass FilterArguments(NamedTuple):\r\n where_clauses: List[str]\r\n params: Dict[str, Union[str, int, float]]\r\n human_descriptions: List[str]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996240802", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996240802, "node_id": "IC_kwDOBm6k_c47YW2i", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T22:25:00Z", "updated_at": "2021-12-16T22:36:04Z", "author_association": "OWNER", "body": "I think that plugin hook would get given the `request` object (and `datasette` and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable.\r\n\r\n`filters_from_request(request, database, table, datasette)` perhaps? (Similar in name to `actor_from_request`).\r\n\r\n```python\r\n@hookspec\r\ndef filters_from_request(request, database, table, datasette):\r\n \"\"\"Return (where_clauses, params_dict, human_descriptions) based on the request\"\"\"\r\n```\r\n\r\nTurns out that's pretty much exactly what I implemented in 5116c4ec8aed5091e1f75415424b80f613518dc6 for #473:\r\n\r\n```python\r\n\r\n@hookspec\r\ndef table_filter():\r\n \"Custom filtering of the current table based on the request\"\r\n```\r\n```python\r\nTableFilter = namedtuple(\"TableFilter\", (\r\n \"human_description_extras\", \"where_clauses\", \"params\")\r\n)\r\n```\r\n```python\r\n # filter_arguments plugin hook support\r\n for awaitable_fn in pm.hook.table_filter():\r\n extras = await awaitable_fn(\r\n view=self, name=name, table=table, request=request\r\n )\r\n human_description_extras.extend(extras.human_description_extras)\r\n where_clauses.extend(extras.where_clauses)\r\n params.update(extras.params)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996227713", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996227713, "node_id": "IC_kwDOBm6k_c47YTqB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T22:02:35Z", "updated_at": "2021-12-16T22:03:55Z", "author_association": "OWNER", "body": "Is there an opportunity to refactor things using a new plugin hook here? Maybe the `register_filters` hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string.\r\n\r\nThat version dealt with `Filter` classes, but those might be a bit too low-level for this.\r\n\r\n`?_spatial_within=GEOJSON` was an interesting idea attached to that issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996225889", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996225889, "node_id": "IC_kwDOBm6k_c47YTNh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T21:59:32Z", "updated_at": "2021-12-16T22:00:42Z", "author_association": "OWNER", "body": "I added a ton of comments to the `data()` method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L322", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996225235", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996225235, "node_id": "IC_kwDOBm6k_c47YTDT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T21:58:24Z", "updated_at": "2021-12-16T21:58:41Z", "author_association": "OWNER", "body": "A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters.\r\n\r\nThe human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says:\r\n\r\n> 1 row where search matches \"dog\" and text2 = \"sara weasel\" sorted by pk\r\n\r\n(Also used in the page ``).\r\n\r\nThe code actually gathers three things:\r\n\r\n- Fragments of the `where` clause, for example ` \"text2\" = :p0`\r\n- Parameters, e.g. `{\"p0\": \"sara weasel\"}`\r\n- Human description components, e.g. `text2 = \"sara weasel\"`\r\n\r\nSome operations such as `?_where=` don't currently provide an extra human description component.\r\n\r\n`_where=` also doesn't populate a parameter, but maybe it could? Would be neat if in the future `?_where=foo+=+:bar` worked and added a `bar` input field to the screen, as seen with custom queries.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-996219117", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 996219117, "node_id": "IC_kwDOBm6k_c47YRjt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-16T21:47:51Z", "updated_at": "2021-12-16T21:49:24Z", "author_association": "OWNER", "body": "Should facets really not be displayed on pages past page one (where `?_next=` is set)? That made sense to me at the time, but I'm now having second thoughts about it.\r\n\r\nI guess it's a useful performance tweak for when crawlers keep hitting the `?_next=` link.\r\n\r\nActually it looks like facets DO display on subsequent pages, e.g. on https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 - but facet suggestions do not, thanks to this code: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L777-L785\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-994085710", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 994085710, "node_id": "IC_kwDOBm6k_c47QItO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-14T22:03:16Z", "updated_at": "2021-12-14T22:04:28Z", "author_association": "OWNER", "body": "There are actually four forms of SQL query used by the table page:\r\n\r\n- `from_sql` - just the `from table_name where ...`\r\n- `sql_no_order_no_limit` - used for faceting, `\"select {select_all_columns} from {table_name} {where}\"`\r\n- `sql` - the above but with order and limit clauses: `\"select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}\"`\r\n- `count_sql` used for the count, built out of `from_sql`: `\"select count(*) {from_sql}\"`\r\n\r\nI'm tempted to encapsulate those in a `Query` class.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-994042389", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 994042389, "node_id": "IC_kwDOBm6k_c47P-IV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-14T21:35:53Z", "updated_at": "2021-12-14T21:35:53Z", "author_association": "OWNER", "body": "Maybe a better way to approach this would be to focus on the JSON side of things - try to get a basic JSON version with `?_extra=` support working, then eventually build that up to the point where it can power the HTML version.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-993794247", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 993794247, "node_id": "IC_kwDOBm6k_c47PBjH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-14T17:09:40Z", "updated_at": "2021-12-14T17:09:40Z", "author_association": "OWNER", "body": "- `table_actions` should be an extra.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-993000787", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 993000787, "node_id": "IC_kwDOBm6k_c47L_1T", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-13T23:19:20Z", "updated_at": "2021-12-14T17:06:05Z", "author_association": "OWNER", "body": "Useful old comment here: https://github.com/simonw/datasette/issues/617#issuecomment-552253893\r\n\r\n> As noted in [#621 (comment)](https://github.com/simonw/datasette/issues/621#issuecomment-552253208) a common pattern in this method is blocks of code that append new items to the `where_clauses`, `params` and `extra_human_descriptions` arrays. This is a useful refactoring opportunity.\r\n> \r\n> Code that fits this pattern:\r\n> \r\n> * The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)`\r\n> * Code that handles `?_where=`: `where_clauses.extend(request.args[\"_where\"])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses\r\n> * The `_through=` code, see [Syntax for ?_through= that works as a form field #621](https://github.com/simonw/datasette/issues/621) for details\r\n> * The code that deals with `?_search=` FTS\r\n> \r\n> The keyset pagination code modifies `where_clauses` and `params` too, but I don't think it's quite going to work with the same abstraction that would cover the above examples.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-992833868", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 992833868, "node_id": "IC_kwDOBm6k_c47LXFM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-13T19:59:17Z", "updated_at": "2021-12-13T19:59:17Z", "author_association": "OWNER", "body": "Built a new plugin to help with this work by improving the display of `?_trace=1` output: https://datasette.io/plugins/datasette-pretty-traces\r\n\r\n![image](https://user-images.githubusercontent.com/9599/145879751-36621f43-ba68-4ccd-b14b-379ed8f2111a.png)\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-991978789", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 991978789, "node_id": "IC_kwDOBm6k_c47IGUl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-12T22:04:19Z", "updated_at": "2021-12-12T22:04:19Z", "author_association": "OWNER", "body": "Idea: in JSON output include a `warnings` block listing any _ parameters that were not recognized.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-991828014", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 991828014, "node_id": "IC_kwDOBm6k_c47Hhgu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-12T03:21:35Z", "updated_at": "2021-12-12T03:21:35Z", "author_association": "OWNER", "body": "No, removing that gave me the following test failure:\r\n```\r\ntests/test_table_api.py::test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] FAILED [100%]\r\n\r\n=============================================================================== FAILURES ================================================================================\r\n______________________________________ test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] ______________________________________\r\n\r\napp_client = <datasette.utils.testing.TestClient object at 0x10d45d2d0>, path = '/fixtures/simple_primary_key.json?content__exact=', expected_rows = [['3', '']]\r\n\r\n @pytest.mark.parametrize(\r\n \"path,expected_rows\",\r\n [\r\n (\"/fixtures/simple_primary_key.json?content=hello\", [[\"1\", \"hello\"]]),\r\n (\r\n \"/fixtures/simple_primary_key.json?content__contains=o\",\r\n [\r\n [\"1\", \"hello\"],\r\n [\"2\", \"world\"],\r\n [\"4\", \"RENDER_CELL_DEMO\"],\r\n ],\r\n ),\r\n (\"/fixtures/simple_primary_key.json?content__exact=\", [[\"3\", \"\"]]),\r\n (\r\n \"/fixtures/simple_primary_key.json?content__not=world\",\r\n [\r\n [\"1\", \"hello\"],\r\n [\"3\", \"\"],\r\n [\"4\", \"RENDER_CELL_DEMO\"],\r\n [\"5\", \"RENDER_CELL_ASYNC\"],\r\n ],\r\n ),\r\n ],\r\n )\r\n def test_table_filter_queries(app_client, path, expected_rows):\r\n response = app_client.get(path)\r\n> assert expected_rows == response.json[\"rows\"]\r\nE AssertionError: assert [['3', '']] == [['1', 'hello'],\\n ['2', 'world'],\\n ['3', ''],\\n ['4', 'RENDER_CELL_DEMO'],\\n ['5', 'RENDER_CELL_ASYNC']]\r\nE At index 0 diff: ['3', ''] != ['1', 'hello']\r\nE Right contains 4 more items, first extra item: ['2', 'world']\r\nE Full diff:\r\nE [\r\nE - ['1',\r\nE - 'hello'],\r\nE - ['2',\r\nE - 'world'],\r\nE ['3',\r\nE ''],\r\nE - ['4',\r\nE - 'RENDER_CELL_DEMO'],\r\nE - ['5',\r\nE - 'RENDER_CELL_ASYNC'],\r\nE ]\r\n\r\n/Users/simon/Dropbox/Development/datasette/tests/test_table_api.py:511: AssertionError\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-991827468", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 991827468, "node_id": "IC_kwDOBm6k_c47HhYM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-12T03:15:00Z", "updated_at": "2021-12-12T03:15:00Z", "author_association": "OWNER", "body": " I don't think this code is necessary any more: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399\r\n\r\nThat dates back from when Datasette was built on top of Sanic and Sanic didn't preserve those query parameters the way I needed it to:\r\n\r\nhttps://github.com/simonw/datasette/blob/1f69269fe93e4cd42e56890126cc0dbcf719c6cb/datasette/views/table.py#L202-L206", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-991823001", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 991823001, "node_id": "IC_kwDOBm6k_c47HgSZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-12T02:25:32Z", "updated_at": "2021-12-12T02:25:32Z", "author_association": "OWNER", "body": "The tests for `TableView` are currently mixed in with everything else in `tests/test_api.py` and `tests/html.py` - might be good to split those out into `test_table_html.py` and `test_table_api.py` since they're such a key part of how Datasette works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-991822853", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 991822853, "node_id": "IC_kwDOBm6k_c47HgQF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-12T02:24:00Z", "updated_at": "2021-12-12T02:24:00Z", "author_association": "OWNER", "body": "Rebuilding `TableView` from the ground up is proving not to be much fun. I'm going to explore starting the refactor of the existing code by separating out the bit that generates the SQL query from the rest of it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-991819781", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 991819781, "node_id": "IC_kwDOBm6k_c47HfgF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-12T01:53:10Z", "updated_at": "2021-12-12T01:53:10Z", "author_association": "OWNER", "body": "I have a hunch that the conclusion of this experiment may end up being that the `asyncinject` trick is kinda neat but the code will be easier to maintain (while still executing in parallel) if it's written using `asyncio.gather` directly instead.\r\n\r\nIt's possible `asyncinject` will end up being neat enough that I'll want to keep it though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-991285527", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 991285527, "node_id": "IC_kwDOBm6k_c47FdEX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-10T20:52:00Z", "updated_at": "2021-12-10T20:52:00Z", "author_association": "OWNER", "body": "If I break this up into `@inject` methods, what methods could I have and what would they do?\r\n\r\n- `resolve_path`: Use request path to resolve the database and table. Could handle hash URLs too (if I don't manage to extract those to a plugin) - would be nice if this could raise a redirect, but I think that will instead have to be one of the things it returns\r\n- `build_sql`: Builds the SQL query based on the querystring (and some DB introspection)\r\n- `execute_count`: Execute the `count(*)`\r\n- `execute_rows`: Execute the `limit 101` to fetch the rows\r\n- `execute_facets`: Execute all requested facets (could this do its own `asyncio.gather()` to run facets in parallel?)\r\n- `suggest_facets`: Execute facet suggestions\r\n\r\nAre there any plugin hooks that would make sense to execute in parallel? Actually there might be: I don't think `extra_template_vars`, `extra_css_urls`, `extra_js_urls`, `extra_body_script` depend on each other so it might be possible to execute them in a parallel chunk (at least any of them that return awaitables).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-981172801", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 981172801, "node_id": "IC_kwDOBm6k_c46e4JB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-28T23:23:51Z", "updated_at": "2021-11-28T23:23:51Z", "author_association": "OWNER", "body": "(I could experiment with merging the two tables by adding a temporary undocumented `?_sql=` parameter to the in-progress table view that sets an alternative query instead of `select cols from table` - added bonus, this will force me to use introspection against the returned columns rather than mixing in the known columns for the specified table)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-981172385", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 981172385, "node_id": "IC_kwDOBm6k_c46e4Ch", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-28T23:21:26Z", "updated_at": "2021-11-28T23:21:26Z", "author_association": "OWNER", "body": "Aside: is there any reason this work can't complete the long-running goal of merging the TableView and QueryView, such that most of the features available for tables become available for arbitrary queries too?\r\n\r\nI had already mentally committed to implementing facets for queries, but I just realized that filters could work too - using either a CTE or a nested query.\r\n\r\nPagination is the one holdout here, since table pagination uses keyset pagination over a known order. But maybe arbitrary queries can only be paginated off you order them first?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-981153186", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 981153186, "node_id": "IC_kwDOBm6k_c46ezWi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-28T21:13:50Z", "updated_at": "2021-11-28T21:13:50Z", "author_association": "OWNER", "body": "I'm also going to use the new `datasette-table` Web Component to help guide the design of the new API, which relates directly to this issue too:\r\n\r\n- #1532", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-974300823", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 974300823, "node_id": "IC_kwDOBm6k_c46EqaX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T18:18:32Z", "updated_at": "2021-11-19T18:18:32Z", "author_association": "OWNER", "body": "> This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.\r\n\r\nI can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-974285803", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 974285803, "node_id": "IC_kwDOBm6k_c46Emvr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T17:56:48Z", "updated_at": "2021-11-19T18:14:30Z", "author_association": "OWNER", "body": "Very confused by this piece of code here: https://github.com/simonw/datasette/blob/1c13e1af0664a4dfb1e69714c56523279cae09e4/datasette/views/table.py#L37-L63\r\n\r\nI added it in https://github.com/simonw/datasette/commit/754836eef043676e84626c4fd3cb993eed0d2976 - in the new world that should probably be replaced by pure JSON.\r\n\r\nAha - this comment explains it: https://github.com/simonw/datasette/issues/521#issuecomment-505279560\r\n\r\n> I think the trick is to redefine what a \"cell_row\" is. Each row is currently a list of cells:\r\n> \r\n> https://github.com/simonw/datasette/blob/6341f8cbc7833022012804dea120b838ec1f6558/datasette/views/table.py#L159-L163\r\n> \r\n> I can redefine the row (the `cells` variable in the above example) as a thing-that-iterates-cells (hence behaving like a list) but that also supports `__getitem__` access for looking up cell values if you know the name of the column.\r\n\r\nThe goal was to support neater custom templates like this:\r\n```html+jinja\r\n{% for row in display_rows %}\r\n <h2 class=\"scientist\">{{ row[\"First_Name\"] }} {{ row[\"Last_Name\"] }}</h2>\r\n ...\r\n```\r\nThis may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-974287570", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 974287570, "node_id": "IC_kwDOBm6k_c46EnLS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T17:59:33Z", "updated_at": "2021-11-19T17:59:33Z", "author_association": "OWNER", "body": "I'm going to try leaning into the `asyncinject` mechanism a bit here. One method can execute and return the raw rows. Another can turn that into the default minimal JSON representation. Then a third can take that (or take both) and use it to inflate out the JSON that the HTML template needs, with those extras and with the rendered cells from plugins.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973700549", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973700549, "node_id": "IC_kwDOBm6k_c46CX3F", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T03:31:20Z", "updated_at": "2021-11-19T03:31:26Z", "author_association": "OWNER", "body": "... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs:\r\n- #1510.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973700322", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973700322, "node_id": "IC_kwDOBm6k_c46CXzi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T03:30:30Z", "updated_at": "2021-11-19T03:30:30Z", "author_association": "OWNER", "body": "Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja.\r\n\r\nI'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well.\r\n\r\nIt would also help cleanup the kind-of-nasty pattern I use in the current `BaseView` where everything returns both a bunch of JSON-serializable data AND an awaitable function that then gets to add extra things to the HTML context.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973698917", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973698917, "node_id": "IC_kwDOBm6k_c46CXdl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T03:26:18Z", "updated_at": "2021-11-19T03:29:03Z", "author_association": "OWNER", "body": "A (likely incomplete) list of features on the table page:\r\n\r\n- [ ] Display table/database/instance metadata\r\n- [ ] Show count of all results\r\n- [ ] Display table of results\r\n - [ ] Special table display treatment for URLs, numbers\r\n - [ ] Allow plugins to modify table cells\r\n - [ ] Respect `?_col=` and `?_nocol=`\r\n- [ ] Show interface for filtering by columns and operations\r\n- [ ] Show search box, support executing FTS searches\r\n- [ ] Sort table by specified column\r\n- [ ] Paginate table\r\n- [ ] Show facet results\r\n- [ ] Show suggested facets\r\n- [ ] Link to available exports\r\n- [ ] Display schema for table\r\n - [ ] Maybe it should show the SQL for the query too?\r\n- [ ] Handle various non-obvious querystring options, like `?_where=` and `?_through=`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973699424", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973699424, "node_id": "IC_kwDOBm6k_c46CXlg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T03:27:49Z", "updated_at": "2021-11-19T03:27:49Z", "author_association": "OWNER", "body": "My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by `asyncinject`, but more importantly they can be used to build a much better JSON representation, where the default representation is lighter and `?_extra=x` options can be used to execute more expensive portions and add them to the response.\r\n\r\nSo the HTML version itself needs to be re-written to use those JSON extras.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973687978", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973687978, "node_id": "IC_kwDOBm6k_c46CUyq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T03:07:47Z", "updated_at": "2021-11-19T03:07:47Z", "author_association": "OWNER", "body": "I was wrong about that, you CAN over-ride default routes already.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973682389", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973682389, "node_id": "IC_kwDOBm6k_c46CTbV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T02:57:39Z", "updated_at": "2021-11-19T02:57:39Z", "author_association": "OWNER", "body": "Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though:\r\n\r\n- #1517 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1518#issuecomment-973681970", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1518", "id": 973681970, "node_id": "IC_kwDOBm6k_c46CTUy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T02:56:31Z", "updated_at": "2021-11-19T02:56:53Z", "author_association": "OWNER", "body": "Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2\r\n\r\n<img width=\"899\" alt=\"fixtures__roadside_attractions__4_rows_and__11__Liked___Twitter\" src=\"https://user-images.githubusercontent.com/9599/142557265-6e10c808-5898-49ed-a8e3-7b5207b2872a.png\">\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1058072543, "label": "Complete refactor of TableView and table.html template"}, "performed_via_github_app": null}