{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1110212021", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1110212021, "node_id": "IC_kwDOBm6k_c5CLH21", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T20:20:27Z", "updated_at": "2022-04-26T20:20:27Z", "author_association": "OWNER", "body": "Closing this because I have a good enough idea of the design for now - the details of the parameters can be figured out when I implement this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109309683", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109309683, "node_id": "IC_kwDOBm6k_c5CHrjz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T04:12:39Z", "updated_at": "2022-04-26T04:12:39Z", "author_association": "OWNER", "body": "I think the rough shape of the three plugin hooks is right. The detailed decisions that are needed concern what the parameters should be, which I think will mainly happen as part of:\r\n\r\n- #1715", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109306070", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109306070, "node_id": "IC_kwDOBm6k_c5CHqrW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T04:05:20Z", "updated_at": "2022-04-26T04:05:20Z", "author_association": "OWNER", "body": "The proposed plugin for annotations - allowing users to attach comments to database tables, columns and rows - would be a great application for all three of those `?_extra=` plugin hooks.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109305184", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109305184, "node_id": "IC_kwDOBm6k_c5CHqdg", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T04:03:35Z", "updated_at": "2022-04-26T04:03:35Z", "author_association": "OWNER", "body": "I bet there's all kinds of interesting potential extras that could be calculated by loading the results of the query into a Pandas DataFrame.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109200774", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109200774, "node_id": "IC_kwDOBm6k_c5CHQ-G", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T01:25:43Z", "updated_at": "2022-04-26T01:26:15Z", "author_association": "OWNER", "body": "Had a thought: if a custom HTML template is going to make use of stuff generated using these extras, it will need a way to tell Datasette to execute those extras even in the absence of the `?_extra=...` URL parameters.\r\n\r\nIs that necessary? Or should those kinds of plugins use the existing `extra_template_vars` hook instead?\r\n\r\nOr maybe the `extra_template_vars` hook gets redesigned so it can depend on other `extras` in some way?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109200335", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109200335, "node_id": "IC_kwDOBm6k_c5CHQ3P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T01:24:47Z", "updated_at": "2022-04-26T01:24:47Z", "author_association": "OWNER", "body": "Sketching out a `?_extra=statistics` table plugin:\r\n\r\n```python\r\nfrom datasette import hookimpl\r\n\r\n@hookimpl\r\ndef register_table_extras(datasette):\r\n    return [statistics]\r\n\r\nasync def statistics(datasette, query, columns, sql):\r\n    # ... need to figure out which columns are integer/floats\r\n    # then build and execute a SQL query that calculates sum/avg/etc for each column\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109174715", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109174715, "node_id": "IC_kwDOBm6k_c5CHKm7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T00:40:13Z", "updated_at": "2022-04-26T00:43:33Z", "author_association": "OWNER", "body": "Some of the things I'd like to use `?_extra=` for, that may or not make sense as plugins:\r\n\r\n- Performance breakdown information, maybe including explain output for a query/table\r\n- Information about the tables that were consulted in a query - imagine pulling in additional table metadata\r\n- Statistical aggregates against the full set of results. This may well be a Datasette core feature at some point in the future, but being able to provide it early as a plugin would be really cool.\r\n- For tables, what are the other tables they can join against?\r\n- Suggested facets\r\n- Facet results themselves\r\n- New custom facets I haven't thought of - though the `register_facet_classes` hook covers that already\r\n- Table schema\r\n- Table metadata\r\n- Analytics - how many times has this table been queried? Would be a plugin thing\r\n- For geospatial data, how about a GeoJSON polygon that represents the bounding box for all returned results? Effectively this is an extra aggregation.\r\n\r\nLooking at https://github-to-sqlite.dogsheep.net/github/commits.json?_labels=on&_shape=objects for inspiration.\r\n\r\nI think there's a separate potential mechanism in the future that lets you add custom columns to a table. This would affect `.csv` and the HTML presentation too, which makes it a different concept from the `?_extra=` hook that affects the JSON export (and the context that is fed to the HTML templates).", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109171871", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109171871, "node_id": "IC_kwDOBm6k_c5CHJ6f", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T00:34:48Z", "updated_at": "2022-04-26T00:34:48Z", "author_association": "OWNER", "body": "Let's try sketching out a `register_table_extras` plugin for something new.\r\n\r\nThe first idea I came up with suggests adding new fields to the individual row records that come back - my mental model for extras so far has been that they add new keys to the root object.\r\n\r\nSo if a table result looked like this:\r\n\r\n```json\r\n{\r\n  \"rows\": [\r\n    {\"id\": 1, \"name\": \"Cleo\"},\r\n    {\"id\": 2, \"name\": \"Suna\"}\r\n  ],\r\n  \"next_url\": null\r\n}\r\n```\r\nI was initially thinking that `?_extra=facets` would add a `\"facets\": {...}` key to that root object.\r\n\r\nHere's a plugin idea I came up with that would probably justify adding to the individual row objects instead:\r\n\r\n- `?_extra=check404s` - does an async `HEAD` request against every column value that looks like a URL and checks if it returns a 404\r\n\r\nThis could also work by adding a `\"check404s\": {\"url-here\": 200}` key to the root object though.\r\n\r\nI think I need some better plugin concepts before committing to this new hook. There's overlap between this and how I want the enrichments mechanism ([see here](https://simonwillison.net/2021/Jan/17/weeknotes-still-pretty-distracted/)) to work.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109165411", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109165411, "node_id": "IC_kwDOBm6k_c5CHIVj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T00:22:42Z", "updated_at": "2022-04-26T00:22:42Z", "author_association": "OWNER", "body": "Passing `pk_values` to the plugin hook feels odd. I think I'd pass a `row` object instead and let the code look up the primary key values on that row (by introspecting the primary keys for the table).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109164803", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109164803, "node_id": "IC_kwDOBm6k_c5CHIMD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T00:21:40Z", "updated_at": "2022-04-26T00:21:40Z", "author_association": "OWNER", "body": "What would the existing https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables feature look like if it was re-imagined as a `register_row_extras()` plugin?\r\n\r\nRough sketch, copying most of the code from https://github.com/simonw/datasette/blob/579f59dcec43a91dd7d404e00b87a00afd8515f2/datasette/views/row.py#L98\r\n\r\n```python\r\nfrom datasette import hookimpl\r\n\r\n@hookimpl\r\ndef register_row_extras(datasette):\r\n    return [foreign_key_tables]\r\n\r\nasync def foreign_key_tables(datasette, database, table, pk_values):\r\n    if len(pk_values) != 1:\r\n        return []\r\n    db = datasette.get_database(database)\r\n    all_foreign_keys = await db.get_all_foreign_keys()\r\n    foreign_keys = all_foreign_keys[table][\"incoming\"]\r\n    if len(foreign_keys) == 0:\r\n        return []\r\n\r\n    sql = \"select \" + \", \".join(\r\n        [\r\n            \"(select count(*) from {table} where {column}=:id)\".format(\r\n                table=escape_sqlite(fk[\"other_table\"]),\r\n                column=escape_sqlite(fk[\"other_column\"]),\r\n            )\r\n            for fk in foreign_keys\r\n        ]\r\n    )\r\n    try:\r\n        rows = list(await db.execute(sql, {\"id\": pk_values[0]}))\r\n    except QueryInterrupted:\r\n        # Almost certainly hit the timeout\r\n        return []\r\n\r\n    foreign_table_counts = dict(\r\n        zip(\r\n            [(fk[\"other_table\"], fk[\"other_column\"]) for fk in foreign_keys],\r\n            list(rows[0]),\r\n        )\r\n    )\r\n    foreign_key_tables = []\r\n    for fk in foreign_keys:\r\n        count = (\r\n            foreign_table_counts.get((fk[\"other_table\"], fk[\"other_column\"])) or 0\r\n        )\r\n        key = fk[\"other_column\"]\r\n        if key.startswith(\"_\"):\r\n            key += \"__exact\"\r\n        link = \"{}?{}={}\".format(\r\n            self.ds.urls.table(database, fk[\"other_table\"]),\r\n            key,\r\n            \",\".join(pk_values),\r\n        )\r\n        foreign_key_tables.append({**fk, **{\"count\": count, \"link\": link}})\r\n    return foreign_key_tables\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109162123", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109162123, "node_id": "IC_kwDOBm6k_c5CHHiL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T00:16:42Z", "updated_at": "2022-04-26T00:16:51Z", "author_association": "OWNER", "body": "Actually I'm going to imitate the existing `register_*` hooks:\r\n\r\n- `def register_output_renderer(datasette)`\r\n- `def register_facet_classes()`\r\n- `def register_routes(datasette)`\r\n- `def register_commands(cli)`\r\n- `def register_magic_parameters(datasette)`\r\n\r\nSo I'm going to call the new hooks:\r\n\r\n- `register_table_extras(datasette)`\r\n- `register_row_extras(datasette)`\r\n- `register_query_extras(datasette)`\r\n\r\nThey'll return a list of `async def` functions. The names of those functions will become the names of the extras.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109160226", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109160226, "node_id": "IC_kwDOBm6k_c5CHHEi", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T00:14:11Z", "updated_at": "2022-04-26T00:14:11Z", "author_association": "OWNER", "body": "There are four existing plugin hooks that include the word \"extra\" but use it to mean something else - to mean additional CSS/JS/variables to be injected into the page:\r\n\r\n- `def extra_css_urls(...)`\r\n- `def extra_js_urls(...)`\r\n- `def extra_body_script(...)`\r\n- `def extra_template_vars(...)`\r\n\r\nI think `extra_*` and `*_extras` are different enough that they won't be confused with each other.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109159307", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109159307, "node_id": "IC_kwDOBm6k_c5CHG2L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T00:12:28Z", "updated_at": "2022-04-26T00:12:28Z", "author_association": "OWNER", "body": "I'm going to keep table and row separate. So I think I need to add three new plugin hooks:\r\n\r\n- `table_extras()`\r\n- `row_extras()`\r\n- `query_extras()`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1720#issuecomment-1109158903", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1720", "id": 1109158903, "node_id": "IC_kwDOBm6k_c5CHGv3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-26T00:11:42Z", "updated_at": "2022-04-26T00:11:42Z", "author_association": "OWNER", "body": "Places this plugin hook (or hooks?) should be able to affect:\r\n\r\n- JSON for a table/view\r\n- JSON for a row\r\n- JSON for a canned query\r\n- JSON for a custom arbitrary query\r\n\r\nI'm going to combine those last two, which means there are three places. But maybe I can combine the table one and the row one as well?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1215174094, "label": "Design plugin hook for extras"}, "performed_via_github_app": null}