{"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-973678931", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 973678931, "node_id": "IC_kwDOBm6k_c46CSlT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T02:51:17Z", "updated_at": "2021-11-19T02:51:17Z", "author_association": "OWNER", "body": "OK, I managed to get a table to render! Here's the code I used - I had to copy a LOT of stuff. https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2\r\n\r\nI'm going to move this work into a new, separate issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-973635157", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 973635157, "node_id": "IC_kwDOBm6k_c46CH5V", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T01:07:08Z", "updated_at": "2021-11-19T01:07:08Z", "author_association": "OWNER", "body": "This exercise is proving so useful in getting my head around how the enormous and complex `TableView` class works again.\r\n\r\nHere's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work:\r\n\r\n```python\r\nfrom datasette.database import QueryInterrupted\r\nfrom datasette.utils import escape_sqlite\r\nfrom datasette.utils.asgi import Response, NotFound, Forbidden\r\nfrom datasette.views.base import DatasetteError\r\nfrom datasette import hookimpl\r\nfrom asyncinject import AsyncInject, inject\r\nfrom pprint import pformat\r\n\r\n\r\nclass Table(AsyncInject):\r\n    @inject\r\n    async def database(self, request, datasette):\r\n        # TODO: all that nasty hash resolving stuff can go here\r\n        db_name = request.url_vars[\"db_name\"]\r\n        try:\r\n            db = datasette.databases[db_name]\r\n        except KeyError:\r\n            raise NotFound(f\"Database '{db_name}' does not exist\")\r\n        return db\r\n\r\n    @inject\r\n    async def table_and_format(self, request, database, datasette):\r\n        table_and_format = request.url_vars[\"table_and_format\"]\r\n        # TODO: be a lot smarter here\r\n        if \".\" in table_and_format:\r\n            return table_and_format.split(\".\", 2)\r\n        else:\r\n            return table_and_format, \"html\"\r\n\r\n    @inject\r\n    async def main(self, request, database, table_and_format, datasette):\r\n        # TODO: if this is actually a canned query, dispatch to it\r\n\r\n        table, format = table_and_format\r\n\r\n        is_view = bool(await database.get_view_definition(table))\r\n        table_exists = bool(await database.table_exists(table))\r\n        if not is_view and not table_exists:\r\n            raise NotFound(f\"Table not found: {table}\")\r\n\r\n        await check_permissions(\r\n            datasette,\r\n            request,\r\n            [\r\n                (\"view-table\", (database.name, table)),\r\n                (\"view-database\", database.name),\r\n                \"view-instance\",\r\n            ],\r\n        )\r\n\r\n        private = not await datasette.permission_allowed(\r\n            None, \"view-table\", (database.name, table), default=True\r\n        )\r\n\r\n        pks = await database.primary_keys(table)\r\n        table_columns = await database.table_columns(table)\r\n\r\n        specified_columns = await columns_to_select(datasette, database, table, request)\r\n        select_specified_columns = \", \".join(\r\n            escape_sqlite(t) for t in specified_columns\r\n        )\r\n        select_all_columns = \", \".join(escape_sqlite(t) for t in table_columns)\r\n\r\n        use_rowid = not pks and not is_view\r\n        if use_rowid:\r\n            select_specified_columns = f\"rowid, {select_specified_columns}\"\r\n            select_all_columns = f\"rowid, {select_all_columns}\"\r\n            order_by = \"rowid\"\r\n            order_by_pks = \"rowid\"\r\n        else:\r\n            order_by_pks = \", \".join([escape_sqlite(pk) for pk in pks])\r\n            order_by = order_by_pks\r\n\r\n        if is_view:\r\n            order_by = \"\"\r\n\r\n        nocount = request.args.get(\"_nocount\")\r\n        nofacet = request.args.get(\"_nofacet\")\r\n\r\n        if request.args.get(\"_shape\") in (\"array\", \"object\"):\r\n            nocount = True\r\n            nofacet = True\r\n\r\n        # Next, a TON of SQL to build where_params and filters and suchlike\r\n        # skipping that and jumping straight to...\r\n        where_clauses = []\r\n        where_clause = \"\"\r\n        if where_clauses:\r\n            where_clause = f\"where {' and '.join(where_clauses)} \"\r\n\r\n        from_sql = \"from {table_name} {where}\".format(\r\n            table_name=escape_sqlite(table),\r\n            where=(\"where {} \".format(\" and \".join(where_clauses)))\r\n            if where_clauses\r\n            else \"\",\r\n        )\r\n        from_sql_params ={}\r\n        params = {}\r\n        count_sql = f\"select count(*) {from_sql}\"\r\n        sql_no_order_no_limit = (\r\n            \"select {select_all_columns} from {table_name} {where}\".format(\r\n                select_all_columns=select_all_columns,\r\n                table_name=escape_sqlite(table),\r\n                where=where_clause,\r\n            )\r\n        )\r\n\r\n        page_size = 100\r\n        offset = \" offset 0\"\r\n\r\n        sql = \"select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}\".format(\r\n            select_specified_columns=select_specified_columns,\r\n            table_name=escape_sqlite(table),\r\n            where=where_clause,\r\n            order_by=order_by,\r\n            page_size=page_size + 1,\r\n            offset=offset,\r\n        )\r\n\r\n        # Fetch rows\r\n        results = await database.execute(sql, params, truncate=True)\r\n        columns = [r[0] for r in results.description]\r\n        rows = list(results.rows)\r\n\r\n        # Fetch count\r\n        filtered_table_rows_count = None\r\n        if count_sql:\r\n            try:\r\n                count_rows = list(await database.execute(count_sql, from_sql_params))\r\n                filtered_table_rows_count = count_rows[0][0]\r\n            except QueryInterrupted:\r\n                pass\r\n\r\n\r\n        vars = {\r\n            \"json\": {\r\n                # THIS STUFF is from the regular JSON\r\n                \"database\": database.name,\r\n                \"table\": table,\r\n                \"is_view\": is_view,\r\n                # \"human_description_en\": human_description_en,\r\n                \"rows\": rows[:page_size],\r\n                \"truncated\": results.truncated,\r\n                \"filtered_table_rows_count\": filtered_table_rows_count,\r\n                # \"expanded_columns\": expanded_columns,\r\n                # \"expandable_columns\": expandable_columns,\r\n                \"columns\": columns,\r\n                \"primary_keys\": pks,\r\n                # \"units\": units,\r\n                \"query\": {\"sql\": sql, \"params\": params},\r\n                # \"facet_results\": facet_results,\r\n                # \"suggested_facets\": suggested_facets,\r\n                # \"next\": next_value and str(next_value) or None,\r\n                # \"next_url\": next_url,\r\n                \"private\": private,\r\n                \"allow_execute_sql\": await datasette.permission_allowed(\r\n                    request.actor, \"execute-sql\", database, default=True\r\n                ),\r\n            },\r\n            \"html\": {\r\n                # ... this is the HTML special stuff\r\n                # \"table_actions\": table_actions,\r\n                # \"supports_search\": bool(fts_table),\r\n                # \"search\": search or \"\",\r\n                \"use_rowid\": use_rowid,\r\n                # \"filters\": filters,\r\n                # \"display_columns\": display_columns,\r\n                # \"filter_columns\": filter_columns,\r\n                # \"display_rows\": display_rows,\r\n                # \"facets_timed_out\": facets_timed_out,\r\n                # \"sorted_facet_results\": sorted(\r\n                #     facet_results.values(),\r\n                #     key=lambda f: (len(f[\"results\"]), f[\"name\"]),\r\n                #     reverse=True,\r\n                # ),\r\n                # \"show_facet_counts\": special_args.get(\"_facet_size\") == \"max\",\r\n                # \"extra_wheres_for_ui\": extra_wheres_for_ui,\r\n                # \"form_hidden_args\": form_hidden_args,\r\n                # \"is_sortable\": any(c[\"sortable\"] for c in display_columns),\r\n                # \"path_with_replaced_args\": path_with_replaced_args,\r\n                # \"path_with_removed_args\": path_with_removed_args,\r\n                # \"append_querystring\": append_querystring,\r\n                \"request\": request,\r\n                # \"sort\": sort,\r\n                # \"sort_desc\": sort_desc,\r\n                \"disable_sort\": is_view,\r\n                # \"custom_table_templates\": [\r\n                #     f\"_table-{to_css_class(database)}-{to_css_class(table)}.html\",\r\n                #     f\"_table-table-{to_css_class(database)}-{to_css_class(table)}.html\",\r\n                #     \"_table.html\",\r\n                # ],\r\n                # \"metadata\": metadata,\r\n                # \"view_definition\": await db.get_view_definition(table),\r\n                # \"table_definition\": await db.get_table_definition(table),\r\n            },\r\n        }\r\n\r\n        # I'm just trying to get HTML to work for the moment\r\n        if format == \"json\":\r\n            return Response.json(dict(vars, locals=locals()), default=repr)\r\n        else:\r\n            return Response.html(repr(vars[\"html\"]))\r\n\r\n    async def view(self, request, datasette):\r\n        return await self.main(request=request, datasette=datasette)\r\n\r\n\r\n@hookimpl\r\ndef register_routes():\r\n    return [\r\n        (r\"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)\", Table().view),\r\n    ]\r\n\r\n\r\nasync def check_permissions(datasette, request, permissions):\r\n    \"\"\"permissions is a list of (action, resource) tuples or 'action' strings\"\"\"\r\n    for permission in permissions:\r\n        if isinstance(permission, str):\r\n            action = permission\r\n            resource = None\r\n        elif isinstance(permission, (tuple, list)) and len(permission) == 2:\r\n            action, resource = permission\r\n        else:\r\n            assert (\r\n                False\r\n            ), \"permission should be string or tuple of two items: {}\".format(\r\n                repr(permission)\r\n            )\r\n        ok = await datasette.permission_allowed(\r\n            request.actor,\r\n            action,\r\n            resource=resource,\r\n            default=None,\r\n        )\r\n        if ok is not None:\r\n            if ok:\r\n                return\r\n            else:\r\n                raise Forbidden(action)\r\n\r\n\r\nasync def columns_to_select(datasette, database, table, request):\r\n    table_columns = await database.table_columns(table)\r\n    pks = await database.primary_keys(table)\r\n    columns = list(table_columns)\r\n    if \"_col\" in request.args:\r\n        columns = list(pks)\r\n        _cols = request.args.getlist(\"_col\")\r\n        bad_columns = [column for column in _cols if column not in table_columns]\r\n        if bad_columns:\r\n            raise DatasetteError(\r\n                \"_col={} - invalid columns\".format(\", \".join(bad_columns)),\r\n                status=400,\r\n            )\r\n        # De-duplicate maintaining order:\r\n        columns.extend(dict.fromkeys(_cols))\r\n    if \"_nocol\" in request.args:\r\n        # Return all columns EXCEPT these\r\n        bad_columns = [\r\n            column\r\n            for column in request.args.getlist(\"_nocol\")\r\n            if (column not in table_columns) or (column in pks)\r\n        ]\r\n        if bad_columns:\r\n            raise DatasetteError(\r\n                \"_nocol={} - invalid columns\".format(\", \".join(bad_columns)),\r\n                status=400,\r\n            )\r\n        tmp_columns = [\r\n            column for column in columns if column not in request.args.getlist(\"_nocol\")\r\n        ]\r\n        columns = tmp_columns\r\n    return columns\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-973568285", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 973568285, "node_id": "IC_kwDOBm6k_c46B3kd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T00:29:20Z", "updated_at": "2021-11-19T00:29:20Z", "author_association": "OWNER", "body": "This is working!\r\n```python\r\nfrom datasette.utils.asgi import Response\r\nfrom datasette import hookimpl\r\nimport html\r\nfrom asyncinject import AsyncInject, inject\r\n\r\n\r\nclass Table(AsyncInject):\r\n    @inject\r\n    async def database(self, request):\r\n        return request.url_vars[\"db_name\"]\r\n\r\n    @inject\r\n    async def main(self, request, database):\r\n        return Response.html(\"Database: {}\".format(\r\n            html.escape(database)\r\n        ))\r\n\r\n    async def view(self, request):\r\n        return await self.main(request=request)\r\n\r\n\r\n@hookimpl\r\ndef register_routes():\r\n    return [\r\n        (r\"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)\", Table().view),\r\n    ]\r\n```\r\nThis project will definitely show me if I actually like the `asyncinject` patterns or not.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-973564260", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 973564260, "node_id": "IC_kwDOBm6k_c46B2lk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T00:27:06Z", "updated_at": "2021-11-19T00:27:06Z", "author_association": "OWNER", "body": "Problem: the fancy `asyncinject` stuff inteferes with the fancy Datasette thing that introspects view functions to look for what parameters they take:\r\n```python\r\nclass Table(asyncinject.AsyncInjectAll):\r\n    async def view(self, request):\r\n        return Response.html(\"Hello from {}\".format(\r\n            html.escape(repr(request.url_vars))\r\n        ))\r\n\r\n\r\n@hookimpl\r\ndef register_routes():\r\n    return [\r\n        (r\"/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)\", Table().view),\r\n    ]\r\n```\r\nThis failed with error: \"Table.view() takes 1 positional argument but 2 were given\"\r\n\r\nSo I'm going to use `AsyncInject` and have the `view` function NOT use the `@inject` decorator.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-973554024", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 973554024, "node_id": "IC_kwDOBm6k_c46B0Fo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T00:21:20Z", "updated_at": "2021-11-19T00:21:20Z", "author_association": "OWNER", "body": "That's annoying: it looks like plugins can't use `register_routes()` to over-ride default routes within Datasette itself. This didn't work:\r\n```python\r\nfrom datasette.utils.asgi import Response\r\nfrom datasette import hookimpl\r\nimport html\r\n\r\n\r\nasync def table(request):\r\n    return Response.html(\"Hello from {}\".format(\r\n        html.escape(repr(request.url_vars))\r\n    ))\r\n\r\n\r\n@hookimpl\r\ndef register_routes():\r\n    return [\r\n        (r\"/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)\", table),\r\n    ]\r\n```\r\nI'll use a `/t/` prefix for the moment, but this is probably something I'll fix in Datasette itself later.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-973542284", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 973542284, "node_id": "IC_kwDOBm6k_c46BxOM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T00:16:44Z", "updated_at": "2021-11-19T00:16:44Z", "author_association": "OWNER", "body": "```\r\nDevelopment % cookiecutter gh:simonw/datasette-plugin\r\nYou've downloaded /Users/simon/.cookiecutters/datasette-plugin before. Is it okay to delete and re-download it? [yes]: yes\r\nplugin_name []: table-new\r\ndescription []: New implementation of TableView, see https://github.com/simonw/datasette/issues/878\r\nhyphenated [table-new]: \r\nunderscored [table_new]: \r\ngithub_username []: simonw\r\nauthor_name []: Simon Willison\r\ninclude_static_directory []:  \r\ninclude_templates_directory []: \r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/878#issuecomment-973527870", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/878", "id": 973527870, "node_id": "IC_kwDOBm6k_c46Bts-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-11-19T00:13:43Z", "updated_at": "2021-11-19T00:13:43Z", "author_association": "OWNER", "body": "New plan: I'm going to build a brand new implementation of `TableView` starting out as a plugin, using the `register_routes()` plugin hook.\r\n\r\nIt will reuse the existing HTML template but will be a completely new Python implementation, based on `asyncinject`.\r\n\r\nI'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on.\r\n\r\nBonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 648435885, "label": "New pattern for views that return either JSON or HTML, available for plugins"}, "performed_via_github_app": null}