{"html_url": "https://github.com/simonw/datasette/issues/1072#issuecomment-719810533", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1072", "id": 719810533, "node_id": "MDEyOklzc3VlQ29tbWVudDcxOTgxMDUzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-30T21:34:38Z", "updated_at": "2020-10-30T21:34:38Z", "author_association": "OWNER", "body": "... no wait, my comments above assume that I'm just building the `datasette-edit-templates` plugin. Does this work as a general solution for all of Datasette? I don't think it does.\r\n\r\nThis may mean I need to delay the whole feature.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 733499930, "label": "load_template hook doesn't work for include/extends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1072#issuecomment-719803880", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1072", "id": 719803880, "node_id": "MDEyOklzc3VlQ29tbWVudDcxOTgwMzg4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-30T21:17:11Z", "updated_at": "2020-10-30T21:17:11Z", "author_association": "OWNER", "body": "Example from the Jinja docs: https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.BaseLoader\r\n\r\n```python\r\nfrom jinja2 import BaseLoader, TemplateNotFound\r\nfrom os.path import join, exists, getmtime\r\n\r\nclass MyLoader(BaseLoader):\r\n\r\n def __init__(self, path):\r\n self.path = path\r\n\r\n def get_source(self, environment, template):\r\n path = join(self.path, template)\r\n if not exists(path):\r\n raise TemplateNotFound(template)\r\n mtime = getmtime(path)\r\n with file(path) as f:\r\n source = f.read().decode('utf-8')\r\n return source, path, lambda: mtime == getmtime(path)\r\n```\r\nAlso available: `jinja2.FunctionLoader(load_func)` which lets me pass it a function like this one:\r\n```\r\n>>> def load_template(name):\r\n... if name == 'index.html':\r\n... return '...'\r\n...\r\n>>> loader = FunctionLoader(load_template)\r\n```\r\n\r\nJust one catch: I need to be able to load templates asynchronously, because they live in the database. Let's hope Jinja has a mechanism for that!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 733499930, "label": "load_template hook doesn't work for include/extends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1072#issuecomment-719813212", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1072", "id": 719813212, "node_id": "MDEyOklzc3VlQ29tbWVudDcxOTgxMzIxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-30T21:42:35Z", "updated_at": "2020-10-30T21:42:35Z", "author_association": "OWNER", "body": "Filed a feature request here: https://github.com/pallets/jinja/issues/1304", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 733499930, "label": "load_template hook doesn't work for include/extends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1072#issuecomment-719809780", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1072", "id": 719809780, "node_id": "MDEyOklzc3VlQ29tbWVudDcxOTgwOTc4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-30T21:32:28Z", "updated_at": "2020-10-30T21:32:28Z", "author_association": "OWNER", "body": "Here's an alternative that would definitely work and would be a lot simpler, at the cost of a fair amount of RAM:\r\n\r\n1. Before rendering the template, load ALL of the most-recent-versions of the templates that are stored in the DB. Use those to populate a `DictLoader`.\r\n2. Render the template.\r\n\r\nThis does mean loading template bodies that we won't use. Provided an instance has less than 100 templates I imagine this will work just fine.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 733499930, "label": "load_template hook doesn't work for include/extends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1072#issuecomment-719986698", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1072", "id": 719986698, "node_id": "MDEyOklzc3VlQ29tbWVudDcxOTk4NjY5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-31T20:48:17Z", "updated_at": "2020-10-31T20:48:17Z", "author_association": "OWNER", "body": "Here's the `datasette-edit-templates` plugin WIP I had before removing the hook: https://github.com/simonw/datasette-edit-templates/tree/82855c2612b84bc09c48fca885f831633a0d1552", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 733499930, "label": "load_template hook doesn't work for include/extends"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1072#issuecomment-719955491", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1072", "id": 719955491, "node_id": "MDEyOklzc3VlQ29tbWVudDcxOTk1NTQ5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-10-31T16:20:58Z", "updated_at": "2020-10-31T16:20:58Z", "author_association": "OWNER", "body": "Here's the proof of concept `FunctionLoader` that showed me that this wasn't going to work:\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 4b28e71..b076be7 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -21,7 +21,7 @@ from pathlib import Path\r\n from markupsafe import Markup\r\n from itsdangerous import URLSafeSerializer\r\n import jinja2\r\n-from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader\r\n+from jinja2 import ChoiceLoader, Environment, FileSystemLoader, FunctionLoader, PrefixLoader\r\n from jinja2.environment import Template\r\n from jinja2.exceptions import TemplateNotFound\r\n import uvicorn\r\n@@ -300,6 +300,7 @@ class Datasette:\r\n template_paths.append(default_templates)\r\n template_loader = ChoiceLoader(\r\n [\r\n+ FunctionLoader(self._load_template_from_plugins),\r\n FileSystemLoader(template_paths),\r\n # Support {% extends \"default:table.html\" %}:\r\n PrefixLoader(\r\n@@ -322,6 +323,17 @@ class Datasette:\r\n self._root_token = secrets.token_hex(32)\r\n self.client = DatasetteClient(self)\r\n \r\n+ def _load_template_from_plugins(self, template):\r\n+ # \"If auto reloading is enabled it\u2019s called to check if the template changed\"\r\n+ uptodatefunc = lambda: True\r\n+ source = pm.hook.load_template(\r\n+ template=template,\r\n+ datasette=self,\r\n+ )\r\n+ if source is None:\r\n+ return None\r\n+ return source, template, uptodatefunc\r\n+\r\n @property\r\n def urls(self):\r\n return Urls(self)\r\n@@ -719,35 +731,7 @@ class Datasette:\r\n else:\r\n if isinstance(templates, str):\r\n templates = [templates]\r\n-\r\n- # Give plugins first chance at loading the template\r\n- break_outer = False\r\n- plugin_template_source = None\r\n- plugin_template_name = None\r\n- template_name = None\r\n- for template_name in templates:\r\n- if break_outer:\r\n- break\r\n- plugin_template_source = pm.hook.load_template(\r\n- template=template_name,\r\n- request=request,\r\n- datasette=self,\r\n- )\r\n- plugin_template_source = await await_me_maybe(plugin_template_source)\r\n- if plugin_template_source:\r\n- break_outer = True\r\n- plugin_template_name = template_name\r\n- break\r\n- if plugin_template_source is not None:\r\n- template = self.jinja_env.from_string(plugin_template_source)\r\n- else:\r\n- template = self.jinja_env.select_template(templates)\r\n- for template_name in templates:\r\n- from_plugin = template_name == plugin_template_name\r\n- used = from_plugin or template_name == template.name\r\n- templates_considered.append(\r\n- {\"name\": template_name, \"used\": used, \"from_plugin\": from_plugin}\r\n- )\r\n+ template = self.jinja_env.select_template(templates)\r\n body_scripts = []\r\n # pylint: disable=no-member\r\n for extra_script in pm.hook.extra_body_script(\r\ndiff --git a/datasette/hookspecs.py b/datasette/hookspecs.py\r\nindex ca84b35..7804def 100644\r\n--- a/datasette/hookspecs.py\r\n+++ b/datasette/hookspecs.py\r\n@@ -50,7 +50,7 @@ def extra_template_vars(\r\n \r\n \r\n @hookspec(firstresult=True)\r\n-def load_template(template, request, datasette):\r\n+def load_template(template, datasette):\r\n \"Load the specified template, returning the template code as a string\"\r\n \r\n \r\ndiff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst\r\nindex 3c57b6a..8f2704e 100644\r\n--- a/docs/plugin_hooks.rst\r\n+++ b/docs/plugin_hooks.rst\r\n@@ -273,15 +273,12 @@ Example: `datasette-cluster-map