20 rows where issue = 733499930 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

issue

  • load_template hook doesn't work for include/extends · 20

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
719986698 https://github.com/simonw/datasette/issues/1072#issuecomment-719986698 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTk4NjY5OA== simonw 9599 2020-10-31T20:48:17Z 2020-10-31T20:48:17Z OWNER

Here's the datasette-edit-templates plugin WIP I had before removing the hook: https://github.com/simonw/datasette-edit-templates/tree/82855c2612b84bc09c48fca885f831633a0d1552

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719955491 https://github.com/simonw/datasette/issues/1072#issuecomment-719955491 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTk1NTQ5MQ== simonw 9599 2020-10-31T16:20:58Z 2020-10-31T16:20:58Z OWNER

Here's the proof of concept FunctionLoader that showed me that this wasn't going to work:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 4b28e71..b076be7 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -21,7 +21,7 @@ from pathlib import Path
from markupsafe import Markup
from itsdangerous import URLSafeSerializer
import jinja2
-from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
+from jinja2 import ChoiceLoader, Environment, FileSystemLoader, FunctionLoader, PrefixLoader
from jinja2.environment import Template
from jinja2.exceptions import TemplateNotFound
import uvicorn
@@ -300,6 +300,7 @@ class Datasette:
template_paths.append(default_templates)
template_loader = ChoiceLoader(
[
+ FunctionLoader(self._load_template_from_plugins),
FileSystemLoader(template_paths),
# Support {% extends "default:table.html" %}:
PrefixLoader(
@@ -322,6 +323,17 @@ class Datasette:
self._root_token = secrets.token_hex(32)
self.client = DatasetteClient(self)

  • def _load_template_from_plugins(self, template):
  • "If auto reloading is enabled it’s called to check if the template changed"

  • uptodatefunc = lambda: True
  • source = pm.hook.load_template(
  • template=template,
  • datasette=self,
  • )
  • if source is None:
  • return None
  • return source, template, uptodatefunc
    +
    @property
    def urls(self):
    return Urls(self)
    @@ -719,35 +731,7 @@ class Datasette:
    else:
    if isinstance(templates, str):
    templates = [templates]
    -
  • Give plugins first chance at loading the template

  • break_outer = False
  • plugin_template_source = None
  • plugin_template_name = None
  • template_name = None
  • for template_name in templates:
  • if break_outer:
  • break
  • plugin_template_source = pm.hook.load_template(
  • template=template_name,
  • request=request,
  • datasette=self,
  • )
  • plugin_template_source = await await_me_maybe(plugin_template_source)
  • if plugin_template_source:
  • break_outer = True
  • plugin_template_name = template_name
  • break
  • if plugin_template_source is not None:
  • template = self.jinja_env.from_string(plugin_template_source)
  • else:
  • template = self.jinja_env.select_template(templates)
  • for template_name in templates:
  • from_plugin = template_name == plugin_template_name
  • used = from_plugin or template_name == template.name
  • templates_considered.append(
  • {"name": template_name, "used": used, "from_plugin": from_plugin}
  • )
  • template = self.jinja_env.select_template(templates)
    body_scripts = []
    # pylint: disable=no-member
    for extra_script in pm.hook.extra_body_script(
    diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py
    index ca84b35..7804def 100644
    --- a/datasette/hookspecs.py
    +++ b/datasette/hookspecs.py
    @@ -50,7 +50,7 @@ def extra_template_vars(

@hookspec(firstresult=True)
-def load_template(template, request, datasette):
+def load_template(template, datasette):
"Load the specified template, returning the template code as a string"

diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst
index 3c57b6a..8f2704e 100644
--- a/docs/plugin_hooks.rst
+++ b/docs/plugin_hooks.rst
@@ -273,15 +273,12 @@ Example: `datasette-cluster-map <https://github.com/simonw/datasette-cluster-map

.. _plugin_hook_load_template:

-load_template(template, request, datasette)

+load_template(template, datasette)
+----------------------------------

template - string
The template that is being rendered, e.g. database.html

-request - object or None
- The current HTTP :ref:internals_request. This can be None if the request object is not available.
-
datasette - :ref:internals_datasette
You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name)
```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719833744 https://github.com/simonw/datasette/issues/1072#issuecomment-719833744 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgzMzc0NA== simonw 9599 2020-10-30T22:50:57Z 2020-10-30T22:50:57Z OWNER

Yeah I'm going to remove the load_template plugin hook and see if it's possible to build the edit templates extension against prepare_jinja2_environment instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719833070 https://github.com/simonw/datasette/issues/1072#issuecomment-719833070 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgzMzA3MA== simonw 9599 2020-10-30T22:48:04Z 2020-10-30T22:48:04Z OWNER

https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L310-L318

So yeah that plugin hook can probably modify the list of loaders available to the Environment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719832853 https://github.com/simonw/datasette/issues/1072#issuecomment-719832853 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgzMjg1Mw== simonw 9599 2020-10-30T22:47:12Z 2020-10-30T22:47:12Z OWNER

Maybe I should ditch this hook entirely in favour of the existing prepare_jinja2_environment hook. Could that add new template loaders?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719832651 https://github.com/simonw/datasette/issues/1072#issuecomment-719832651 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgzMjY1MQ== simonw 9599 2020-10-30T22:46:25Z 2020-10-30T22:46:25Z OWNER

I tried using a FunctionLoader and got this error on startup:

  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 989, in __init__
    for filepath in self.ds.jinja_env.list_templates()
  File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/environment.py", line 810, in list_templates
    names = self.loader.list_templates()
  File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py", line 434, in list_templates
    found.update(loader.list_templates())
  File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py", line 99, in list_templates
    raise TypeError("this loader cannot iterate over all templates")
TypeError: this loader cannot iterate over all templates

So if I'm going to define a custom Jinja loader I'll need to teach plugins to answer the "list templates" query.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719819331 https://github.com/simonw/datasette/issues/1072#issuecomment-719819331 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgxOTMzMQ== simonw 9599 2020-10-30T22:00:43Z 2020-10-30T22:00:43Z OWNER

I'll try getting that to work. If I can't get it to work I'll drop the plugin hook for the moment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719819234 https://github.com/simonw/datasette/issues/1072#issuecomment-719819234 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgxOTIzNA== simonw 9599 2020-10-30T22:00:21Z 2020-10-30T22:00:21Z OWNER

There might be a way to save this. Async template loading can't be supported, but what if you could define a load_template() hook which returned a sync function that returned templates...

Then the datasette-edit-templates plugin could reply to load_template by loading all DB templates into memory and returning a load_template sync function that looked up the values in those already-loaded templates.

It could even maintain an in-memory cache that gets updated when a template is edited.

If I do this, I could remove the ability to return an async function from load_template() but add that in the future should Jinja implement a mechanism for async template loading.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719814279 https://github.com/simonw/datasette/issues/1072#issuecomment-719814279 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgxNDI3OQ== simonw 9599 2020-10-30T21:45:33Z 2020-10-30T21:45:33Z OWNER

Sadly I'm going to bump load_template from Datasette 0.51 - I don't think I should block the release on resolving this issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719813970 https://github.com/simonw/datasette/issues/1072#issuecomment-719813970 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgxMzk3MA== simonw 9599 2020-10-30T21:44:40Z 2020-10-30T21:44:40Z OWNER

I'm pretty sure that run_in_executor() workaround won't work. https://github.com/django/asgiref/blob/7becc9daca2628c46af1cb7e46b4c47c1ea27adf/asgiref/sync.py#L83 for example says "You cannot use AsyncToSync in the same thread as an async event loop".

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719813212 https://github.com/simonw/datasette/issues/1072#issuecomment-719813212 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgxMzIxMg== simonw 9599 2020-10-30T21:42:35Z 2020-10-30T21:42:35Z OWNER

Filed a feature request here: https://github.com/pallets/jinja/issues/1304

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719811312 https://github.com/simonw/datasette/issues/1072#issuecomment-719811312 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgxMTMxMg== simonw 9599 2020-10-30T21:36:49Z 2020-10-30T21:36:49Z OWNER

There's one other option: in datasette-edit-templates I could maybe use asyncio.get_event_loop().run_in_executor(...) to load the templates asynchronously within the Jinja template loader mechanism.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719810533 https://github.com/simonw/datasette/issues/1072#issuecomment-719810533 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgxMDUzMw== simonw 9599 2020-10-30T21:34:38Z 2020-10-30T21:34:38Z OWNER

... no wait, my comments above assume that I'm just building the datasette-edit-templates plugin. Does this work as a general solution for all of Datasette? I don't think it does.

This may mean I need to delay the whole feature.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719810023 https://github.com/simonw/datasette/issues/1072#issuecomment-719810023 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgxMDAyMw== simonw 9599 2020-10-30T21:33:06Z 2020-10-30T21:33:06Z OWNER

The ideal solution is for Jinja to offer async template loading. I'll file a feature request, then I'll implement the second option above (async load all templates from the DB before each render).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719809780 https://github.com/simonw/datasette/issues/1072#issuecomment-719809780 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgwOTc4MA== simonw 9599 2020-10-30T21:32:28Z 2020-10-30T21:32:28Z OWNER

Here's an alternative that would definitely work and would be a lot simpler, at the cost of a fair amount of RAM:

  1. Before rendering the template, load ALL of the most-recent-versions of the templates that are stored in the DB. Use those to populate a DictLoader.
  2. Render the template.

This does mean loading template bodies that we won't use. Provided an instance has less than 100 templates I imagine this will work just fine.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719809259 https://github.com/simonw/datasette/issues/1072#issuecomment-719809259 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgwOTI1OQ== simonw 9599 2020-10-30T21:31:10Z 2020-10-30T21:31:10Z OWNER

How can we tell what template Jinja will need to render?

One approach that could work:

  1. Set up a dummy template loader which records the name of the template that was requested
  2. Load the template
  3. Now we know the list of templates that were requested. Async load those
  4. The dummy template loader can now return the ones we have loaded. Load the template again.
  5. Did it request any more templates? If so, load those, and repeat.
  6. Keep on with this loop until a template load (which might even have to be a render) fails to request any templates that we have not yet loaded.
  7. Render the template.

This is GROSS. It feels like a huge waste of CPU, and it could lead to very weird behaviour if any template variables have side effects.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719807502 https://github.com/simonw/datasette/issues/1072#issuecomment-719807502 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgwNzUwMg== simonw 9599 2020-10-30T21:26:49Z 2020-10-30T21:26:49Z OWNER

It looks like Jinja does not have a mechanism for asynchronous template loading - the loader API is synchronous.

One option may be to figure out which templates are needed (including inherited templates and includes) before rendering the template. Then async load those templates from the database into a DictLoader, then pass that DictLoader to Jinja.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719803880 https://github.com/simonw/datasette/issues/1072#issuecomment-719803880 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTgwMzg4MA== simonw 9599 2020-10-30T21:17:11Z 2020-10-30T21:17:11Z OWNER

Example from the Jinja docs: https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.BaseLoader

from jinja2 import BaseLoader, TemplateNotFound
from os.path import join, exists, getmtime

class MyLoader(BaseLoader):

    def __init__(self, path):
        self.path = path

    def get_source(self, environment, template):
        path = join(self.path, template)
        if not exists(path):
            raise TemplateNotFound(template)
        mtime = getmtime(path)
        with file(path) as f:
            source = f.read().decode('utf-8')
        return source, path, lambda: mtime == getmtime(path)

Also available: jinja2.FunctionLoader(load_func) which lets me pass it a function like this one:

>>> def load_template(name):
...     if name == 'index.html':
...         return '...'
...
>>> loader = FunctionLoader(load_template)

Just one catch: I need to be able to load templates asynchronously, because they live in the database. Let's hope Jinja has a mechanism for that!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719785005 https://github.com/simonw/datasette/issues/1072#issuecomment-719785005 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTc4NTAwNQ== simonw 9599 2020-10-30T20:36:22Z 2020-10-30T20:36:22Z OWNER

It should be easy enough to show a comment that says which original template names were considered, but I may not be able to show which one was actually used (or which ones came from plugins).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  
719784606 https://github.com/simonw/datasette/issues/1072#issuecomment-719784606 https://api.github.com/repos/simonw/datasette/issues/1072 MDEyOklzc3VlQ29tbWVudDcxOTc4NDYwNg== simonw 9599 2020-10-30T20:35:33Z 2020-10-30T20:35:33Z OWNER

To fix this I think I need to move the load_template implementation into a Jinja template loader.

I'm not sure I'll be able to keep the Templates considered comment working though:

https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L745-L750

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
load_template hook doesn't work for include/extends 733499930  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);