issue_comments

Should mostly be a case of backing out the changes from this commit: 81dea4b07ab2b6f4eaaf248307d2b588472054a1

Yeah I'm going to remove the load_template plugin hook and see if it's possible to build the edit templates extension against prepare_jinja2_environment instead.

https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L310-L318

So yeah that plugin hook can probably modify the list of loaders available to the Environment.

Maybe I should ditch this hook entirely in favour of the existing prepare_jinja2_environment hook. Could that add new template loaders?

I tried using a FunctionLoader and got this error on startup: python File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 989, in __init__ for filepath in self.ds.jinja_env.list_templates() File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/environment.py", line 810, in list_templates names = self.loader.list_templates() File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py", line 434, in list_templates found.update(loader.list_templates()) File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py", line 99, in list_templates raise TypeError("this loader cannot iterate over all templates") TypeError: this loader cannot iterate over all templates So if I'm going to define a custom Jinja loader I'll need to teach plugins to answer the "list templates" query.

I'll try getting that to work. If I can't get it to work I'll drop the plugin hook for the moment.

There might be a way to save this. Async template loading can't be supported, but what if you could define a load_template() hook which returned a sync function that returned templates...

Then the datasette-edit-templates plugin could reply to load_template by loading all DB templates into memory and returning a load_template sync function that looked up the values in those already-loaded templates.

It could even maintain an in-memory cache that gets updated when a template is edited.

If I do this, I could remove the ability to return an async function from load_template() but add that in the future should Jinja implement a mechanism for async template loading.

Sadly I'm going to bump load_template from Datasette 0.51 - I don't think I should block the release on resolving this issue.

I'm pretty sure that run_in_executor() workaround won't work. https://github.com/django/asgiref/blob/7becc9daca2628c46af1cb7e46b4c47c1ea27adf/asgiref/sync.py#L83 for example says "You cannot use AsyncToSync in the same thread as an async event loop".

Filed a feature request here: https://github.com/pallets/jinja/issues/1304

There's one other option: in datasette-edit-templates I could maybe use asyncio.get_event_loop().run_in_executor(...) to load the templates asynchronously within the Jinja template loader mechanism.

... no wait, my comments above assume that I'm just building the datasette-edit-templates plugin. Does this work as a general solution for all of Datasette? I don't think it does.

This may mean I need to delay the whole feature.

The ideal solution is for Jinja to offer async template loading. I'll file a feature request, then I'll implement the second option above (async load all templates from the DB before each render).

Here's an alternative that would definitely work and would be a lot simpler, at the cost of a fair amount of RAM:

1. Before rendering the template, load ALL of the most-recent-versions of the templates that are stored in the DB. Use those to populate a DictLoader.
2. Render the template.

This does mean loading template bodies that we won't use. Provided an instance has less than 100 templates I imagine this will work just fine.

How can we tell what template Jinja will need to render?

One approach that could work:

1. Set up a dummy template loader which records the name of the template that was requested
2. Load the template
3. Now we know the list of templates that were requested. Async load those
4. The dummy template loader can now return the ones we have loaded. Load the template again.
5. Did it request any more templates? If so, load those, and repeat.
6. Keep on with this loop until a template load (which might even have to be a render) fails to request any templates that we have not yet loaded.
7. Render the template.

This is GROSS. It feels like a huge waste of CPU, and it could lead to very weird behaviour if any template variables have side effects.

It looks like Jinja does not have a mechanism for asynchronous template loading - the loader API is synchronous.

One option may be to figure out which templates are needed (including inherited templates and includes) before rendering the template. Then async load those templates from the database into a DictLoader, then pass that DictLoader to Jinja.

Example from the Jinja docs: https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.BaseLoader

```python from jinja2 import BaseLoader, TemplateNotFound from os.path import join, exists, getmtime

class MyLoader(BaseLoader):

def __init__(self, path):
    self.path = path

def get_source(self, environment, template):
    path = join(self.path, template)
    if not exists(path):
        raise TemplateNotFound(template)
    mtime = getmtime(path)
    with file(path) as f:
        source = f.read().decode('utf-8')
    return source, path, lambda: mtime == getmtime(path)

Also available: `jinja2.FunctionLoader(load_func)` which lets me pass it a function like this one:

def load_template(name): ... if name == 'index.html': ... return '...' ... loader = FunctionLoader(load_template) ```

Just one catch: I need to be able to load templates asynchronously, because they live in the database. Let's hope Jinja has a mechanism for that!

It should be easy enough to show a comment that says which original template names were considered, but I may not be able to show which one was actually used (or which ones came from plugins).

To fix this I think I need to move the load_template implementation into a Jinja template loader.

I'm not sure I'll be able to keep the Templates considered comment working though:

https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L745-L750

Fixed: https://latest.datasette.io/-/messages

OK, new hook specification is: python @hookspec(firstresult=True) def load_template(template, request, datasette): "Load the specified template, returning the template code as a string"

I'm having second thoughts about the design of the plugin hook.

Consider the following: python plugin_template_source = pm.hook.load_template( template=template_name, database=context.get("database"), table=context.get("table"), columns=context.get("columns"), view_name=self.name, request=request, datasette=self.ds, ) It's a bit gross that database, table and columns are pulled out of the context like that. This doesn't make sense for pages that are rendered by plugins, for example.

So maybe for the first release of this plugin hook I should cut it down to just seeing template, request and datasette. I can add the table/view/etc stuff back in later if it turns out to be necessary.

Bringing over a comment from #1042:

I'd like to do this all in the datasette.render_template() method to ensure it's available to plugins as well, not just core code that uses the BaseView class.

This code is the problem:

https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133

I think I'll fix this by moving the select_templates mechanism into datasette.render_templates().

I should include an example in the documentation that shows loading templates from a database table.

Next steps: build a demonstration plugin against this.

Documentation: https://docs.datasette.io/en/latest/authentication.html#debug-menu

I mainly want this so I can add that debug menu to my Dogsheep.

Document the new permission here: https://docs.datasette.io/en/stable/authentication.html#built-in-permissions

Permission idea: debug-menu

Then this can make it available to root: https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_permissions.py

Maybe rename it to actions_menu and have it work for database, view, table and query pages using different arguments on each.

Yes, this should be possible - no point restricting what plugin authors can do with the feature. Will need to add some extra arguments to the plugin hook for this.

I've now added two new plugin hooks: menu_links() and table_actions().

I'm going to close this issue. Further work (on column actions and and database actions) can happen in separate tickets, but I won't include them in Datasette 0.51 since they're much less interesting than table and instance actions.

Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table

The cog only appears if at least one table action has been registered by a plugin. It looks like this:

This will use a very similar implementation to the navigation menu in #1064 - similar plugin hook and I'll use a <details><summary> to implement it.

I'm going to go with a list of {"label": ..., "href": ...} as the first iteration of this. The logout link will not be returned as part of the plugin output. A default plugin will provide the debug tools if the user is logged in as root.

I'm torn on this one. I think I have a very slight preference for plugins returning structured objects as opposed to HTML. Less likely to regret that choice in the future?

Plugins returning HTML makes more sense for some of the other areas that plugins will be able to inject content - e.g. injecting content on the table or row page above the table.

If I'm going to have that as a pattern though it may make sense to use HTML here, since that will be consistent with other places that plugins can inject additional content.

Or... plugins could return HTML - maybe optionally using helper functions to generate common HTML such that plugins which use the helpers can have their HTML modified in the future.

Should plugins be able to add forms like the logout form here, or should they be restricted to adding navigation links?

I can't think of a reason a plugin would need to add a form. The logout form is a special case to protect against logout-csrf attacks.

So I think plugins get to return a list of dictionaries, each with a label and an href:

python return [{ "label": "Upload CSVs", "href": datasette.urls.path("/-/upload-csvs") }] But... is there an argument for returning headings, to divide up the menu?

I think so. I also like the idea that a default plugin checks for the root user and outputs links to the different debugging tools - maybe those should be wrapped in a section heading.

How should the plugin hook work?

Here's the first version of the HTML: html <div class="nav-menu-inner"> <ul> <li><a href="{{ urls.instance() }}">Home</a></li> <li><a href="{{ urls.path('/-/plugins') }}">Installed plugins</a></li> <li><a href="{{ urls.path('/-/versions') }}">Software versions</a></li> <li><a href="{{ urls.path('/-/metadata') }}">Metadata</a></li> {% if show_logout %} <form action="{{ urls.logout() }}" method="post"> <input type="hidden" name="csrftoken" value="{{ csrftoken() }}"> <button class="button-as-link">Log out</button> </form>{% endif %} </ul> </div>

So what should go in this menu?

If the user is logged in as root, I'll link to the various debug pages.

If they're not logged in at all I don't think the menu should appear.

If they are logged in as anyone, it should display to give them access to the "log out" button.

Plugins can add links to it. If those plugins add links, the menu will display.

Tips for making this accessible: https://css-tricks.com/accessible-svgs/

Here's what the prototype looks like so far:

I used a <details><summary> for this: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L16-L36

I added a bit of JavaScript so that clicking outside the menu would close it: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L59-L74

Demos:

https://latest.datasette.io/fixtures/binary_data.csv?_size=max

csv rowid,data 1,http://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data 2,http://latest.datasette.io/fixtures/binary_data/2.blob?_blob_column=data 3,

https://latest.datasette.io/fixtures.csv?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max

csv rowid,data 1,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d 2,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=b835b0483cedb86130b9a2c280880bf5fadc5318ddf8c18d0df5204d40df1724 3,