issue_comments

(It may well be that implementing #1168 involves a switch to async metadata)

Great, let's drop fallback then.

My instinct at the moment is to ship this plugin hook as-is but with a warning that it may change before Datasette 1.0 - then before 1.0 either figure out an async variant or finish the database-backed metadata concept from #1168 and recommend that as an alternative.

Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.

@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

Answering my own question: here's how Brandon implements it in his datasette-live-config plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/init.py#L50-L160

That's using a completely separate SQLite connection (actually wrapped in sqlite-utils) and making blocking synchronous calls to it.

This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

But... it's weird. Everywhere else in Datasette land uses await db.execute(...) - but here's an example where users are encouraged to use blocking calls instead.

The other alternative is to finish the work to build a _metadata internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.

So instead, all metadata for all attached databases would be loaded into an in-memory database called _metadata. Plugins that want to modify stored metadata could then do so by directly writing to that table.

Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.

The await thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls datasette.metadata(...).

One solution that could work: introduce a new method, maybe await datasette.get_metadata(...), which uses this plugin hook - and keep the existing datasette.metadata() method (which doesn't call the hook) around. This would ensure existing plugins keep on working.

Then, upgrade those plugins separately - with the goal of deprecating and removing .metadata() entirely in Datasette 1.0 - having upgraded the plugins in the meantime.

Here's where the plugin hook is called, demonstrating the fallback= argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472

I'm not convinced of the use-case for passing fallback= to the hook here - is there a reason a plugin might care whether fallback is True or False, seeing as the metadata() method already respects that fallback logic on line 459?

Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.

@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

The documentation says:

datasette: You can use this to access plugin configuration options via datasette.plugin_config(your_plugin_name), or to execute SQL queries.

That's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run await db.execute(...) so you can't execute SQL queries.

I can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/

BUT... that requires changing the ds.metadata() function to be awaitable, which will affect every existing plugn that uses that documented internal method!

The documentation doesn't describe the fallback argument at the moment.

Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback

Link to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback