issue_comments

I'm going to take this in a different direction.

I'm not happy with how metadata.(json|yaml) keeps growing new features. Rather than having a single plugin hook for all of metadata.json I'm going to split out the feature that shows actual real metadata for tables and databases - source, license etc - into its own plugin-powered mechanism.

So I'm going to close this ticket and spin up a new one for that.

Here's an example plugin I set up using the experimental hook in d11fd2cbaa6b31933b1319f81b5d1520726cb0b6

```python import json from datasette import hookimpl import threading import requests import time

def change_over_time(m, metadata_value): while True: print(metadata_value) fetched = requests.get(metadata_value).json() counter = m["counter"] m.clear() m["counter"] = counter + 1 m.update(fetched) m["counter"] += 1 m["title"] = "{} {}".format(m.get("title", ""), m["counter"]) time.sleep(10)

@hookimpl(trylast=True) def load_metadata(metadata_value): m = { "counter": 0, } x = threading.Thread(target=change_over_time, args=(m, metadata_value), daemon=True) x.start() x.setName("datasette-metadata-counter") return m ``` It runs a separate thread that fetches the provided URL every 10 seconds:

datasette -m metadata.json --memory -p 8069 -m https://gist.githubusercontent.com/simonw/e8e4fcd7c0a9c951f7dd976921992157/raw/b702d18a6a078a0fb94ef1cee62e11a3396e0336/demo-metadata.json I learned a bunch of things from this prototype.

First, this is the wrong place to run the code:

https://github.com/simonw/datasette/blob/d11fd2cbaa6b31933b1319f81b5d1520726cb0b6/datasette/cli.py#L337-L343

I wanted the plugin hook to be able to receive a datasette instance, so implementations could potentially run their own database queries. Calling the hook in the CLI function here happens BEFORE the Datasette() instance is created, so that doesn't work.

I wanted to build a demo of a plugin that would load metadata periodically from an external URL (see #238) - but this threaded implementation is pretty naive. It results in a hit every 10 seconds even if no-one is using Datasette!

A smarter implementation would be to fetch and cache the results - then only re-fetch them if more than 10 seconds have passed since the last time the metadata was accessed.

But... doing this neatly requires asyncio - and the plugin isn't running inside an event loop (since uvicorn.run(ds.app()...) has not run yet so the event loop hasn't even started).

I could try and refactor everything so that all calls to read from metadata happen via await, but this feels like a pretty invasive change. It would be necessary if metadata might be read via a SQL query though.

Or maybe I could set it up so the plugin can start itself running in the event loop and call back to the datasette object to update metadata whenever it feels like it?

This needs to play nicely with asyncio - which means that the plugin hook needs to be able to interact with the event loop somehow.

That said... I don't particularly want to change everywhere that accesses metadata into a await call. So this is tricky.

I think only one plugin gets to work at a time. The plugin can return a dictionary which is used for live lookups of metadata every time it's accessed - which means the plugin can itself mutate that dictionary.

A plugin hook for this would enable #639. Renaming this issue.

Another potential use-case for this hook: loading metadata via a URL