html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/357#issuecomment-647189045,https://api.github.com/repos/simonw/datasette/issues/357,647189045,MDEyOklzc3VlQ29tbWVudDY0NzE4OTA0NQ==,9599,simonw,2020-06-21T22:19:58Z,2020-06-21T22:19:58Z,OWNER,"I'm going to take this in a different direction.
I'm not happy with how `metadata.(json|yaml)` keeps growing new features. Rather than having a single plugin hook for all of `metadata.json` I'm going to split out the feature that shows actual real metadata for tables and databases - `source`, `license` etc - into its own plugin-powered mechanism.
So I'm going to close this ticket and spin up a new one for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",348043884,Plugin hook for loading metadata.json,
https://github.com/simonw/datasette/issues/357#issuecomment-558461851,https://api.github.com/repos/simonw/datasette/issues/357,558461851,MDEyOklzc3VlQ29tbWVudDU1ODQ2MTg1MQ==,9599,simonw,2019-11-26T05:05:21Z,2019-11-26T05:05:21Z,OWNER,"Here's an example plugin I set up using the experimental hook in d11fd2cbaa6b31933b1319f81b5d1520726cb0b6
```python
import json
from datasette import hookimpl
import threading
import requests
import time
def change_over_time(m, metadata_value):
while True:
print(metadata_value)
fetched = requests.get(metadata_value).json()
counter = m[""counter""]
m.clear()
m[""counter""] = counter + 1
m.update(fetched)
m[""counter""] += 1
m[""title""] = ""{} {}"".format(m.get(""title"", """"), m[""counter""])
time.sleep(10)
@hookimpl(trylast=True)
def load_metadata(metadata_value):
m = {
""counter"": 0,
}
x = threading.Thread(target=change_over_time, args=(m, metadata_value), daemon=True)
x.start()
x.setName(""datasette-metadata-counter"")
return m
```
It runs a separate thread that fetches the provided URL every 10 seconds:
```
datasette -m metadata.json --memory -p 8069 -m https://gist.githubusercontent.com/simonw/e8e4fcd7c0a9c951f7dd976921992157/raw/b702d18a6a078a0fb94ef1cee62e11a3396e0336/demo-metadata.json
```
I learned a bunch of things from this prototype.
First, this is the wrong place to run the code:
https://github.com/simonw/datasette/blob/d11fd2cbaa6b31933b1319f81b5d1520726cb0b6/datasette/cli.py#L337-L343
I wanted the plugin hook to be able to receive a `datasette` instance, so implementations could potentially run their own database queries. Calling the hook in the CLI function here happens BEFORE the `Datasette()` instance is created, so that doesn't work.
I wanted to build a demo of a plugin that would load metadata periodically from an external URL (see #238) - but this threaded implementation is pretty naive. It results in a hit every 10 seconds even if no-one is using Datasette!
A smarter implementation would be to fetch and cache the results - then only re-fetch them if more than 10 seconds have passed since the last time the metadata was accessed.
But... doing this neatly requires asyncio - and the plugin isn't running inside an event loop (since `uvicorn.run(ds.app()...)` has not run yet so the event loop hasn't even started).
I could try and refactor everything so that all calls to read from `metadata` happen via `await`, but this feels like a pretty invasive change. It would be necessary if metadata might be read via a SQL query though.
Or maybe I could set it up so the plugin can start itself running in the event loop and call back to the `datasette` object to update metadata whenever it feels like it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",348043884,Plugin hook for loading metadata.json,
https://github.com/simonw/datasette/issues/357#issuecomment-558459823,https://api.github.com/repos/simonw/datasette/issues/357,558459823,MDEyOklzc3VlQ29tbWVudDU1ODQ1OTgyMw==,9599,simonw,2019-11-26T04:55:44Z,2019-11-26T04:56:24Z,OWNER,"This needs to play nicely with `asyncio` - which means that the plugin hook needs to be able to interact with the event loop somehow.
That said... I don't particularly want to change everywhere that accesses metadata into a `await` call. So this is tricky.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",348043884,Plugin hook for loading metadata.json,
https://github.com/simonw/datasette/issues/357#issuecomment-558446045,https://api.github.com/repos/simonw/datasette/issues/357,558446045,MDEyOklzc3VlQ29tbWVudDU1ODQ0NjA0NQ==,9599,simonw,2019-11-26T03:43:17Z,2019-11-26T03:43:17Z,OWNER,I think only one plugin gets to work at a time. The plugin can return a dictionary which is used for live lookups of metadata every time it's accessed - which means the plugin can itself mutate that dictionary.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",348043884,Plugin hook for loading metadata.json,
https://github.com/simonw/datasette/issues/357#issuecomment-558432963,https://api.github.com/repos/simonw/datasette/issues/357,558432963,MDEyOklzc3VlQ29tbWVudDU1ODQzMjk2Mw==,9599,simonw,2019-11-26T02:40:31Z,2019-11-26T02:40:31Z,OWNER,A plugin hook for this would enable #639. Renaming this issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",348043884,Plugin hook for loading metadata.json,
https://github.com/simonw/datasette/issues/357#issuecomment-410818501,https://api.github.com/repos/simonw/datasette/issues/357,410818501,MDEyOklzc3VlQ29tbWVudDQxMDgxODUwMQ==,9599,simonw,2018-08-06T19:04:54Z,2018-08-06T19:04:54Z,OWNER,Another potential use-case for this hook: loading metadata via a URL,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",348043884,Plugin hook for loading metadata.json,