{"html_url": "https://github.com/simonw/datasette/issues/357#issuecomment-647189045", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/357", "id": 647189045, "node_id": "MDEyOklzc3VlQ29tbWVudDY0NzE4OTA0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-06-21T22:19:58Z", "updated_at": "2020-06-21T22:19:58Z", "author_association": "OWNER", "body": "I'm going to take this in a different direction.\r\n\r\nI'm not happy with how `metadata.(json|yaml)` keeps growing new features. Rather than having a single plugin hook for all of `metadata.json` I'm going to split out the feature that shows actual real metadata for tables and databases - `source`, `license` etc - into its own plugin-powered mechanism.\r\n\r\nSo I'm going to close this ticket and spin up a new one for that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 348043884, "label": "Plugin hook for loading metadata.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/357#issuecomment-558461851", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/357", "id": 558461851, "node_id": "MDEyOklzc3VlQ29tbWVudDU1ODQ2MTg1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-26T05:05:21Z", "updated_at": "2019-11-26T05:05:21Z", "author_association": "OWNER", "body": "Here's an example plugin I set up using the experimental hook in d11fd2cbaa6b31933b1319f81b5d1520726cb0b6\r\n\r\n```python\r\nimport json\r\nfrom datasette import hookimpl\r\nimport threading\r\nimport requests\r\nimport time\r\n\r\n\r\ndef change_over_time(m, metadata_value):\r\n    while True:\r\n        print(metadata_value)\r\n        fetched = requests.get(metadata_value).json()\r\n        counter = m[\"counter\"]\r\n        m.clear()\r\n        m[\"counter\"] = counter + 1\r\n        m.update(fetched)\r\n        m[\"counter\"] += 1\r\n        m[\"title\"] = \"{} {}\".format(m.get(\"title\", \"\"), m[\"counter\"])\r\n        time.sleep(10)\r\n\r\n\r\n@hookimpl(trylast=True)\r\ndef load_metadata(metadata_value):\r\n    m = {\r\n        \"counter\": 0,\r\n    }\r\n    x = threading.Thread(target=change_over_time, args=(m, metadata_value), daemon=True)\r\n    x.start()\r\n    x.setName(\"datasette-metadata-counter\")\r\n    return m\r\n```\r\nIt runs a separate thread that fetches the provided URL every 10 seconds:\r\n\r\n```\r\ndatasette -m metadata.json --memory -p 8069 -m https://gist.githubusercontent.com/simonw/e8e4fcd7c0a9c951f7dd976921992157/raw/b702d18a6a078a0fb94ef1cee62e11a3396e0336/demo-metadata.json\r\n```\r\nI learned a bunch of things from this prototype.\r\n\r\nFirst, this is the wrong place to run the code:\r\n\r\nhttps://github.com/simonw/datasette/blob/d11fd2cbaa6b31933b1319f81b5d1520726cb0b6/datasette/cli.py#L337-L343\r\n\r\nI wanted the plugin hook to be able to receive a `datasette` instance, so implementations could potentially run their own database queries. Calling the hook in the CLI function here happens BEFORE the `Datasette()` instance is created, so that doesn't work.\r\n\r\nI wanted to build a demo of a plugin that would load metadata periodically from an external URL (see #238) - but this threaded implementation is pretty naive. It results in a hit every 10 seconds even if no-one is using Datasette!\r\n\r\nA smarter implementation would be to fetch and cache the results - then only re-fetch them if more than 10 seconds have passed since the last time the metadata was accessed.\r\n\r\nBut... doing this neatly requires asyncio - and the plugin isn't running inside an event loop (since `uvicorn.run(ds.app()...)` has not run yet so the event loop hasn't even started).\r\n\r\nI could try and refactor everything so that all calls to read from `metadata` happen via `await`, but this feels like a pretty invasive change. It would be necessary if metadata might be read via a SQL query though.\r\n\r\nOr maybe I could set it up so the plugin can start itself running in the event loop and call back to the `datasette` object to update metadata whenever it feels like it?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 348043884, "label": "Plugin hook for loading metadata.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/357#issuecomment-558459823", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/357", "id": 558459823, "node_id": "MDEyOklzc3VlQ29tbWVudDU1ODQ1OTgyMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-26T04:55:44Z", "updated_at": "2019-11-26T04:56:24Z", "author_association": "OWNER", "body": "This needs to play nicely with `asyncio` - which means that the plugin hook needs to be able to interact with the event loop somehow.\r\n\r\nThat said... I don't particularly want to change everywhere that accesses metadata into a `await` call. So this is tricky.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 348043884, "label": "Plugin hook for loading metadata.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/357#issuecomment-558446045", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/357", "id": 558446045, "node_id": "MDEyOklzc3VlQ29tbWVudDU1ODQ0NjA0NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-26T03:43:17Z", "updated_at": "2019-11-26T03:43:17Z", "author_association": "OWNER", "body": "I think only one plugin gets to work at a time. The plugin can return a dictionary which is used for live lookups of metadata every time it's accessed - which means the plugin can itself mutate that dictionary.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 348043884, "label": "Plugin hook for loading metadata.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/357#issuecomment-558432963", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/357", "id": 558432963, "node_id": "MDEyOklzc3VlQ29tbWVudDU1ODQzMjk2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2019-11-26T02:40:31Z", "updated_at": "2019-11-26T02:40:31Z", "author_association": "OWNER", "body": "A plugin hook for this would enable #639. Renaming this issue.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 348043884, "label": "Plugin hook for loading metadata.json"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/357#issuecomment-410818501", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/357", "id": 410818501, "node_id": "MDEyOklzc3VlQ29tbWVudDQxMDgxODUwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2018-08-06T19:04:54Z", "updated_at": "2018-08-06T19:04:54Z", "author_association": "OWNER", "body": "Another potential use-case for this hook: loading metadata via a URL", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 348043884, "label": "Plugin hook for loading metadata.json"}, "performed_via_github_app": null}