I'm landing this despite the cog failures. I'll fix them on main if I have to.

Codecov Report

All modified lines are covered by tests :white_check_mark:

Comparison is base (3feed1f) 92.46% compared to head (8ae479c) 92.69%. Report is 112 commits behind head on main.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Oh! I think I broke Cog on main and these tests are running against this branch rebased against main.

Weird, the cog check is failing in CI.

Run cog --check docs/*.rst cog --check docs/*.rst shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib Check failed Checking docs/authentication.rst Checking docs/binary_data.rst Checking docs/changelog.rst Checking docs/cli-reference.rst Checking docs/configuration.rst (changed)

This is such a well thought out contribution. I don't think I've seen such a thoroughly considered PR on any project in recent memory.

Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the table and database views.

I tried running this locally just now. I made one edit: diff diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py index 7bdb9f3f..f9dfa8e6 100644 --- a/demos/plugins/example_js_manager_plugins.py +++ b/demos/plugins/example_js_manager_plugins.py @@ -15,6 +15,6 @@ def extra_js_urls(view_name): if view_name in PERMITTED_VIEWS: return [ { - "url": f"/-/demos/plugins/static/table-example-plugins.js", + "url": f"/static/table-example-plugins.js", } ] And then started it running like this: bash wget https://datasette.io/content.db bash datasette content.db \ --plugins-dir demos/plugins \ --static static:datasette/demos/plugins/static

It didn't quite work for me - I got this error on a table page:

And this error on a query page:

Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

I just did a github search for user:simonw "def extra_js_urls(" ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code

Ok @hydrosquall a couple things before this PR should be good to go:

Thank you @asg017 ! I've pushed both suggested changes onto this branch.

Not sure how difficult it'll be to inject it server-side

If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a "template" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.

In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both version.py and datasette-manager.js, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a "manual sync" might be good enough.

In terms of how to integrate this into Datasette, a few options I can see working:

This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.

I'm also curious to see how "plugins for a plugin' would work

I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a "public API of other plugins", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that.

here's a list of Simon's Datasette plugins that use "extra_js_urls()"

Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)

I look forward to trying this out 👍

Ok @hydrosquall a couple things before this PR should be good to go:

• Can we move datasette/static/table-example-plugins.js into demos/plugins/static?
• For datasetteManager.VERSION, can we fill that in or just comment it out for now? Not sure how difficult it'll be to inject it server-side. I imagine we could also have a small build process with esbuild/rollup that just injects a version string into manager.js directly, so we don't have to worry about server-rendering (but that can be a future PR)

In terms of how to integrate this into Datasette, a few options I can see working:

• Push this as-is and figure it out before the next release
• Hide this feature behind a settings flag (--setting unstable-js-plugins on) and use that setting to hide/show <script src="{{ urls.static('datasette-manager.js') }}" defer></script> in base.html

I'll let @simonw decide which one to work with. I kindof like the idea of having an "unstable" opt-in process to enable JS plugins, to give us time to try it out with a wide variety of plugins until we feel its ready.

I'm also curious to see how "plugins for a plugin' would work, like #1542. For example, if the leaflet plugin showed default markers, but also included its own hook for other plugins to add more markers/styling. I'm imagine that the individual plugin would re-create their own plugin system compared to this, since handling "plugins of plugins" at the top with Datasette seems really convoluted.

Also for posterity, here's a list of Simon's Datasette plugins that use "extra_js_urls()", which probably means they can be ported/re-written to use this new plugin system:

• datasette-vega
• datasette-cluster-map
• datasette-leaflet-geojson
• datasette-pretty-traces
• datasette-youtube-embed
• datasette-leaflet-freedraw
• datasette-hovercards
• datasette-mp3-audio
• datasette-geojson-map

:wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch

Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.

Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

Ah, I understand now! Thanks for explaining.

Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?

The latter - that you could return a promise of arrays, so it parallels the "await me maybe" pattern in Datasette, where you can return either a value, a callable or an awaitable.

I have a hunch that what you're describing might be achievable without adding Promises to the API with something

Oops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

This isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place.

This could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent.

especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag

Ah, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with.

My experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin.

Again, a fairly minor thing, though.

Hey @cldellow , thanks for the thoughtful feedback and describing the "lazy facets" feature!

It sounds like the postTask API might be relevant for the types of network request scheduling you have in mind.

Addressing your points inline below:

It might also be nice if the plugins could return Promises.

Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?

I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable .

I have a hunch that what you're describing might be achievable without adding Promises to the API with something like

fetch('/api/with-custom-facets').then(myFacets => { // reusing the go() idiom go(manager, myFacets); })

but I'd like to confirm if that's the case before investigating adding support.

bulletproof plugin registration code that is robust against the order in which the script tags load

Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.

I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:

I can see that this approach brings advantages

• Plugin developers don't need to know the name of the datasette initialization event to start their plugin
• Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener
• One less event listener sitting in memory

It also has some minor costs

• A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up
• Some risk in encouraging people to mutate global state
• (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag).

Also, just checking - is this how I'd write bulletproof plugin registration code that is robust against the order in which the script tags load (eg if both my code and the Datasette code are loaded via a <script async src='...'/> tag)?

```js if (window.DATASETTE) go(window.DATASETTE); else document.addEventListener("datasette_init", (evt) => go(evt.detail));

function go(manager) { manager.registerPlugin(...) } ```

I don't know if it'd make sense, but you could also consider the asynchronous queuing pattern that Google Analytics uses (see this Stack Overflow post for more details):

```js DATASETTE = DATASETTE || []; DATASETTE.push(go);

function go(manager) { manager.registerPlugin(...); } ```

Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(

tl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return Promises.

The long version: I read the design notes and example plugin. I think I'd be able to use this in datasette-ui-extras for my lazy-facets feature.

The lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.

First, at page render time: (A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large. (B) there's a python plugin that implements the extra_body_script hook, to write out the list of column names for future use by JavaScript

Second, at page load time: there is some JavaScript that: (C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B) (D) wires up the column menus to add Facet-by-this options for each facet

With the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.

If the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.

Does that make sense? Sorry for the wall of text!

Javascript Plugin Docs (alpha)

Motivation

The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code.

Setup

No external/NPM dependencies are needed.

Plugin behavior is coordinated by the Datasette manager. Every page has 1 manager.

There are 2 ways to add your plugin to the manager.

1. Read window.__DATASETTE__ if the manager was already loaded.

js const manager = window.__DATASETTE__;

1. Wait for the datasette_init event to fire if your code was loaded before the manager is ready.

```js document.addEventListener("datasette_init", function (evt) { const { detail: manager } = evt;

// register plugin here }); ```

1. Add plugin to the manager by calling manager.registerPlugin in a JS file. Each plugin will supply 1 or more hooks with

2. unique name (YOUR_PLUGIN_NAME)

3. a numeric version (starting at 0.1),
4. configuration value, the details vary by hook. (In this example, getColumnActions takes a function)

js manager.registerPlugin("YOUR_PLUGIN_NAME", { version: 0.1, makeColumnActions: (columnMeta) => { return [ { label: "Copy name to clipboard", // evt = native click event onClick: (evt) => copyToClipboard(columnMeta.column), } ]; }, });

There are 2 plugin hooks available to manager.registerPlugin:

• makeColumnActions - Add items to the cog menu for headers on datasette table pages
• makeAboveTablePanelConfigs - Add items to "tabbed" panel above the <table/> on pages that use the Datasette table template.

While there are additional properties on the manager, but it's not advised to depend on them directly as the shape is subject to change.

1. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling this to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.

I welcome ideas for more hooks, or feedback on the current design!

Examples

See the example plugins file for additional examples.

Hooks API Guide

makeAboveTablePanelConfigs

Provide a function with a list of panel objects. Each panel object should contain

1. A unique string id
2. A string label for the tab
3. A render function. The first argument is reference to an HTML Element.

Example:

js manager.registerPlugin("panel-plugin-graphs", { version: 0.1, makeAboveTablePanelConfigs: () => { return [ { id: 'first-panel', label: "My new panel", render: node => { const description = document.createElement('p'); description.innerText = 'Hello world'; node.appendChild(description); } } ]; }, });

makeColumnActions

Provide a function that returns a list of action objects. Each action object has

1. A string label for the menu dropdown label
2. An onClick render function.

Example:

```js manager.registerPlugin("column-name-plugin", { version: 0.1, getColumnActions: (columnMeta) => {

  // Info about selected column. 
  const { columnName, columnNotNull, columnType, isPk } = columnMeta;

  return [
    {
      label: "Copy name to clipboard",
      onClick: (evt) => copyToClipboard(column),
    }
  ];
},

}); ```

The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:

• columnName: string (name of the column)
• columnNotNull: boolean
• columnType: sqlite datatype enum (text, number, etc)
• isPk: Whether this is the primary key: boolean

You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).

Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.

comments

These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week.

I know TypeScript can be a little controversial

FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files.

I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.

Maybe they should have more "action-y" names

This is a great observation. I'm inclined towards something like make* or build* since to me add* make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current get* setup. I'll go through and update these.

Maybe we can make it easier to do pure-js datasette plugins?

I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at all.

And then do the PERMITTED_VIEWS filtering in JS rather than Python.

One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal.

Research notes

• I stuck to the "minimal dependencies" ethos of datasette (no React, Typescript, JS linting, etc).
• Main threads on JS plugin development
• Main: sketch of pluggy-inspired system: https://github.com/simonw/datasette/issues/983
• Main: provide locations in Datasette HTML that are designed for multiple plugins to safely cooperate with each other (starting with the panel, but eventually could extend to "search boxes"): https://github.com/simonw/datasette/issues/1191
• Main: HTML hooks for JS plugin authors: https://github.com/simonw/datasette/issues/987
• Prior threads on JS plugins in Datasette for future design directions
• Idea: pass useful strings to JS plugins: https://github.com/simonw/datasette/issues/1565
• Idea: help with plugin dependency loading: https://github.com/simonw/datasette/issues/1542 . (IMO - the plugin providing the dependency can emit an event once it's done. Other plugins can listen for it, or ask the manager to inform them when the dependency is available).
• Idea: help plugins to manage state in shareable URLs (plugins shouldn't have to interact with the URL directly, should have some basic insulation from clobbering each others' keys): https://github.com/simonw/datasette/issues/1144
• Articles on plugins reviewed
• https://css-tricks.com/designing-a-javascript-plugin-system/
• Plugin/Extension systems reviewed (mostly JS).
• Yarn: https://yarnpkg.com/advanced/plugin-tutorial
• Tappable https://github.com/webpack/tapable (used by Auto, webpack)
• Pluggy: https://pluggy.readthedocs.io/en/stable/
• VSCode: https://code.visualstudio.com/api/get-started/your-first-extension
• Chrome: https://developer.chrome.com/docs/extensions/reference/
• Figma/Figjam Widget: https://www.figma.com/widget-docs/
• Datadog Apps: Programming Model
• Storybook: https://storybook.js.org/docs/react/addons/addons-api

Notes from 1:1 - it is possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters