issue_comments

... actually not going to include this in 0.54, I need to write a couple of plugins myself using it before I even make it available in preview.

I'm going to ship a version of this in Datasette 0.54 with a warning that the interface should be considered unstable (see #1202) so that we can start trying this out.

Oh that's interesting, I hadn't thought about plugins firing events - just responding to events fired by the rest of the application.

I thought about using browser events, but they don't quite match the API that I'm looking to provide. In particular, the great thing about Pluggy is that if you have multiple handlers registered for a specific plugin hook each of those handlers can return a value, and Pluggy will combine those values into a list of replies.

This is great for things like plugin hooks that add extra menu items - each plugin can return a menu item (maybe as a label/URL/click-callback object) and the calling code can then add all of those items to the menu. See https://docs.datasette.io/en/stable/plugin_hooks.html#table-actions-datasette-actor-database-table for a Python example.

I'm on the fence about relying on JavaScript modules. I need to think about browser compatibility for them - but I'm already commited to requiring support for () => {} arrow functions so maybe I'm committed to module support too already?

https://github.com/mishoo/UglifyJS/issues/1905#issuecomment-300485490 says:

sourceMappingURL aren't added by default in 3.x due to one of the feature requests not to - some users are putting them within HTTP response headers instead.

So the command line for that would be:

js $ uglifyjs main.js -cmo main.min.js --source-map url=main.min.js.map

For inlining the plugins.min.js file into the Jinja templates I could use the trick described here: https://stackoverflow.com/a/41404611 - which adds a {{ include_file('file.txt') }} function to Jinja.

You'll be able to add JavaScript plugins using a bunch of different mechanisms:

• In a custom template, dropping the code in to a <script> block
• A bookmarklet that injects an extra script (I'm really excited to try this out)
• A separate script.js file that's loaded into Datasette using the "extra_js_urls" metadata option, documented here: https://docs.datasette.io/en/stable/custom_templates.html#custom-css-and-javascript
• A plugin you can install, like datasette-vega or datasette-cluster-map - since plugins can bundle their own script files that then get loaded on pages via this hook: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette

If I'm going to do that, it would be good if subsequent plugins that register against the load event are executed straight away. That's a bit of a weird edge-case in plugin world - it would involve the bulkier code that gets loaded redefining how datasette.plugins.register works to special-case the 'load' hook.

Maybe the tiny bootstrap code could define a datasette.plugins.onload(callbackFunction) method which gets upgraded later into something that fires straight away? Would add more bytes though.

Eventually I'd like to provide a whole bunch of other datasette.X utility functions that plugins can use - things like datasette.addTabbedContentPane() or similar.

But I don't want to inline those into the page.

So... I think the basic plugin system remains inline - maybe from an inlined file called plugins-bootstrap.js. Then a separate plugins.js contains the rest of the API functionality.

If a plugin wants to take advantage of those APIs, maybe it registers itself using datasette.plugins.register('load', () => ...) - that load hook can then be fired once the bulkier plugin code has been loaded.

I think I need to keep the mechanism whereby a plugin can return undefined in order to indicate that it has nothing to say for that specific item - that's borrowed from Pluggy and I've used it a bunch in my Python plugins. That makes the code a bit longer.

I'll write some example plugins to help me decide if the filtering-out-of-undefined mechanism is needed or not.

Using object destructuring like that is a great idea. I'm going to play with your version - it's delightfully succinct.

https://twitter.com/dracos/status/1344402639476424706 points out that plugins returning 0 will be ignored.

This should probably check for result !== undefined instead - knocks the size up to 250.

This one minifies to 241: javascript var datasette = datasette || {}; datasette.plugins = (() => { var registry = {}; return { register: (hook, fn, parameters) => { if (!registry[hook]) { registry[hook] = []; } registry[hook].push([fn, parameters]); }, call: (hook, args) => { args = args || {}; var results = []; (registry[hook] || []).forEach(([fn, parameters]) => { /* Call with the correct arguments */ var result = fn.apply(fn, parameters.map(parameter => args[parameter])); if (result) { results.push(result); } }); return results; } }; })(); var datasette=datasette||{};datasette.plugins=(()=>{var a={};return{register:(t,r,e)=>{a[t]||(a[t]=[]),a[t].push([r,e])},call:(t,r)=>{r=r||{};var e=[];return(a[t]||[]).forEach(([a,t])=>{var s=a.apply(a,t.map(a=>r[a]));s&&e.push(s)}),e}}})();

FixMyStreet inlines some JavaScript, and it's always a good idea to copy what they're doing when it comes to web performance: https://github.com/mysociety/fixmystreet/blob/23e9564b58a86b783ce47f3c0bf837cbd4fe7282/templates/web/base/common_header_tags.html#L19-L25

Note var fixmystreet=fixmystreet||{}; which is shorter - https://twitter.com/dracos/status/1344399909794045954

Another option: have both "dev" and "production" versions of the plugin mechanism script. Make it easy to switch between the two. Build JavaScript unit tests that exercise the "production" APIs against the development version, and have extra tests that just work against the features in the development version.

Started a Twitter thread about this here: https://twitter.com/simonw/status/1344392603794477056

This one is 683 bytes with Uglify - I like how https://skalman.github.io/UglifyJS-online/ shows you the minified character count as you edit the script: ```javascript window.datasette = window.datasette || {}; window.datasette.plugins = (() => { var registry = {}; var definitions = {}; var stringify = JSON.stringify;

function extractParameters(fn) {
    var match = /\((.*)\)/.exec(fn.toString());
    if (match && match[1].trim()) {
        return match[1].split(',').map(s => s.trim());
    } else {
        return [];
    }
}

function isSubSet(a, b) {
    return a.every(parameter => b.includes(parameter))
}

return {
    _r: registry,
    define: (hook, parameters) => {
        definitions[hook] = parameters || [];
    },
    register: (hook, fn, parameters) => {
        parameters = parameters || extractParameters(fn);
        if (!definitions[hook]) {
            throw 'Hook "' + hook + '" not defined';
        }
        /* Check parameters is a subset of definitions[hook] */
        var validParameters = definitions[hook];
        if (!isSubSet(parameters, validParameters)) {
            throw '"' + hook + '" valid args: ' + stringify(validParameters);
        }
        if (!registry[hook]) {
            registry[hook] = [];
        }
        registry[hook].push([fn, parameters]);
    },

    call: (hook, args) => {
        args = args || {};
        if (!definitions[hook]) {
            throw '"' + hook + '" hook not defined';
        }
        if (!isSubSet(Object.keys(args), definitions[hook])) {
            throw '"' + hook + '" valid args: ' + stringify(definitions[hook]);
        }

        var implementations = registry[hook] || [];
        var results = [];
        implementations.forEach(([fn, parameters]) => {
            /* Call with the correct arguments */
            var callWith = parameters.map(parameter => args[parameter]);
            var result = fn.apply(fn, callWith);
            if (result) {
                results.push(result);
            }
        });
        return results;
    }       
};

})(); ``window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var t={},r={},e=JSON.stringify;function i(t,r){return t.every(t=>r.includes(t))}return{_r:t,define:(t,e)=>{r[t]=e||[]},register:(a,n,o)=>{if(o=o||function(t){var r=/((.*))/.exec(t.toString());return r&&r[1].trim()?r[1].split(",").map(t=>t.trim()):[]}(n),!r[a])throw'Hook "'+a+'" not defined';var d=r[a];if(!i(o,d))throw'"'+a+'" valid args: '+e(d);t[a]||(t[a]=[]),t[a].push([n,o])},call:(a,n)=>{if(n=n||{},!r[a])throw'"'+a+'" hook not defined';if(!i(Object.keys(n),r[a]))throw'"'+a+'" valid args: '+e(r[a]);var o=t[a]||[],d=[];return o.forEach(([t,r])=>{var e=r.map(t=>n[t]),i=t.apply(t,e);i&&d.push(i)}),d}}})();`

I'm going to write a few example plugins and try them out against the longer and shorter versions of the script, to get a better feel for how useful the longer versions with the error handling and explicit definition actually are.

I gotta admit that 262 byte version is pretty tempting, if it's going to end up in the <head> of every single page.

262 bytes if I remove the parameter introspection code, instead requiring plugin authors to specify the arguments they take: javascript window.datasette = window.datasette || {}; window.datasette.plugins = (() => { var registry = {}; return { register: (hook, fn, parameters) => { if (!registry[hook]) { registry[hook] = []; } registry[hook].push([fn, parameters]); }, call: (hook, args) => { args = args || {}; var results = []; (registry[hook] || []).forEach(([fn, parameters]) => { /* Call with the correct arguments */ var callWith = parameters.map(parameter => args[parameter]); var result = fn.apply(fn, callWith); if (result) { results.push(result); } }); return results; } }; })(); window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var a={};return{register:(t,e,r)=>{a[t]||(a[t]=[]),a[t].push([e,r])},call:(t,e)=>{e=e||{};var r=[];return(a[t]||[]).forEach(([a,t])=>{var s=t.map(a=>e[a]),d=a.apply(a,s);d&&r.push(d)}),r}}})();

Removing the datasette.plugin.define() method and associated error handling reduces the uglified version from 683 bytes to 380 bytes. I think the error checking is worth the extra 303 bytes per page load, even if it's only really needed for a better developer experience. ```javascript window.datasette = window.datasette || {}; window.datasette.plugins = (() => { var registry = {};

function extractParameters(fn) {
    var match = /\((.*)\)/.exec(fn.toString());
    if (match && match[1].trim()) {
        return match[1].split(',').map(s => s.trim());
    } else {
        return [];
    }
}
return {
    register: (hook, fn, parameters) => {
        parameters = parameters || extractParameters(fn);
        if (!registry[hook]) {
            registry[hook] = [];
        }
        registry[hook].push([fn, parameters]);
    },

    call: (hook, args) => {
        args = args || {};
        var implementations = registry[hook] || [];
        var results = [];
        implementations.forEach(([fn, parameters]) => {
            /* Call with the correct arguments */
            var callWith = parameters.map(parameter => args[parameter]);
            var result = fn.apply(fn, callWith);
            if (result) {
                results.push(result);
            }
        });
        return results;
    }
};

})(); ``window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var t={};return{register:(r,a,e)=>{e=e||function(t){var r=/((.*))/.exec(t.toString());return r&&r[1].trim()?r[1].split(",").map(t=>t.trim()):[]}(a),t[r]||(t[r]=[]),t[r].push([a,e])},call:(r,a)=>{a=a||{};var e=t[r]||[],i=[];return e.forEach(([t,r])=>{var e=r.map(t=>a[t]),n=t.apply(t,e);n&&i.push(n)}),i}}})();`

I'm going to need to add JavaScript unit tests for this new plugin system.

This version minifies to 702 characters: ```javascript window.datasette = window.datasette || {}; window.datasette.plugins = (() => { var registry = {}; var definitions = {}; var stringify = JSON.stringify;

function extractParameters(fn) {
    var match = /\((.*)\)/.exec(fn.toString());
    if (match && match[1].trim()) {
        return match[1].split(',').map(s => s.trim());
    } else {
        return [];
    }
}

function isSubSet(a, b) {
    return a.every(parameter => b.includes(parameter))
}

return {
    _registry: registry,
    define: (hook, parameters) => {
        definitions[hook] = parameters || [];
    },
    register: (hook, fn, parameters) => {
        parameters = parameters || extractParameters(fn);
        if (!definitions[hook]) {
            throw '"' + hook + '" is not a defined hook';
        }
        /* Check parameters is a subset of definitions[hook] */
        var validParameters = definitions[hook];
        if (!isSubSet(parameters, validParameters)) {
            throw '"' + hook + '" valid args are ' + stringify(validParameters);
        }
        if (!registry[hook]) {
            registry[hook] = [];
        }
        registry[hook].push([fn, parameters]);
    },

    call: (hook, args) => {
        args = args || {};
        if (!definitions[hook]) {
            throw '"' + hook + '" hook is not defined';
        }
        if (!isSubSet(Object.keys(args), definitions[hook])) {
            throw '"' + hook + '" valid args: ' + stringify(definitions[hook]);
        }

        var implementations = registry[hook] || [];
        var results = [];
        implementations.forEach(([fn, parameters]) => {
            /* Call with the correct arguments */
            var callWith = parameters.map(parameter => args[parameter]);
            var result = fn.apply(fn, callWith);
            if (result) {
                results.push(result);
            }
        });
        return results;
    }       
};

})(); ``` Or 701 characters using https://skalman.github.io/UglifyJS-online/

Using raw string exceptions, throw '"' + hook + '" hook has not been defined';, knocks it down to 795 characters.

If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.

I need to decide how this code is going to be loaded. Putting it in a blocking <script> element in the head would work, but I'd rather not block loading of the rest of the page. Using a <script async> method would be nicer, but then I have to worry about plugins attempting to register themselves before the page has fully loaded.

Running it through https://javascript-minifier.com/ produces this, which is 855 characters - so maybe I could inline that into the header of the page?

window.datasette={},window.datasette.plugins=function(){var r={},n={};function e(r,n){return r.every(r=>n.includes(r))}return{define:function(r,e){n[r]=e||[]},register:function(t,i,o){if(o=o||function(r){var n=/\((.*)\)/.exec(r.toString());return n&&n[1].trim()?n[1].split(",").map(r=>r.trim()):[]}(i),!n[t])throw new Error('"'+t+'" is not a defined plugin hook');if(!n[t])throw new Error('"'+t+'" is not a defined plugin hook');var a=n[t];if(!e(o,a))throw new Error('"'+t+'" valid parameters are '+JSON.stringify(a));r[t]||(r[t]=[]),r[t].push([i,o])},_registry:r,call:function(t,i){if(i=i||{},!n[t])throw new Error('"'+t+'" hook has not been defined');if(!e(Object.keys(i),n[t]))throw new Error('"'+t+'" valid arguments are '+JSON.stringify(n[t]));var o=r[t]||[],a=[];return o.forEach(([r,n])=>{var e=n.map(r=>i[r]),t=r.apply(r,e);t&&a.push(t)}),a}}}();

This version adds datasette.plugins.define() plus extra validation of both .register() and .call(): ```javascript window.datasette = {}; window.datasette.plugins = (function() { var registry = {}; var definitions = {};

function extractParameters(fn) {
    var match = /\((.*)\)/.exec(fn.toString());
    if (match && match[1].trim()) {
        return match[1].split(',').map(s => s.trim());
    } else {
        return [];
    }
}

function define(hook, parameters) {
    definitions[hook] = parameters || [];
}

function isSubSet(a, b) {
    return a.every(parameter => b.includes(parameter))
}

function register(hook, fn, parameters) {
    parameters = parameters || extractParameters(fn);
    if (!definitions[hook]) {
        throw new Error('"' + hook + '" is not a defined plugin hook');
    }
    if (!definitions[hook]) {
        throw new Error('"' + hook + '" is not a defined plugin hook');
    }
    /* Check parameters is a subset of definitions[hook] */
    var validParameters = definitions[hook];
    if (!isSubSet(parameters, validParameters)) {
        throw new Error('"' + hook + '" valid parameters are ' + JSON.stringify(validParameters));
    }
    if (!registry[hook]) {
        registry[hook] = [];
    }
    registry[hook].push([fn, parameters]);
}

function call(hook, args) {
    args = args || {};
    if (!definitions[hook]) {
        throw new Error('"' + hook + '" hook has not been defined');
    }
    if (!isSubSet(Object.keys(args), definitions[hook])) {
        throw new Error('"' + hook + '" valid arguments are ' + JSON.stringify(definitions[hook]));
    }

    var implementations = registry[hook] || [];
    var results = [];
    implementations.forEach(([fn, parameters]) => {
        /* Call with the correct arguments */
        var callWith = parameters.map(parameter => args[parameter]);
        var result = fn.apply(fn, callWith);
        if (result) {
            results.push(result);
        }
    });
    return results;
}
return {
    define: define,
    register: register,
    _registry: registry,
    call: call
};

})(); Usage:javascript datasette.plugins.define('numbers', ['a', 'b']) datasette.plugins.register('numbers', (a, b) => a + b) datasette.plugins.register('numbers', (a, b) => a * b) datasette.plugins.call('numbers', {a: 4, b: 6}) ```

This could work to define a plugin hook: javascript datasette.plugins.define('numbers', ['a' ,'b'])

This implementation doesn't have an equivalent of "hookspecs" which can identify if a registered plugin implementation matches a known signature. I should add that, it will provide a better developer experience if someone has a typo.

Initial prototype: ```javascript window.datasette = {}; window.datasette.plugins = (function() { var registry = {};

function extractParameters(fn) {
    var match = /\((.*)\)/.exec(fn.toString());
    if (match && match[1].trim()) {
        return match[1].split(',').map(s => s.trim());
    } else {
        return [];
    }
}

function register(hook, fn, parameters) {
    parameters = parameters || extractParameters(fn);
    if (!registry[hook]) {
        registry[hook] = [];
    }
    registry[hook].push([fn, parameters]);
}

function call(hook, args) {
    args = args || {};
    var implementations = registry[hook] || [];
    var results = [];
    implementations.forEach(([fn, parameters]) => {
        /* Call with the correct arguments */
        var callWith = parameters.map(parameter => args[parameter]);
        var result = fn.apply(fn, callWith);
        if (result) {
            results.push(result);
        }
    });
    return results;
}
return {
    register: register,
    _registry: registry,
    call: call
};

})(); Usage example:javascript datasette.plugins.register('numbers', (a, b) => a + b) datasette.plugins.register('numbers', (a, b) => a * b) datasette.plugins.call('numbers', {a: 4, b: 6}) / Returns [10, 24] / ```

The column_actions hook is the obvious first place to try this out. What are some demo plugins I could build for it?

• Word cloud for this column
• Count values (essentially linking to the SQL query for that column, as an extended version of the facet counts) - would be great if this could include pagination somehow, via #856.
• Extract this column into a separate table
• Add an index to this column

Then to call the plugins: javascript datasette.plugins.call('column_actions', {database: 'database', table: 'table'})

Potential design: javascript datasette.plugins.register('column_actions', function(database, table, column, actor) { /* ... *l }) Or if you want to be explicit to survive minification: javascript datasette.plugins.register('column_actions', function(database, table, column, actor) { /* ... *l }, ['database', 'table', 'column', 'actor']) I'm making that list of parameter names an optional third argument to the register() function. If that argument isn't passed, introspection will be used to figure out the parameter names.

Pluggy does dependency injection by introspecting the named arguments to the Python function, which I really like.

That's tricker in JavaScript. It looks like the only way to introspect a function is to look at the .toString() representation of it and parse the (parameter, list) using a regular expression.

Even more challenging: JavaScript developers love minifying their code, and minification can shorten the function parameter names.

From https://code-maven.com/dependency-injection-in-angularjs it looks like Angular.js does dependency injection and solves this by letting you optionally provide a separate list of the arguments your function uses:

javascript angular.module('DemoApp', []) .controller('DemoController', ['$scope', '$log', function($scope, $log) { $scope.message = "Hello World"; $log.debug('logging hello'); }]);

I can copy that approach: I'll introspect by default, but provide a documented mechanism for explicitly listing your parameter names so that if you know your plugin code will be minified you can use that instead.

I'm going to introduce a global datasette object which holds all the documented JavaScript API for plugin authors.

I think I'm going to try building a very lightweight clone of the core API design of Pluggy - not the advanced features, just the idea that plugins can register and a call to plugin.nameOfHook() will return the concatenated results of all of the registered hooks.

@yozlet just spotted this comment. Wow that is interesting!

With the right plugin hooks on the page (see also #987) one relatively simple way to do that could be with bookmarklets - users could install bookmarklets which, when executed against a Datasette page in their browser, use the existing JavaScript plugin integration points to add all kinds of functionality.

Doing full sandboxing is certainly daunting, but it looks like Figma figured it out so TIL it's technically feasible.

I had a look around and there isn't an obvious pluggy equivalent in JavaScript world at the moment. Lots of frameworks like jQuery and Vue have their own custom plugin mechanisms.

https://github.com/rekit/js-plugin is a simple standalone plugin mechanism. Not quite as full-featured as Pluggy though - in particular I like how Pluggy supports multiple plugins returning results for the same hook that get concatenated into a list of results.

https://css-tricks.com/designing-a-javascript-plugin-system/ has some ideas.