issue_comments

Weird... tests are failing in Travis, despite passing on my local machine. https://travis-ci.org/simonw/datasette/builds/367423706

This could also help workaround the current predicament that a single plugin can only define one prepare_connection hook.

Here's where that SQL gets constructed at the moment:

https://github.com/simonw/datasette/blob/10a34f995c70daa37a8a2aa02c3135a4b023a24c/datasette/app.py#L761-L771

But what would that SQL look like for _sort_desc?

I think the correct SQL is this: https://datasette-issue-189-demo-3.now.sh/salaries-7859114-7859114?sql=select+rowid%2C+*+from+%5B2017+Maryland+state+salaries%5D%0D%0Awhere+%28middle_initial+is+not+null+or+%28middle_initial+is+null+and+rowid+%3E+%3Ap0%29%29%0D%0Aorder+by+middle_initial+limit+101&p0=391

select rowid, * from [2017 Maryland state salaries] where (middle_initial is not null or (middle_initial is null and rowid > :p0)) order by middle_initial limit 101

Though this will also need to be taken into account for #198

I could use $null as a magic value that means None. Since I'm applying quote_plus() to actual values, any legit strings that look like this will be encoded as %24null:

```

urllib.parse.quote_plus('$null') '%24null' ```

Relevant code:

https://github.com/simonw/datasette/blob/904f1c75a3c17671d25c53b91e177c249d14ab3b/datasette/app.py#L828-L832

So there are two tricky problems to solve here:

• I need a way of encoding null into that _next= that is unambiguous from the string None or null. This means introducing some kind of escaping mechanism in those strings. I already use URL encoding as part of the construction of those components here, maybe that can help here?
• I need to figure out what the SQL should be for the "next" set of results if the previous value was null. Thankfully we use the primary key as a tie-breaker so this shouldn't be impossible.

Yikes, definitely a bug.

Annoyingly, the following only results in the last of the two prepare_connection hooks being registered:

``` from datasette import hookimpl import pint import random

ureg = pint.UnitRegistry()

@hookimpl def prepare_connection(conn): def convert_units(amount, from_, to_): "select convert_units(100, 'm', 'ft');" return (amount * ureg(from_)).to(to_).to_tuple()[0] conn.create_function('convert_units', 3, convert_units)

@hookimpl def prepare_connection(conn): conn.create_function('random_integer', 2, random.randint) ```

resource_stream returns a file-like object which may be better for serving from Sanic.

I should check if it's possible to have two template registration function plugins in a single plugin module. If it isn't maybe I should use class plugins instead of module plugins.

This looks like a good example: https://github.com/funkey/nyroglancer/commit/d4438ab42171360b2b8e9020f672846dd70c8d80

Packaging JS and CSS in a pip installable wheel is fiddly but possible. http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resources

from pkg_resources import resource_string
foo_config = resource_string(__name__, 'foo.conf')

I think this is pretty hard. @coleifer has done some work in this direction, including https://github.com/coleifer/pysqlite3 which ports the standalone pysqlite module to Python 3.

OK, from that prototype in f2720b0c6b7172ebe8820 it looks like pluggy provides a solid path forward.

Next steps:

• [x] Build a demo plugin that uses setuptools entrypoints to register with the datasette plugin manager via pluggy
• [x] Figure out a mechanism for registering plugins without first needing to publish them to PyPI. Can I load plugins from a special plugins/ directory similar to the --template-dir=templates/ option already supported by Datasette? #211

I think this is a good improvement. If you fix the tests I'll merge it.

Here's the result of running this:

datasette publish heroku fivethirtyeight.db \
    --plugins-dir=plugins/ --title="FiveThirtyEight" --branch=plugins-dir

https://intense-river-24599.herokuapp.com/fivethirtyeight-2628db9?sql=select+convert_units%28100%2C+%27m%27%2C+%27ft%27%29

Added unit tests in 33c6bcadb962457be6b0c7f369826b404e2bcef5

This worked as well:

datasette package fivethirtyeight.db \
    --plugins-dir=plugins/ --title="FiveThirtyEight" --branch=plugins-dir

Here's the result of running:

datasette publish now fivethirtyeight.db \
    --plugins-dir=plugins/ --title="FiveThirtyEight" --branch=plugins-dir

https://datasette-phjtvzwwzl.now.sh/fivethirtyeight-2628db9?sql=select+convert_units%28100%2C+%27m%27%2C+%27ft%27%29

Where plugins/pint_plugin.py contains the following: ``` from datasette import hookimpl import pint

ureg = pint.UnitRegistry()

@hookimpl def prepare_connection(conn): def convert_units(amount, from_, to_): "select convert_units(100, 'm', 'ft');" return (amount * ureg(from_)).to(to_).to_tuple()[0] conn.create_function('convert_units', 3, convert_units) ```

This needs unit tests. I also need to manually test the datasette package and datesette publish commands.

The easiest way to implement this in Python 2 would be execfile(...) - but that was removed in Python 3. According to https://stackoverflow.com/a/437857/6083 2to3 replaces that with this, which ensures the filename is associated with the code for debugging purposes:

with open("somefile.py") as f: code = compile(f.read(), "somefile.py", 'exec') exec(code, global_vars, local_vars)

Implementing it this way would force this kind of plugin to be self-contained in a single file. I think that's OK: if you want a more complex plugin you can use the standard pluggy-powered setuptools mechanism to build it.

I think Vega-Lite is the way to go here: https://vega.github.io/vega-lite/

I've been playing around with it and Datasette with some really positive initial results:

https://vega.github.io/editor/#/gist/vega-lite/simonw/89100ce80573d062d70f780d10e5e609/decada131575825875c0a076e418c661c2adb014/vice-shootings-gender-race-by-department.vl.json

https://vega.github.io/editor/#/gist/vega-lite/simonw/5f69fbe29380b0d5d95f31a385f49ee4/7087b64df03cf9dba44a5258a606f29182cb8619/trees-san-francisco.vl.json

Slight code design problem... when I tried installing my branch in a fresh virtual environment I got this error, because setup.py now depends on pluggy (from importing __version__):

File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/setup.py", line 2, in <module> from datasette import __version__ File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/__init__.py", line 2, in <module> from .hookspecs import hookimpl # noqa File "/private/var/folders/jj/fngnv0810tn2lt_kd3911pdc0000gp/T/pip-req-build-dftqdezt/datasette/hookspecs.py", line 1, in <module> from pluggy import HookimplMarker ModuleNotFoundError: No module named 'pluggy'

Looks like I've run into point 6 on https://packaging.python.org/guides/single-sourcing-package-version/ :

I created https://github.com/simonw/datasette-plugin-demos which is now published to PyPI and can be installed with pip install datasette-plugin-demos - I've confirmed that if you DO install it my Datasette plugins branch picks up the plugins, and select random_integer(1, 4) works as it should.