Let me know if you want help prototyping any of this, because I'm thinking about it and trying stuff out. Happy to be a sounding board, if it helps.

Yeah I think this all hinges on:
- #1101

Also this comment about streaming full JSON arrays (not just newline-delimited) using this trick:
- https://github.com/simonw/datasette/issues/1356#issuecomment-1017016553

I'm about ready to figure these out, as with so much it's still a little bit blocked on the refactor stuff from:
- #1518

Thinking about this more, as well as #1356 and various other tickets related to output formats, I think there's a missing plugin hook for formatting results, separate from register_output_renderer (or maybe part of it, depending on #1101).

Right now, as I understand it, getting output in any format goes through the normal view stack -- a table, a row or a query -- and so by the time register_output_renderer gets it, the results have already been truncated or paginated. What I'd want, I think, is to be able to register ways to format results independent of where those results are sent.

It's possible this could be done using conn.row_factory (maybe in the prepare_connection hook), but I'm not sure that's where it belongs.

Another option is some kind of registry of serializers, which register_output_renderer and other plugin hooks could use. What I'm trying to avoid here is writing a plugin that also needs plugins for formats I haven't thought of yet.

Just shipped 0.60.1 with the fix - and tested that pip install datasette does indeed work correctly on Python 3.6.

So I don't need to release 0.60.1 AND 0.60.2 after all - I can just release 0.60.1 with a bug fix that it no longer breaks installation for Python 3.6.

Out of curiosity, I installed this latest setup.py file using both Python 3.6 and Python 3.10, ran pip freeze on both of them and created a Gist to compare the difference. The result is here: https://gist.github.com/simonw/2e7d5b1beba675ef9a5bcd310cadc372/revisions

From that, it looks like the Python packages in my dependencies which have released new versions that don't work with Python 3.6 are:

• https://pypi.org/project/janus/#history
• https://pypi.org/project/Pint/#history
• https://pypi.org/project/platformdirs/#history
• https://pypi.org/project/uvicorn/#history (already discussed)

Sure enough, for the first three of those browsing through their recent versions on PyPI confirms that they switched from e.g. "Requires: Python >=3.6" on https://pypi.org/project/janus/0.7.0/ to "Requires: Python >=3.7" on https://pypi.org/project/janus/1.0.0/

OK, the environment markers approach seems to work!

https://github.com/simonw/datasette/runs/4890775227?check_suite_focus=true - the tests passed on Python 3.6 for this commit with the pinned dependencies: https://github.com/simonw/datasette/commit/41060e7e7cb838328c879de6a98ae794dc1886d0

This whole thing reminds me of my ongoing internal debate about version pinning: should the Datasette package released to PyPI pin to the exact versions of the dependencies that are known to work, or should it allow a range of dependencies so users can pick other versions of the dependencies to use in their environment?

As I understand it, the general rule is to use exact pinning for applications but use ranges for libraries.

Datasette is almost entirely an application... but it can also be used as a library - and in fact I'm hoping to encourage that usage more in the future, see:
- #1398

I'd also like to release a packaged version of Datasette that doesn't require Uvicorn, for running on AWS Lambda and other function-as-a-service platforms. Those platforms have their own HTTP layer and hence don't need the Uvicorn dependency.

Maybe the answer is to have a datasette-core package which provides the core of Datasette with unpinned dependencies and no Uvicorn, and then have the existing datasette package provide the Datasette CLI tool with Uvicorn and pinned dependencies?

Another option from https://twitter.com/samuel_hames/status/1484327636860293121 - environment markers, described in https://www.python.org/dev/peps/pep-0508/#environment-markers

Found some examples of those in use using GitHub code search: https://cs.github.com/?scopeName=All+repos&scope=&q=%22%3Bpython_version%22+path%3Asetup.py - in particular https://github.com/xmendez/wfuzz/blob/1b695ee9a87d66a7d7bf6cae70d60a33fae51541/setup.py#L31-L38

install_requires = [
    'pycurl',
    'pyparsing<2.4.2;python_version<="3.4"',
    'pyparsing>=2.4*;python_version>="3.5"',
    'six',
    'configparser;python_version<"3.5"',
    'chardet',
]

So maybe I can ship 0.60.1 with loose dependencies except for the uvicorn one on Python 3.6, using an environment marker.

Here's my setup.py at the moment: https://github.com/simonw/datasette/blob/ffca55dfd7cc9b53522c2e5a2fa1ff67c9beadf2/setup.py#L44-L61

One other problem: there might be packages in that list right now which don't specify their 3.6 Python version but which will, at some point in the future, release a new version that doesn't work with 3.6 (like Uvicorn did) - in which case Python 3.6 installs would break in the future.

I think what I'll do then is ship the 0.60.1 Python 3.6 version with strict upper limits on each version which are the current, tested-with-Datasette-on-Python3.6 versions.

Problem: if I ship this, it will be the most recent release of Datasette - but unlike other previous releases it has exactly pinned versions of all of the dependencies. Which is bad for people who run pip install datasette but want to not be stuck to those exact library versions.

So maybe I ship this as 0.60.1, then ship a 0.60.2 release directly afterwards which unpins the dependencies again and requires Python 3.7?

I used the combo of pyenv and pipenv to run tests and figure out what the most recent versions of each dependency were that worked on Python 3.6. I also clicked around in the latest releases on pages such as https://pypi.org/project/aiofiles

cd /tmp
git clone git@github.com:simonw/datasette
cd /tmp/datasette
pipenv shell --python 3.6.10
pip install -e '.[test]'
pytest

I also used pip freeze | grep black to see which version was installed, since packages with python_requires= in them would automatically install the highest compatible version.

I think there are two possible solutions then:

1. Convince Uvicorn to publish one last 0.16.1 version which includes that python_requires= line, such that there's a version of Uvicorn on PyPI that Python 3.6 can still install.
2. Release a 0.60.1 version of Datasette which pins that Uvicorn version, and hence can be installed.

I've made the request for 1) in Uvicorn Gitter here: https://gitter.im/encode/community?at=61ea044a6d9ba23328d0fa28

I'm going to investigate option 2) myself now.

On Twitter: https://twitter.com/simonw/status/1484317711672877065

Here's the problem: Uvicorn only added python_requires to their setup.py a few days ago, which means the releases they have out on PyPI at the moment don't specify the Python version they need, which is why this mechanism doesn't work as expected:
- https://github.com/encode/uvicorn/pull/1328

Got a couple of TILs out of this:

• Promoting the stable version of the documentation using rel=canonical
• Linking from /latest/ to /stable/ on Read The Docs

https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for "You are looking at v0.36 but you should be looking at 0.40" but doesn't hand the case I need here which is "you are looking at /latest/ but you should be looking at /stable/".

Correction! That tool DOES support that, as can be seen in their example configuration for their own documentation:

https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38

versionwarning_messages = {
    'latest': 'This is a custom message only for version "latest" of this documentation.',
}
versionwarning_admonition_type = 'tip'
versionwarning_banner_title = 'Tip'
versionwarning_body_selector = 'div[itemprop="articleBody"]'

https://sphinx-version-warning.readthedocs.io/ looks like it can show a banner for "You are looking at v0.36 but you should be looking at 0.40" but doesn't hand the case I need here which is "you are looking at /latest/ but you should be looking at /stable/".

Just shipped my fix here: https://docs.datasette.io/en/latest/

https://user-images.githubusercontent.com/9599/150434433-e2f0d472-77a8-430b-88e5-f22e2c6e8ef0.png">

Or you can use https://sphinx-version-warning.readthedocs.io/! 😄

Here's a jQuery recipe that seems to do the right thing:

jQuery(function ($) {
  // If this is a /latest/ doc page, show banner linking to /stable/
  if (!/\/latest\//.test(location.pathname)) {
    return;
  }
  var stableUrl = location.pathname.replace("/latest/", "/stable/");
  // Check it's not a 404
  fetch(stableUrl, { method: "HEAD" }).then((response) => {
    if (response.status == 200) {
      var warning = $(
        `<div class="admonition warning">
           <p class="first admonition-title">Note</p>
           <p class="last">
             This documentation covers the <strong>development version</strong> of Datasette.</p>
             <p>See <a href="${stableUrl}">this page</a> for the current stable release.
           </p>
        </div>`
      );
      warning.find("a").attr("href", stableUrl);
      var body = $("div.body");
      if (!body.length) {
        body = $("div.document");
      }
      body.prepend(warning);
    }
  });
});

Turns out that banner is something that ReadTheDocs implemented - I found it using GitHub code search, it's produced by this piece of JavaScript: https://github.com/readthedocs/readthedocs.org/blob/0852d7c10d725d954d3e9a93513171baa1116d9f/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L13-L21

function init(data) {
    var rtd = rtddata.get();

    /// Out of date message

    if (data.is_highest) {
        return;
    }

    var currentURL = window.location.pathname.replace(rtd['version'], data.slug);
    var warning = $(
        '<div class="admonition warning"> ' +
        '<p class="first admonition-title">Note</p> ' +
        '<p class="last"> ' +
        'You are not reading the most recent version of this documentation. ' +
        '<a href="#"></a> is the latest version available.' +
        '</p>' +
        '</div>');

    warning
      .find('a')
      .attr('href', currentURL)
      .text(data.slug);

    var body = $("div.body");
    if (!body.length) {
        body = $("div.document");
    }
    body.prepend(warning);
}

And here's where that module is called from the rest of their code: https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/core/static-src/core/js/doc-embed/footer.js#L66-L86

I'm tempted to imitate their JavaScript but check for /latest/ in the URL and use that to append a similar message warning about this being the documentation for the in-development version.

Huh, I had forgotten I already have a banner on older versions:

Might want to consider SEO here too - I want people from search engines to land on /stable/, I wonder if I should noindex or rel=canonical the other documentation versions? Not sure what best practices for that is.

I think I want a banner at the top of the page making it obvious which version the documentation is talking about. This can be pretty low key for the current stable release, but should be visually more prominent for the /latest/ branch and for older releases.

Wrote up a TIL: https://til.simonwillison.net/pixelmator/pixel-editing-favicon

Resized it down to 208 bytes with https://squoosh.app

https://user-images.githubusercontent.com/9599/150401693-d19c7fdf-7f5d-49b4-8f1e-5c9e62cbdbed.png">

Here's a comparison between my hand-edited version and the one I have now:

The new 32x32 image:

https://user-images.githubusercontent.com/9599/150401295-8873fa40-8859-40e3-99d7-e53276319a15.png" width="32" height="32">

One last go at tidying this up. I decided to do a 32x32 pixel version in Pixelmator, using this trick to access a pixel brush: https://osxdaily.com/2016/11/17/enable-pixel-brush-pixelmator-mac/

https://user-images.githubusercontent.com/9599/150399724-10b17a97-3ecf-4d82-868c-f8a642f16838.png">

Frustrating how the white boxes are all exactly four pixels high and ALMOST all four pixels wide, but one of them has to be three pixels wide to fit the space.

Now live on https://latest.datastte.io/ - I'm happy enough with this for the moment:

https://user-images.githubusercontent.com/9599/150283069-dd23e688-ec77-4200-a892-38e0f8892f30.png">

Oops, I pushed the one I liked least out of the two options! Fixing now.

New favicon now live on https://latest.datasette.io/ - see also https://latest.datasette.io/favicon.ico

The new /favicon.ico view: https://github.com/simonw/datasette/blob/0467723ee57c2cbc0f02daa47cef632dd4651df0/datasette/app.py#L182-L192

My attempt with one less grid line looked awful when shrunk down to the browser size:

https://user-images.githubusercontent.com/9599/150280445-a482de8e-8b5f-4145-8567-36ac7105c303.png">

File here:

I'm going with the first attempt for the moment.

I'm not crazy about the look - I think this version of the grid may have too many lines for this particular display size.

I'm going to try reducing the number of lines in Figma to see if I like that better. https://www.figma.com/file/LKjceTFNtKm6wCbScDqm1Y/Datasette-Logo

Here's what it looks like in Firefox, Chrome and Safari:

I'm going to drop it in datasette/static/favicon.png and rewrite the async def favicon() function to return it, with caching headers.

Here's the 128x128 one - 11kb, I resized it using Preview:

Now running it through Squoosh using OxiPNG effort=3 colours=24 - brought it down to 1.36KB.

https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/ suggests 64x64, I'm going with 128x128 just in case anyone invents a retina-retina screen.

This one is 101KB and 1536*1536 which is a bit big! https://github.com/simonw/datasette-app/blob/main/build/icon.png

Fixed: https://docs.datasette.io/en/latest/plugins.html#seeing-what-plugins-are-installed

I don't have solid tests in place for exercising SpatiaLite, but this change feels safe enough that I'm not going to add tests for it.

I dropped support for Python 3.6 in fae3983c51f4a3aca8335f3e01ff85ef27076fbf so no free to use contextvars for this.

Dropped support for Python 3.6: updated setup.py, changed all references in the documentation to say 3.7 is the minimum required version, got rid of the GitHub Actions tests against 3.6.

OK, now that Uvicorn has dropped 3.6 support - see #1606 - I think this decision is easy to make. I'm dropping 3.6.

Relevant:
- https://github.com/simonw/datasette/issues/1577

https://pypistats.org/packages/datasette shows Python 3.6 is 0.24% of Datasette downloads.

https://user-images.githubusercontent.com/9599/150273301-18ec2ec8-f9ce-49a3-b57d-e74826a5c7ae.png">

https://github.com/simonw/latest-datasette-with-all-plugins/commit/1e12ffe70be791e3281b41810e837515314c1317 shows that 5 days ago Datasette upgraded from Uvicorn 0.16 to 0.17

Sure enough, in the changelog for 0.17: https://github.com/encode/uvicorn/blob/0.17.0/CHANGELOG.md

• Drop Python 3.6 support (#1261) 06/01/22

I need to add sqlite_stat1 to the hidden tables too, see:
- https://github.com/simonw/datasette/issues/1587
- https://github.com/simonw/sqlite-utils/issues/366

Oh interesting, I hadn't heard about KNN2. It looks like it was added to SpatiaLite on 21st June 2021 in https://www.gaia-gis.it/fossil/libspatialite/info/03786a62cdb4ab17 but the most recent release of SpatiaLite is 5.0.1 from 7th February 2021 - so it's not yet in a release.

A problem with this is that if you're using --query you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is .csv and plugin formats can't handle full streams, see #1062 and #1177.

I figured out a neat pattern for streaming JSON arrays in this TIL: https://til.simonwillison.net/python/output-json-array-streaming

Right now, I usually have a line in a Makefile like this:

combined.geojson: project.db
    pipenv run datasette project.db --get /project/combined.geojson \
     --load-extension spatialite \
     --setting sql_time_limit_ms 5000 \
     --setting max_returned_rows 20000 \
     -m metadata.yml > $@

That all assumes I've loaded whatever I need into project.db and created a canned query called combined (and then uses datasette-geojson for geojson output).

It works, but as you can see, it's a lot to manage, a lot of boilerplate, and it wasn't obvious how to get there. If there's an error in the canned query, I get an HTML error page, so that's hard to debug. And it's only one query, so each output needs a line like this. Make isn't ideal, either, for that reason.

The thing I really liked with datafreeze was doing templated filenames. I have a project now where I need to export a bunch of litttle geojson files, based on queries, and it would be awesome to be able to do something like this:

databases:
  project:
    queries:
      boundaries:
        sql: "SELECT * FROM boundaries"
        filename: "boundaries/{id}.geojson"
        mode: "item"
        format: geojson

And then do:

datasette freeze -m metadata.yml project.db

For HTML export, maybe there's a template argument, or format: template or something. And that gets you a static site generator, kinda for free.

Oh that's interesting. I was thinking about this from a slightly different angle recently - pondering what a static site generator built on top of Datasette might look like.

Just a sketch at the moment, but I was imagining a YAML configuration file with a SQL query that returns a list of paths - then a tool that runs that query and uses the equivalent of datasette --get to create a static copy of each of those paths.

I think these two ideas can probably be merged. I'd love to know more about how you are solving this right now!

I think both of these are Spatialite specific. They get generated when you first initialize the extension. KNN is actually deprecated in favor of KNN2, as I understand it.

Good idea - though I'm nervous about accidentally hiding a data_licenses table outside of the SpatiaLite case. I'll only hide that one if SpatiaLite is loaded.

Potential design:

datasette publish cloudrun data.db \
  --service my-service \
  --domain demo.datasette.io

I think I'm OK with calling this --domain even when it is being used with a subdomain.

This will require datasette.io to already have been verified with Google.

Not sure how best to handle the DNS part - maybe print out instructions for the necessary CNAME?

Also people can use a custom base template and link to a custom favicon if they want to.

For the moment then I will hard-code a new favicon and leave it to ASGI plugins if people want to define their own.

I don't think abusing the template loader mechanism for this will work: Jinja provides an API for loading text templates, but I don't think it can be sensibly abused to open binary image files instead. Loaded code is here: https://github.com/pallets/jinja/blob/main/src/jinja2/loaders.py

I'd like this to be customizable. I'm going to load it from the template loading system such that a custom favicon for a specific instance can be dropped in using a file in templates/.

Codecov Report

Merging #1602 (29f5772) into main (cb29119) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1602   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb29119...29f5772. Read the comment docs.

@simonw I have a PR here https://github.com/simonw/sqlite-utils/pull/385 that adds Spatialite helpers on the Python side. Please let me know how it looks.

There's actually already a way to move regular Datasette --setting options to a settings.json file thanks to configuration directory mode: https://docs.datasette.io/en/stable/settings.html

Maybe extending that mechanism to handle plugins would be a neat path forward here.

The datasette publish --plugin-secret name setting value option already implements something a bit like this. https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values

It's a bit of a messy hack to compensate for metadata being visible. Maybe I could replace that mechanism with the proposed plugin configuration rethink from this issue.

I still like the debug benefits of making plugin settings public - perhaps add a rule that if a plugin setting has a secret: prefix it gets redacted on a new /-/plugin-settings page.

This is now shipped!

• Upgraded Pluggy dependency to 1.0. #1575
• Now using Plausible analytics for the Datasette documentation.
• The db.execute_write() internals method now defaults to blocking until the write operation has completed. Previously it defaulted to queuing the write and then continuing to run code while the write was in the queue. #1579
• explain query plan is now allowed with varying amounts of white space in the query. #1588
• New CLI reference page showing the output of --help for each of the datasette sub-commands. This lead to several small improvements to the help copy. #1594
• Fixed bug where columns with a underscore prefix could result in unnecessary hidden form fields. #1527

Commits since 0.60a1: https://github.com/simonw/datasette/compare/0.60a1...3664ddd40

OK, I'm going to recommend a workaround for this instead. Here's asgi.py updated to strip the prefix before passing the request on to Datasette:

import pathlib
from asgi_cors import asgi_cors
from channels.routing import URLRouter
from django.urls import re_path
from datasette.app import Datasette


def rewrite_path(app, prefix_to_strip):
    async def rewrite_path_app(scope, receive, send):
        if (
            scope["type"] == "http"
            and "path" in scope
            and scope["path"].startswith(prefix_to_strip)
        ):
            scope["path"] = scope["path"][len(prefix_to_strip) :]
            if "raw_path" in scope:
                scope["raw_path"] = scope["raw_path"][len(prefix_to_strip) :]
        await app(scope, receive, send)

    return rewrite_path_app


datasette_ = Datasette(
    files=["fixtures.db"],
    settings={"base_url": "/datasettes/", "plugins": {}},
)
application = URLRouter(
    [
        re_path(
            r"^datasettes/.*",
            asgi_cors(rewrite_path(datasette_.app(), "/datasettes"), allow_all=True),
        ),
    ]
)

This works on my laptop - please re-open the ticket if it doesn't work for you!

I think this prefixed string mechanism is supposed to prevent the base_url prefix from being applied twice: https://github.com/simonw/datasette/blob/3664ddd400062123e99500d28b160c7944408c1a/datasette/url_builder.py#L9-L16

But with a bit of extra logging all of the inputs to that are NOT prefixed strings:

Urls.path called with: /datasettes/fixtures/compound_three_primary_keys?_sort=content (PrefixedUrlString = False)
  returning /datasettes/datasettes/fixtures/compound_three_primary_keys?_sort=content

So it looks like urls.path(...) is indeed the code responsible for doubling up that /datasettes/ prefix.

Since this is a special case bug for when using Datasette as a library I wonder if a good fix here would be to support something like this:

application = URLRouter([
  re_path(r"^datasettes/.*", asgi_cors(datasette_.app(remove_path_prefix="datasettes/"), allow_all=True)),
])

Bug is fixed on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1

There are four places in the documentation that use .. literalinclude:: at the moment which I can replace - I can actually just link directly to the new https://docs.datasette.io/en/latest/cli-reference.html page instead of embedding the help directly.

Oh! This is because _city_id has a leading underscore (for testing purposes). I think I need to filter out any keys that contain __ in that case.

What happens to columns that contain a __? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like _where and _m2m_through= and _col - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments

That's because that page has this unnecessary hidden form field:

<input type="hidden" name="_city_id__gt" value="1">

That field is added by this bit in the template: https://github.com/simonw/datasette/blob/515f8d38ebae203efc15ca79a8b42848276b35e5/datasette/templates/table.html#L119-L122

Which is populated here: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/views/table.py#L813-L821

Fixed:

% datasette --help        
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette is an open source multi-tool for exploring and publishing data

  About Datasette: https://datasette.io/
  Full documentation: https://docs.datasette.io/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect    Generate JSON summary of provided database files
...

% datasette inspect --help
Usage: datasette inspect [OPTIONS] [FILES]...

  Generate JSON summary of provided database files

  This can then be passed to "datasette --inspect-file" to speed up count
  operations against immutable database files.

Options:
  --inspect-file TEXT
  --load-extension TEXT  Path to a SQLite extension to load
  --help                 Show this message and exit.

Copying those in here:

• New plugin hook: filters_from_request(request, database, table, datasette), which runs on the table page and can be used to support new custom query string parameters that modify the SQL query. (#473)
• The number of unique values in a facet is now always displayed. Previously it was only displayed if the user specified ?_facet_size=max. (#1556)
• Fixed bug where ?_facet_array=tags&_facet=tags would only display one of the two selected facets. (#625)
• Facets of type date or array can now be configured in metadata.json, see Facets in metadata.json. Thanks, David Larlet. (#1552)
• New ?_nosuggest=1 parameter for table views, which disables facet suggestion. (#1557)
• Label columns detected for foreign keys are now case-insensitive, so Name or TITLE will be detected in the same way as name or title. (#1544)
• The query string variables exposed by request.args will now include blank strings for arguments such as foo in ?foo=&bar=1 rather than ignoring those parameters entirely. (#1551)
• Database write connections now execute the prepare_connection(conn, database, datasette) plugin hook. (#1564)
• The Datasette() constructor no longer requires the files= argument, and is now documented at Datasette class. (#1563)
• The tracing feature now traces write queries, not just read queries. (#1568)
• Added two methods for writing to the database: await db.execute_write_script(sql, block=False) and await db.execute_write_many(sql, params_seq, block=False). (#1570)
• Made several performance improvements to the database schema introspection code that runs when Datasette first starts up. (#1555)
• Fixed bug where writable canned queries could not be used with custom templates. (#1547)

Seeing as this area of the code has produced so many bugs in the past, I think part of the fix may be to write comprehensive documentation about how routing works for the internals documentation. Doing so might help me figure this bug out!

Most of these are already written for these two alpha releases:
- https://github.com/simonw/datasette/releases/tag/0.60a0
- https://github.com/simonw/datasette/releases/tag/0.60a1

This works now: https://latest.datasette.io/fixtures?sql=explain+query+plan++select+*+from+facetable

Challenge: explain the difference between view as in SQL view, and view as in the code that serves TableView / DatabaseView etc.

See also:
- #1412

Going with this for the copy:

Datasette Desktop is a packaged Mac application which bundles Datasette together with Python and allows you to install and run Datasette directly on your laptop. This is the best option for local installation if you are not comfortable using the command line.

https://docs.datasette.io/en/latest/cli-reference.html

I went with a simpler pattern that sqlite-utils because Datasette has a lot less commands: https://github.com/simonw/datasette/blob/4b23f01f3e668c8f2a2f1a294be49f49b4073969/docs/cli-reference.rst#L9-L35

The implementation can be almost exactly the same as this: https://github.com/simonw/sqlite-utils/blame/74586d3cb26fa3cc3412721985ecdc1864c2a31d/docs/cli-reference.rst#L11-L76

I need to do something extra to ensure the output of datasette --help-settings is shown too.

You can already run datasette --help-settings to see detailed help on available settings. Maybe datasette --help-plugin-settings could do the same thing for plugin settings?

Or I could even have available plugin settings show up as a list at the bottom of the datasette --help-settings output - which currently looks like this:

% datasette --help-settings
Settings:
  default_page_size            Default page size for the table view
                               (default=100)
  max_returned_rows            Maximum rows that can be returned from a table
                               or custom query (default=1000)
  num_sql_threads              Number of threads in the thread pool for
                               executing SQLite queries (default=3)
  sql_time_limit_ms            Time limit for a SQL query in milliseconds
                               (default=1000)
  default_facet_size           Number of values to return for requested facets
                               (default=30)
  facet_time_limit_ms          Time limit for calculating a requested facet
                               (default=200)
  facet_suggest_time_limit_ms  Time limit for calculating a suggested facet
                               (default=50)
  hash_urls                    Include DB file contents hash in URLs, for far-
                               future caching (default=False)
  allow_facet                  Allow users to specify columns to facet using
                               ?_facet= parameter (default=True)
  allow_download               Allow users to download the original SQLite
                               database files (default=True)
  suggest_facets               Calculate and display suggested facets
                               (default=True)
  default_cache_ttl            Default HTTP cache TTL (used in Cache-Control:
                               max-age= header) (default=5)
  default_cache_ttl_hashed     Default HTTP cache TTL for hashed URL pages
                               (default=31536000)
  cache_size_kb                SQLite cache size in KB (0 == use SQLite
                               default) (default=0)
  allow_csv_stream             Allow .csv?_stream=1 to download all rows
                               (ignoring max_returned_rows) (default=True)
  max_csv_mb                   Maximum size allowed for CSV export in MB - set
                               0 to disable this limit (default=100)
  truncate_cells_html          Truncate cells longer than this in HTML table
                               view - set 0 to disable (default=2048)
  force_https_urls             Force URLs in API output to always use https://
                               protocol (default=False)
  template_debug               Allow display of template debug information
                               with ?_context=1 (default=False)
  trace_debug                  Allow display of SQL trace debug information
                               with ?_trace=1 (default=False)
  base_url                     Datasette URLs should use this base path
                               (default=/)

Another option: if I make plugin settings a higher level concept in Datasette than they are at the moment, I could allow them to be set either using --options OR using the existing metadata.yml/json mechanism. https://docs.datasette.io/en/stable/plugins.html#plugin-configuration

I want to make changes to that anyway, because I'm increasingly uncomfortable with plugin settings ending up in the "metadata" mechanism.

I try to stick pretty closely to what Click supports, and Click likes you to define options explicitly so that it can display them in the output of --help.

But... that makes me think that actually showing these options in --help is likely a better idea. My --plugin-setting concept would have help that looks something like this:

% datasette serve --help
  ...
  --plugin-setting <TEXT TEXT TEXT>...     Setting for a specified plugin.

That's not great help!

The alternative would be to allow plugins to register their extra options with the command - which would mean the help output could look like this instead:

% datasette serve --help
  ...
  --tiddlywiki-db <TEXT>    Name of database to use for datasette-tiddlywiki

This feels like a good argument to me for plugins to explicitly register their settings. I'm not sure if I should enforce the name-of-plugin- prefix, or if I should allow plugins to define any setting they like. I'm still nervous about plugins over-riding existing or future default options to that command.

One more thing I'm going to add: A method to add a geometry column, which I'll need to do to create a spatial index on a table.

Other thing: If there get to be enough utils, I think it's worth moving all the spatialite stuff into its own file (gis.py or something) just so it's easier to find later.

Some proposals I'd add to sqlite-utils:

Some version of this, from geojson-to-sqlite:

def init_spatialite(db, lib):
    db.conn.enable_load_extension(True)
    db.conn.load_extension(lib)
    # Initialize SpatiaLite if not yet initialized
    if "spatial_ref_sys" in db.table_names():
        return
    db.conn.execute("select InitSpatialMetadata(1)")

Also a function for creating a spatial index:

db.conn.execute("select CreateSpatialIndex(?, ?)", [table, "geometry"])

I don't know the nuances of updating a spatial index, or checking if one already exists. This could be a CLI method like:

sqlite-utils spatial-index spatial.db table-name column-name

Came here to add this. I might pick it up.

Would also add a utility to create (and update and delete?) a spatial index. It's not much code but I have to look it up every time.

Codecov Report

Merging #1593 (df73ebb) into main (8c401ee) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1593   +/-   ##
=======================================
  Coverage   92.09%   92.09%           
=======================================
  Files          34       34           
  Lines        4516     4516           
=======================================
  Hits         4159     4159           
  Misses        357      357           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c401ee...df73ebb. Read the comment docs.

Wrote a lot more about this feature here: https://simonwillison.net/2022/Jan/12/how-i-build-a-feature/

Twitter: https://twitter.com/dracos/status/1481290103519592459

Is it possible to parse things like --ext-{plugin}-{arg} VALUE ?

The challenge here is avoiding clashes. What if a plugin adds an option that I later want to use for a new Datasette core feature? Or what if two plugins define the same option?

Maybe the solution is to make them use namespaces defined by the plugin name.

How about this:

datasette t.db --plugin-setting datasette-tiddlywiki db t.db

It's a bit verbose having an option that itself then takes THREE strings: plugin name, setting name, setting value - but it would work.

In my example, path matching happens at the application layer (being the Django channels URLRouter). That might be a somewhat exotic solution that would normally be solved by a proxy like Apache or Nginx. However, in my specific use case, this is a "feature" enabling me to do simple management of databases and metadata from within a Django admin app instance mapped in that same router.

Thank you for the quick reply! Just a quick observation, I am running this locally without a proxy, whereas your fly example seems to be running behind an apache proxy (if the name is accurate). Can it be that the apache proxy strips the prefix before it passes on the request to the daphne backend?

Looking closer at the code quoted above, it doesn't modify path or raw_path at all - ALL it does is add the route_path to the scope.

Oh wait! It looks like route_path is something I invented there.

Yup, I added it in https://github.com/simonw/datasette/commit/a63412152518581c6a3d4e142b937e27dabdbfdb - commit message says:

• new route_path key in request.scope storing the path that was used for routing with the base_url prefix stripped

So actually part of the mystery here is: why does the Fly hosted one NOT have that key?