issue_comments

I'm going to try using Jinja macros to implement this: https://jinja.palletsprojects.com/en/2.11.x/templates/#macros

Maybe using one of these tricks to auto-load the macro? http://codyaray.com/2015/05/auto-load-jinja2-macros

Here's the incomplete sketch of a test - to go at the bottom of test_cli.py.

@pytest.mark.parametrize(
    "filename", ["test-database (1).sqlite", "database (1).sqlite"]
)
def test_weird_database_names(ensure_eventloop, tmpdir, filename):
    # https://github.com/simonw/datasette/issues/1181
    runner = CliRunner()
    db_path = str(tmpdir / filename)
    sqlite3.connect(db_path).execute("vacuum")
    result1 = runner.invoke(cli, [db_path, "--get", "/"])
    assert result1.exit_code == 0, result1.output
    homepage_html = result1.output
    assert False

I'm going to prototype this in a branch.

I think this ends up being a whole collection of new plugin hooks, something like:

• include_table_top
• include_table_bottom
• include_row_top
• include_row_bottom
• include_database_top
• include_database_bottom
• include_query_bottom
• include_query_bottom
• include_index_bottom
• include_index_bottom

Plugins that want to provide extra context to the template can already do so using the extra_template_vars() plugin hook.

So this hook could work by returning a list of template filenames to be included. Those templates can be bundled with the plugin. Since they are included they will have access to the template context and to any extra_template_vars() values.

I'm not sure how I missed this issue but it's almost a year later and I'm finally taking a look at your Parquet work.

This is yet more evidence that allowing plugins to provide their own custom Database objects would be a good idea.

I started exploring what Datasette would like on PostgreSQL in #670 - my concern was that I would need to add a large amount of database abstraction code which would dramatically increase the complexity of the core project, but my thinking now is that it might be tractable - Datasette doesn't actually construct SQL in complex ways anywhere outside of the TableView class so abstracting away just that bit should be feasible.

Also related: #857 (comprehensive documentation of variables available to templates) - since then the plugin hook could be fed the full template context and use that to do its thing.

Or maybe the plugin hooks get to return the name of a template that should be {% include %} into the page at that point? But the plugin may want to add extra context that is available to that template include.

This relates to #987 (documented HTML hooks for JavaScript plugins) but is not quite the same thing.

I think the startup() plugin hook at https://docs.datasette.io/en/stable/plugin_hooks.html#startup-datasette should be able to fit this. You can write a plugin which uses that hook to execute CREATE VIRTUAL TABLE against one or more databases when Datasette first starts running.

Would that work here?

Make sense. If you're coming from the sqlite3 side of things, rather than the datasette side, wanting the fts methods to work for views makes more sense. sqlite3 allows fts5 tables on views, so I was looking for CLI functionality to build the fts virtual tables. Ultimately, though, sharing fts virtual tables across tables and derivative views is likely more efficient.

Maybe an explicit error message like, "fts is not supported for views" rather than just throwing an exception that the method doesn't exist" might be helpful. Not critical though.

Thanks.

I found and fixed another bug, this one around importing the tweets table. @simonw let me know if you'd prefer this broken out into multiple PRs, happy to do that if it makes review/merging easier.

This plugin hook currently returns a string of JavaScript. It could optionally return this instead of a string:

{
  "script": "string of JavaScript goes here",
  "module": true
}

Updated documentation: https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript and https://docs.datasette.io/en/latest/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette

@simonw : Did you have the time to take a look at this ?

I don't think it makes sense to call .enable_fts() on a view does it? When I'm working with views and FTS I tend to write my queries against a FTS table for one of the tables that is used by the view - https://docs.datasette.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view describes how I do that in Datasette for example.

Can you expand on your use-case for FTS and views? I'm ready to be convinced otherwise, but I don't see how it would work right now.

I'm going to change the error message to list the allowed pragmas.

That allow-list was added in #761 but is not currently documented. It's here in the code:

https://github.com/simonw/datasette/blob/8e8fc5cee5c78da8334495c6d6257d5612c40792/datasette/utils/__init__.py#L173-L186

https://docs.datasette.io/en/stable/sql_queries.html?highlight=pragma#named-parameters documentation is out-of-date as well:

Datasette disallows custom SQL containing the string PRAGMA, as SQLite pragma statements can be used to change database settings at runtime. If you need to include the string "pragma" in a query you can do so safely using a named parameter.

Y, thank you to both of you!

Great, really happy I could help! Reverse proxies get tricky.

Should Datasette have subcommands for this? datasette enable-counts data.db and datasette disable-counts data.db and datasette reset-counts data.db commands?

Maybe. The sqlite-utils CLI tool could be used here instead, but that won't be easily available if Datasette was installed as a standalone binary or using brew install datasette or pipx install datasette.

So how would this work?

I think I'm going to automatically use these values if the _counts table exists, unless a ignore_counts_table boolean setting has been set. I won't bother looking to see if the triggers have been created.

Fantastic!

+1

Yep! Fixes it. If I navigate to https://corpora.tika.apache.org/datasette, I get a 404 (database not found: datasette), but if I navigate to https://corpora.tika.apache.org/datasette/file_profiles/, everything WORKS!

Thank you!

https://docs.datasette.io/en/latest/ecosystem.html

It can talk about Dogsheep too.

I'll leave the page there but change it into more of a blurb about the existence of the plugins and tools directories.

I'm going to add a unit test that tries a variety of weird database names.

Yes, that logic is definitely at fault. It looks like it applies urllib.parse.unquote_plus() AFTER it's tried to do the - hash splitting thing, which is why it's failing here:

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184-L198

@henry501 it looks like you spotted a bug in the documentation - I just addressed that, the fix is now live here: https://docs.datasette.io/en/latest/deploying.html#running-datasette-behind-a-proxy

@tballison I think that's the solution! It looks like you need to use this in your config:

ProxyPass /datasette http://127.0.0.1:8001/datasette

Instead of this:

ProxyPass /datasette http://127.0.0.1:8001/

Give that a go and let me know if it fixes it.

I found this issue while troubleshooting the same behavior with an nginx reverse proxy. The solution was to make sure I set:

proxy_pass http://server:8001/baseurl/
instead of just:

proxy_pass http://server:8001
The custom SQL query and header links are now correct.

I requested a D-U-N-S number as a first step in getting a developer certificate: https://developer.apple.com/support/D-U-N-S/

The async_call_with_supported_arguments version should be able to await any of the lazy arguments that are awaitable - can use await_me_maybe for that.

Relevant code: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/__init__.py#L919-L940

In that case maybe there are three new arguments: path, full_path and url.

I'll also add request.full_path for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90

I think I'll call this full_path for consistency with Django.

Django calls this HttpRequest.get_full_path() - for the path plus the querystring.

This parameter will return the URL path, with querystring arguments, to the HTML version of the page - e.g. /github/issue_comments or /github/issue_comments?_sort_desc=created_at

Open questions:

• What should it be called? path could be misleading since it also includes the querystring.
• Should I provide a url or full_url version which includes https://blah.com/...?

In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3:

import pandas
df = pandas.read_json(
  "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array"
)

And:

d3 = require("d3@5")
rows = d3.json(
    "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array"
)

Once again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both pandas.read_json() and d3.json() - but it leaves nowhere to put the extra information like pagination and suchlike!

Even given this I still think the correct default is an object with "rows", "total" and "next_url" keys. I should commit to that and implement it - this thought exercise has been running for far too long.

Issue fixed - https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on displays the correct schemes now.

I can't think of a reason anyone on Cloud Run would ever NOT want the force_https_urls option, but just in case I've made it so if you pass --extra-options --setting force_https_urls off to publish cloudrun your setting will be respected.

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/publish/cloudrun.py#L105-L110

Deploying that change now to test it.

Easiest fix would be for publish cloudrun to set force_https_urls:

datasette publish now used to do this: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/publish/now.py#L59-L63

An optional path argument to https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette which shows the path WITHOUT the .Notebook extension would be useful here.

https://latest-with-plugins.datasette.io/-/asgi-scope

{'asgi': {'spec_version': '2.1', 'version': '3.0'},
 'client': ('169.254.8.129', 54971),
 'headers': [(b'host', b'latest-with-plugins.datasette.io'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko'
              b'/20100101 Firefox/84.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'dnt', b'1'),
             (b'cookie',
              b'_ga_LL6M7BK6D4=GS1.1.1609886546.49.1.1609886923.0; _ga=GA1.1'
              b'.894633707.1607575712'),
             (b'upgrade-insecure-requests', b'1'),
             (b'x-client-data', b'CgSL6ZsV'),
             (b'x-cloud-trace-context',
              b'e776af843c657d2a3da28a73b726e6fe/14187666787557102189;o=1'),
             (b'x-forwarded-for', b'148.64.98.14'),
             (b'x-forwarded-proto', b'https'),
             (b'forwarded', b'for="148.64.98.14";proto=https'),
             (b'accept-encoding', b'gzip, deflate, br'),
             (b'content-length', b'0')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope',
 'root_path': '',
 'scheme': 'http',
 'server': ('169.254.8.130', 8080),
 'type': 'http'}

Note the 'scheme': 'http' but also the (b'x-forwarded-proto', b'https').

I used from datasette.utils import path_with_format in https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py just now.

Moving this to the Datasette repo.

https://latest-with-plugins.datasette.io/fixtures/sortable.json has the bug too - the next_url is http:// when it should be https://.

https://latest-with-plugins.datasette.io/-/settings says "force_https_urls": false

Weird... https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/app.py#L609-L613

    def absolute_url(self, request, path):
        url = urllib.parse.urljoin(request.url, path)
        if url.startswith("http://") and self.setting("force_https_urls"):
            url = "https://" + url[len("http://") :]
        return url

That looks like it should work. Needs more digging.

https://github.com/simonw/datasette-export-notebook/blob/aec398eab4f34791d240d7bc47b6eec575b357be/datasette_export_notebook/__init__.py#L18-L23

Maybe this is a bug in datasette.absolute_url? Perhaps it doesn't take the scheme into account.

With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the stream_rows() method could output [ first, then each row followed by a comma, then ] after the very last row.

Idea: instead of returning a dictionary, register_output_renderer could return an object. The object could have the following properties:

• .extension - the extension to use
• .can_render(...) - says if it can render this
• .can_stream(...) - says if streaming is supported
• async .stream_rows(rows_iterator, send) - method that loops through all rows and uses send to send them to the response in the correct format

I can then deprecate the existing dict return type for 1.0.

Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:

pandas.read_json("https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on", lines=True)

That's really useful, thanks @rcoup

https://latest-with-plugins.datasette.io/fixtures/roadside_attraction_characteristics/1.json-preview returns a 500 error at the moment - a KeyError on 'filtered_table_rows_count'.

See https://docs.datasette.io/en/stable/internals.html

I'm happy with how this has evolved, so I'm closing the issue.

This needs to be done for Datasette 1.0. At the very least I need to ensure it's clear that datasette.utils is not part of the public API unless explicitly marked as such.

Known public APIs might be worth adding type annotations to as well.

It might be neater to move all of the non-public functions into a separate module - datasette.utils.internal perhaps.

Idea: introduce a @documented decorator which marks specific functions as part of the public, documented API. The unit tests can then confirm that anything with that decorator is both documented and tested.

@documented
def escape_css_string(s):
    return _css_re.sub(
        lambda m: "\\" + (f"{ord(m.group()):X}".zfill(6)),
        s.replace("\r\n", "\n"),
    )

Or maybe @public?

We did this for Sno under macOS — it's a PyInstaller binary/setup which uses Packages for packaging.

• Building & Signing
• Packaging & Notarizing
• Github Workflow has the CI side of it

FYI (if you ever get to it) for Windows you need to get a code signing certificate. And if you want automated CI, you'll want to get an "EV CodeSigning for HSM" certificate from GlobalSign, which then lets you put the certificate into Azure Key Vault. Which you can use with azuresigntool to sign your code & installer. (Non-EV certificates are a waste of time, the user still gets big warnings at install time).

I was able to fix this, at least enough to get my archive to import. Not sure if there's more work to be done here or not.

This now works for me, though I'm entirely ensure if it's a just-my-export thing or a wider issue. Also, this doesn't contain any tests. So I'm not sure if there's more work to be done here, or if this is good enough.

Correction: the failure is on lists-member.js (I was thrown by the block variable name, but that's just a coincidence)

Some tips here: https://github.com/tiangolo/fastapi/issues/78

Not sure where exactly to put the actual docs (presumably somewhere in docs/contributing.rst) but I've made a slight change to make it easier to run locally (copying the approach in excalidraw): https://github.com/simonw/datasette/compare/main...benpickles:prettier-docs

I'm seeing the problem on database page. Index page and table page runs quite fast.

• Tables have <10 columns (id, url, title, body_html, date, author, meta (for keeping unstructured json)). I've added index on date columns (using sqlite-utils) in addition to the index present on id columns.
• All tables have FTS enabled on text and varchar columns (title, body_html etc) to speed up searching.
• There are couple of tables related with foreign keys (think a thread in a forum and posts in that thread, related with thread_id)

https://github.com/oleksis/pyinstaller-manylinux looks useful, via https://twitter.com/oleksis/status/1346341987876823040

Bit uncomfortable that it looks like you need to include your Apple ID username and password in the CI configuration to do this. I'll use GitHub Secrets for this but I don't like it - I'll definitely setup a dedicated code signing account that's not my access-to-everything AppleID for this.

https://github.com/search?l=YAML&q=gon+json&type=Code reveals some examples of people using gon in workflows.

These look useful:

• https://github.com/coherence/hub-server/blob/3b7e9c7c5bce9e244b14b854f1f89d66f53a5a39/.github/workflows/release_build.yml
• https://github.com/simoncozens/pilcrow/blob/5abc145e7fb9577086afe47b48fd730cb8195386/.github/workflows/buildapp.yaml

This looks VERY useful: https://github.com/mitchellh/gon - " Sign, notarize, and package macOS CLI tools and applications written in any language. Available as both a CLI and a Go library."

And it installs like this:

brew install mitchellh/gon/gon

Oh wow maybe I need to Notarize it too? https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution

https://github.com/actions/virtual-environments/issues/1820#issuecomment-719549887 looks useful - not sure if those notes are for iOS or macOS though.

Next step is to automate this all!

I think the way to do this is to have a new plugin hook that returns two SQL where clauses: one returning a list of resources that the user should be able to access (the allow-list) and one returning a list of resources they are explicitly forbidden from accessing (the deny-list). Either of these can be blank.

Datasette can then combine those into a full SQL query and use it to answer the question "show me a list of resources that the user is allowed to perform action X on". It can also answer the existing question, "is user X allowed to perform action Y on resource Z"?

I tested it by running a tmate session on the GitHub macOS machines, and it worked!

Mac-1609795972770:tmp runner$ wget 'https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip'
--2021-01-04 21:34:10--  https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream [following]
--2021-01-04 21:34:14--  https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.43.164
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.43.164|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8297283 (7.9M) [application/octet-stream]
Saving to: ‘datasette-0.53-macos-binary.zip’

datasette-0.53-maco 100%[===================>]   7.91M  --.-KB/s    in 0.1s    

2021-01-04 21:34:14 (73.4 MB/s) - ‘datasette-0.53-macos-binary.zip’ saved [8297283/8297283]

Mac-1609795972770:tmp runner$ unzip datasette-0.53-macos-binary.zip 
Archive:  datasette-0.53-macos-binary.zip
   creating: datasette-0.53-macos-binary/
  inflating: datasette-0.53-macos-binary/datasette  
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet along...
  uninstall  Uninstall Python packages (e.g.
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --get /-/versions.json
{"python": {"version": "3.9.1", "full": "3.9.1 (default, Dec 10 2020, 10:36:35) \n[Clang 12.0.0 (clang-1200.0.32.27)]"}, "datasette": {"version": "0.53"}, "asgi": "3.0", "uvicorn": "0.13.3", "sqlite": {"version": "3.34.0", "fts_versions": ["FTS5", "FTS4", "FTS3"], "extensions": {"json1": null}, "compile_options": ["COMPILER=clang-12.0.0", "ENABLE_COLUMN_METADATA", "ENABLE_FTS3", "ENABLE_FTS3_PARENTHESIS", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_GEOPOLY", "ENABLE_JSON1", "ENABLE_PREUPDATE_HOOK", "ENABLE_RTREE", "ENABLE_SESSION", "MAX_VARIABLE_NUMBER=250000", "THREADSAFE=1"]}}
Mac-1609795972770:tmp runner$ 

As an experiment, I put the macOS one in a zip file and attached it to the latest release:

mkdir datasette-0.53-macos-binary
cp dist/datasette datasette-0.53-macos-binary
zip -r datasette-0.53-macos-binary.zip datasette-0.53-macos-binary

It's available here: https://github.com/simonw/datasette/releases/tag/0.53 - download URL is https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip

(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# dist/datasette --get /-/databases.json
[{"name": ":memory:", "path": null, "size": 0, "is_mutable": true, "is_memory": true, "hash": null}]
(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# ls -lah dist/datasette 
-rwxr-xr-x 1 root root 8.9M Jan  4 21:05 dist/datasette

Works on Linux/Ubuntu too, except I had to do export BASE= on a separate line. I also did this:

apt-get install python3 python3-venv
python3 -m venv pyinstaller-venv
source pyinstaller-venv/bin/activate
pip install wheel
pip install datasette pyinstaller

export DATASETTE_BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))')

pyinstaller -F \
    --add-data "$DATASETTE_BASE/templates:datasette/templates" \
    --add-data "$DATASETTE_BASE/static:datasette/static" \
    --hidden-import datasette.publish \
    --hidden-import datasette.publish.heroku \
    --hidden-import datasette.publish.cloudrun \
    --hidden-import datasette.facets \
    --hidden-import datasette.sql_functions \
    --hidden-import datasette.actor_auth_cookie \
    --hidden-import datasette.default_permissions \
    --hidden-import datasette.default_magic_parameters \
    --hidden-import datasette.blob_renderer \
    --hidden-import datasette.default_menu_links \
    --hidden-import uvicorn \
    --hidden-import uvicorn.logging \
    --hidden-import uvicorn.loops \
    --hidden-import uvicorn.loops.auto \
    --hidden-import uvicorn.protocols \
    --hidden-import uvicorn.protocols.http \
    --hidden-import uvicorn.protocols.http.auto \
    --hidden-import uvicorn.protocols.websockets \
    --hidden-import uvicorn.protocols.websockets.auto \
    --hidden-import uvicorn.lifespan \
    --hidden-import uvicorn.lifespan.on \
    $(which datasette)

That BASE= trick seems to work with zsh but not with bash.

Updated pyinstaller recipe - lots of hidden imports needed now:

pip install wheel
pip install datasette pyinstaller

BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))') \
    pyinstaller -F \
        --add-data "$BASE/templates:datasette/templates" \
        --add-data "$BASE/static:datasette/static" \
        --hidden-import datasette.publish \
        --hidden-import datasette.publish.heroku \
        --hidden-import datasette.publish.cloudrun \
        --hidden-import datasette.facets \
        --hidden-import datasette.sql_functions \
        --hidden-import datasette.actor_auth_cookie \
        --hidden-import datasette.default_permissions \
        --hidden-import datasette.default_magic_parameters \
        --hidden-import datasette.blob_renderer \
        --hidden-import datasette.default_menu_links \
        --hidden-import uvicorn \
        --hidden-import uvicorn.logging \
        --hidden-import uvicorn.loops \
        --hidden-import uvicorn.loops.auto \
        --hidden-import uvicorn.protocols \
        --hidden-import uvicorn.protocols.http \
        --hidden-import uvicorn.protocols.http.auto \
        --hidden-import uvicorn.protocols.websockets \
        --hidden-import uvicorn.protocols.websockets.auto \
        --hidden-import uvicorn.lifespan \
        --hidden-import uvicorn.lifespan.on \
        $(which datasette)

(pyinstaller-datasette) pyinstaller-datasette % file dist/datasette
dist/datasette: Mach-O 64-bit executable x86_64
(pyinstaller-datasette) pyinstaller-datasette % ls -lah dist/datasette
-rwxr-xr-x  1 simon  wheel   8.0M Jan  4 12:58 dist/datasette

I'm surprised it's only 8MB!

For reasons I've written about elsewhere, I'm in favor of modules. It has several beneficial effects. One, old browsers just ignore it all together. Two, if you include the same plain script on the page more than once, it will be executed twice, but if you include the same module script on a page twice, it will only execute once. Three, you get a module local namespace, instead of having to use the global window namespace or a function private namespace.

OTOH, if you are going to use an old style script, the code from before isn't ideal, because you wipe out your registry if the script it included more than once. Also you may as well use object methods and splat arguments.

The event based architecture probably makes more sense though. Just make up some event names prefixed with datasette: and listen for them on the root. The only concern with that approach is it can sometimes be tricky to make sure your plugins are run after datasette has run. Maybe

function mycallback(){
  // whatever
}

if (window.datasette) {
  window.datasette.init(mycallback);
} else {
  document.addEventListener('datasette:init', mycallback);
}

I fixed this in #1115 - you can run --load-extension=spatialite now and it will look for the extension in common places.

Sadly publish.datasettes.com was broken by changes to Zeit, and I don't think I'll be bringing it back.

Now that Digital Ocean has App Platform this is less necessary, especially since the documentation covers how to use App Platform here: https://docs.datasette.io/en/stable/deploying.html#deploying-using-buildpacks

I'm more likely to do Lambda than Elastic Beanstalk, especially now the size limit for Lambdas has been increased as part of their support for Docker.

I'm going to close this. Plugins can register their own CLI tools (see https://github.com/simonw/click-app) if they need to.

Is this still a good idea? I don't have any pressing need for it at the moment.

For the csvs-to-sqlite case I'm going with datasette insert instead, see #1160.

I'm not using Sanic any more, but this is still very feasible. If I ever do it I'll write a plugin.

Wontfix, Cloud Run is already implemented and is a better fit for Datasette.

That's pretty elegant: each plugin gets its own namespace and can register new settings.

I renamed --config to --setting and changed it to work like this:

datasette --setting sql_time_limit_ms 1000

Note the lack of colons.

This actually makes colons cleaner to use for plugins - I could support this:

datasette --setting datasette-insert:unsafe 1

Relevant new feature in sqlite-utils: the ability to use triggers to maintain fast counts. This optimization could help a lot here. https://github.com/simonw/sqlite-utils/issues/212

This will be helped by the new package.json added in #1170.

Thank you very much!

I was thinking JavaScript plugins going with server side template extensions custom HTML. Attach my own widgets on there and listen for Datasette events to refresh when user interacts with main UI. Like a map view or table that updates according to selected column. There's certainly other ways to look at this. Perhaps you could list possible hooks or high level design doc on what would be possible with the plugin system?

Re: modules. I would like to see modules supported at least in development. The developer experience is so much better than what JavaScript coding has been in the past. With large parts of NPM at your disposal I’d imagine even less experienced coder can whisk a custom plugin in no time. Proper production build system (like one you get with Pika or Parcel) could package everything up into bundles that older browsers can understand. Though that does come with performance and size penalties alongside the added complexity.