Worth noting that it already sets --cors automatically without you needing to specify it:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/init.py#L374-L374

I wonder if that's actually surprising behaviour that we should change before 1.0.

Maybe I should add --cors and --crossdb to datasette publish cloudrun as well?

Found more when I searched for YAML.

Here's the most interesting: https://github.com/labordata/warehouse/blob/0029a72fc1ceae9091932da6566f891167179012/.github/workflows/build.yml#L59

--extra-options="--crossdb --setting sql_time_limit_ms 100000 --cors --setting facet_time_limit_ms 500 --setting allow_facet off --setting trace_debug 1"

Uses both --cors and --crossdb.

https://github.com/search?q=datasette+publish+extra-options+language%3AShell&type=code&l=Shell shows 17 matches, I'll copy in illustrative examples here:

--extra-options="--setting sql_time_limit_ms 5000" --extra-options="--config default_cache_ttl:3600 --config hash_urls:1" --extra-options "--setting sql_time_limit_ms 3500 --setting default_page_size 20 --setting trace_debug 1" --extra-options="--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000" --extra-options="--setting sql_time_limit_ms 5000" --extra-options "--setting suggest_facets off --setting allow_download on --setting truncate_cells_html 0 --setting max_csv_mb 0 --setting sql_time_limit_ms 2000"

I'd really like to remove --extra-options. I think the new design makes that completely obsolete?

Maybe it doesn't. You still need --extra-options for the --crossdb option for example.

Here's the full help for Cloud Run at the moment: bash datasette publish cloudrun --help ``` Usage: datasette publish cloudrun [OPTIONS] [FILES]...

Publish databases to Datasette running on Cloud Run

Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve --branch TEXT Install datasette from a GitHub branch e.g. main --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --install TEXT Additional packages (e.g. plugins) to install --plugin-secret <TEXT TEXT TEXT>... Secrets to pass to plugins, e.g. --plugin- secret datasette-auth-github client_id xxx --version-note TEXT Additional note to show on /-/versions --secret TEXT Secret used for signing secure values, such as signed cookies --title TEXT Title for metadata --license TEXT License label for metadata --license_url TEXT License URL for metadata --source TEXT Source label for metadata --source_url TEXT Source URL for metadata --about TEXT About label for metadata --about_url TEXT About URL for metadata -n, --name TEXT Application name to use when building --service TEXT Cloud Run service to deploy (or over-write) --spatialite Enable SpatialLite extension --show-files Output the generated Dockerfile and metadata.json --memory TEXT Memory to allocate in Cloud Run, e.g. 1Gi --cpu [1|2|4] Number of vCPUs to allocate in Cloud Run --timeout INTEGER Build timeout in seconds --apt-get-install TEXT Additional packages to apt-get install --max-instances INTEGER Maximum Cloud Run instances --min-instances INTEGER Minimum Cloud Run instances --help Show this message and exit. ```

I think the actual design of this is pretty simple. Current help starts like this:

``` Usage: datasette publish cloudrun [OPTIONS] [FILES]...

Publish databases to Datasette running on Cloud Run

Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve `` The-sand-c` short options are not being used.

So I think -c/--config can point to a JSON or YAML datasette.yaml file, and -s/--setting key value can mirror the new -s/--setting option in datasette serve itself (a shortcut for populating the config file directly from the CLI).

Here's the relevant help section from datasette serve: -m, --metadata FILENAME Path to JSON/YAML file containing license/source metadata -c, --config FILENAME Path to JSON/YAML Datasette configuration file -s, --setting SETTING... nested.key, value setting to use in Datasette configuration

1.0a7 is out with this fix as well now: https://docs.datasette.io/en/1.0a7/changelog.html#a7-2023-09-21

Tests all pass now.

That worked

https://github.com/simonw/datasette/commit/e4f868801a6633400045f59584cfe650961c3fa6 is the latest commit right now and https://latest.datasette.io/-/versions shows that as the deployed version.

Tested that locally with Python 3.9 from pyenv and it worked.

Still fails in Python 3.9: https://github.com/simonw/datasette/actions/runs/6266752548/job/17018363302 plugin_info["name"] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' Test failed: datasette-json-html should not have been loaded

The test fails as expected now. Closing this issue, will solve the remaining problems in: - #2057

https://github.com/simonw/datasette/actions/runs/6267146158/job/17019594849 failed on 3.9 this time.

plugin_info["name"] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' Test failed: datasette-json-html should not have been loaded

The @add_common_publish_arguments_and_options decorator described here is bad. If I update it to support a new config option all plugins that use it will break.

https://github.com/simonw/datasette/blob/f130c7c0a88e50cea4121ea18d1f6db2431b6fab/docs/plugin_hooks.rst#L347-L355

I want to deprecate it and switch to a different, better design to address the same problem.

As soon as datasette publish cloudrun has this I can re-enable this bit of the demo deploy:

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

Which should fix this broken demo from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://todomvc.datasette.io/

I'm going to disable this bit of the deploy for the moment, which will break the demo linked to from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

... which raises the challenge that datasette publish doesn't yet know what to do with a config file!

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L114-L122

Correct usage is now: bash python tests/fixtures.py fixtures.db fixtures-config.json fixtures-metadata.json \ plugins --extra-db-filename extra_database.db Test tables written to fixtures.db - metadata written to fixtures-metadata.json - config written to fixtures-config.json Wrote plugin: plugins/register_output_renderer.py Wrote plugin: plugins/view_name.py Wrote plugin: plugins/my_plugin.py Wrote plugin: plugins/messages_output_renderer.py Wrote plugin: plugins/sleep_sql_function.py Wrote plugin: plugins/my_plugin_2.py Test tables written to extra_database.db

New error: "Error: Metadata should end with .json"

https://github.com/simonw/datasette/actions/runs/6266720924/job/17018265851

That does seem to fix the problem!

Hunch: https://pypi.org/project/importlib-metadata/ may help here.

I think I see the problem - it's from here: https://github.com/simonw/datasette/commit/b2ec8717c3619260a1b535eea20e618bf95aa30b#diff-5dbc88d6e5c3615caf10e32a9d6fc6ff683f5b5814948928cb84c3ab91c038b6L770

The config and metadata Click options are the wrong way round:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L785-L786

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L801

Maybe plugins/ does not exist? It should have been created by this line:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L41-L42

This looks to be the step that is failing:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L50-L60

We're planning a breaking change in 1.0a7: - #2191

Since that's a breaking change I'm going to ship 1.0a7 right now with this fix, then ship that breaking change as 1.0a8 instead.

Release 0.64.4: https://docs.datasette.io/en/stable/changelog.html#v0-64-4

No that's not it actually, it's something else.

Got to this point: bash DATASETTE_LOAD_PLUGINS=datasette-init python -i $(which datasette) plugins That fails and drops me into a debugger:

File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 186, in plugins app = Datasette([], plugins_dir=plugins_dir) File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 405, in __init__ for plugin in get_plugins() File "/Users/simon/Dropbox/Development/datasette/datasette/plugins.py", line 89, in get_plugins plugin_info["name"] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name'

The problem is here: 86 distinfo = plugin_to_distinfo.get(plugin) 87 if distinfo is None: 88 breakpoint() 89 -> assert False 90 if distinfo.name is None: 91 breakpoint() 92 assert False 93 if distinfo: 94 plugin_info["version"] = distinfo.version (Pdb) distinfo (Pdb) plugin <module 'datasette.sql_functions' from '/Users/simon/Dropbox/Development/datasette/datasette/sql_functions.py'> That plugin_to_distinfo is missing some stuff.

So the problem is the get_plugins() function returning plugins with None for their name:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/plugins.py#L61-L91

Relevant code: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/app.py#L1127-L1146

In the debugger: ```

import pdb pdb.pm() /Users/simon/Dropbox/Development/datasette/datasette/app.py(1136)_plugins() -> ps.sort(key=lambda p: p["name"]) (Pdb) ps [{'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['prepare_connection', 'render_cell'], 'version': '1.0.1'}, {'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['startup'], 'version': '0.2'}] ```

In my Python 3.8 environment I ran: bash datasette install datasette-init datasette-json-html And now datasette plugins produces this error: File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 192, in plugins click.echo(json.dumps(app._plugins(all=all), indent=4)) File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1136, in _plugins ps.sort(key=lambda p: p["name"]) TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'

Yes, the new script seems to work. On Python 3.11:

tests/test-datasette-load-plugins.sh echo $? 0 On Python 3.8: tests/test-datasette-load-plugins.sh Test failed: datasette-json-html not found echo $? 1

GPT-4 says:

In the script, you're using a subshell ( ... ) to group commands. If you exit 1 within the subshell, it will only exit the subshell and not the main script. This is why GitHub Actions does not see it as a failure.

It suggested doing this instead:

```bash

!/bin/bash

PLUGINS=$(datasette plugins) if ! echo "$PLUGINS" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q true; then echo "Test failed: datasette-json-html not found" exit 1 fi

PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins) if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q false; then echo "Test failed: datasette-json-html should not have been loaded" exit 1 fi

if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-init")' | grep -q true; then echo "Test failed: datasette-init should have been loaded" exit 1 fi

PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins) if ! echo "$PLUGINS3" | grep -q '[]'; then echo "Test failed: datasette plugins should have returned []" exit 1 fi ```

The script:

https://github.com/simonw/datasette/blob/b0d0a0e5de8bb5b9b6c253e8af451a532266bcf1/tests/test-datasette-load-plugins.sh#L1-L29

I'm not sure why those exit 1 lines did not cause a CI failure.

Here's the failure in CI, which did not cause the workflow to fail even though it should have:

That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the "Test DATASETTE_LOAD_PLUGINS" test shows errors but did not fail the CI run.

I'll imitate certbot:

https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot-ci/setup.py#L9

python 'importlib_resources>=1.3.1; python_version < "3.9"', Looks like 1.3 is the minimum version needed for compatibility with the 3.9 standard library, according to https://github.com/python/importlib_resources/blob/main/README.rst#compatibility

https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot/certbot/_internal/constants.py#L13C29-L16

python if sys.version_info >= (3, 9): # pragma: no cover import importlib.resources as importlib_resources else: # pragma: no cover import importlib_resources

I think I can fix this using https://importlib-resources.readthedocs.io/en/latest/using.html - maybe as a dependency only installed if the Python version is less than 3.9.

Confirmed: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files

importlib.resources.files(package) [...] New in version 3.9.

This broke in Python 3.8:

if plugin.__name__ not in DEFAULT_PLUGINS: try: if (importlib.resources.files(plugin.__name__) / "static").is_dir(): E AttributeError: module 'importlib.resources' has no attribute 'files'

I'm going to release this in 1.0a7, and I'll backport it to a 0.64.4 release too.

TODO: add similar checks for permissions/allow/canned queries