Also tested this manually like so:

bash sqlite-utils create-table strict.db strictint id integer size integer --strict sqlite-utils create-table strict.db notstrictint id integer size integer sqlite-utils install sqlite-utils-shell sqlite-utils shell strict.db Attached to strict.db Type 'exit' to exit. sqlite-utils> insert into strictint (size) values (4); 1 row affected sqlite-utils> insert into strictint (size) values ('four'); An error occurred: cannot store TEXT value in INTEGER column strictint.size sqlite-utils> insert into notstrictint (size) values ('four'); 1 row affected sqlite-utils> commit; Done

I'm unable to replicate this issue. This is with a fresh install of sqlite-utils==3.35.2: ``` (base) ~ python3.12 Python 3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import sqlite_utils db = sqlite_utils.Database(memory=True) db["foo"].insert({"bar": 1})

Thanks for opening an issue - this should help future Google searchers figure out what's going on here.

Another approach here could be to store large integers as TEXT in SQLite (or even as BLOB).

Both storing as REAL and storing as TEXT/BLOB feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.

This looks really great on first glance - design is good, implementation is solid, tests and documentation look great.

Looks like a couple of mypy failures in the tests at the moment:

``` mypy sqlite_utils tests

sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type "type[Table]", variable has type "type[View]") [assignment] tests/test_lookup.py:156: error: Name "test_lookup_new_table" already defined on line 5 [no-redef] Found 2 errors in 2 files (checked 54 source files) Error: Process completed with exit code 1. ```

Updated documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-add-column - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-column

I'll add bytes too - float already works. This makes sense because when you are working with the Python API you use str and float and bytes and int to specify column types.

That keep_blank_values=True is from https://github.com/simonw/datasette/commit/0934844c0b6d124163d0185fb6a41ba5a71433da

Commit message:

request.post_vars() no longer discards empty values

Relevant test:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/tests/test_internals_request.py#L19-L27

It should return a MultiParams:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/init.py#L900-L917

Change needs to be made before 1.0.

python return MultiParams(urllib.parse.parse_qs(body.decode("utf-8"))) Need to remember why I was using keep_blank_values=True there and check that using MultiParams doesn't conflict with that reason.

I found this problem while trying to use WTForms with this pattern:

```python choices = [(col, col) for col in await db.table_columns(table)]

class ConfigForm(Form): template = TextAreaField("Template") api_token = PasswordField("OpenAI API token") columns = MultiCheckboxField('Columns', choices=choices) ```

I'm going to abandon this PR and ship the 3.12 testing change directly to main.

Thanks!

Manually tested. Before:

After:

And a GIF of the fix after applying: - #598

Here's an animated GIF that demonstrates the bug:

Thanks!

The GIS tests now pass in that container too: bash pytest tests/test_gis.py ``` ======================== test session starts ========================= platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0 rootdir: /tmp/sqlite-utils plugins: hypothesis-6.88.1 collected 12 items

tests/test_gis.py ............ [100%]

========================= 12 passed in 0.48s ========================= ```

See details of how I tested this here: - https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126

Short version: having applied this fix, the following command (on simulated aarch64): bash sqlite-utils memory "select spatialite_version()" --load-extension=spatialite Outputs: json [{"spatialite_version()": "5.0.1"}]

Testing this manually on macOS using Docker Desk top like this: bash docker run -it --rm arm64v8/ubuntu /bin/bash Then inside the container: bash uname -m Outputs: aarch64

Then: bash apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y cd /tmp git clone https://github.com/simonw/sqlite-utils cd sqlite-utils python3 -m venv venv source venv/bin/activate pip install -e '.[test]' sqlite-utils memory "select spatialite_version()" --load-extension=spatialite Which output: Traceback (most recent call last): File "/tmp/sqlite-utils/venv/bin/sqlite-utils", line 33, in <module> sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/tmp/sqlite-utils/sqlite_utils/cli.py", line 1959, in memory _load_extensions(db, load_extension) File "/tmp/sqlite-utils/sqlite_utils/cli.py", line 3232, in _load_extensions if ":" in ext: TypeError: argument of type 'NoneType' is not iterable Then I ran this: bash git checkout -b MikeCoats-spatialite-paths-linux-arm main git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm And now: bash sqlite-utils memory "select spatialite_version()" --load-extension=spatialite Outputs: json [{"spatialite_version()": "5.0.1"}]

The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.

Thanks for this!

Datasette is using that now, see: - #1893

I'm landing this despite the cog failures. I'll fix them on main if I have to.

Oh! I think I broke Cog on main and these tests are running against this branch rebased against main.

Weird, the cog check is failing in CI.

Run cog --check docs/*.rst cog --check docs/*.rst shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib Check failed Checking docs/authentication.rst Checking docs/binary_data.rst Checking docs/changelog.rst Checking docs/cli-reference.rst Checking docs/configuration.rst (changed)

Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade

I can't replicate this bug. Closing, but please re-open if it's still happening.

As an aside, the link I promote is https://datasette.io/discord which redirects:

curl -i 'https://datasette.io/discord' HTTP/2 301 location: https://discord.gg/ktd74dm5mw content-type: text/plain x-cloud-trace-context: 8dcfd08d3d1fa44f7ee78568e0f5305e;o=1 date: Thu, 12 Oct 2023 21:54:17 GMT server: Google Frontend content-length: 0 curl -i 'https://discord.gg/ktd74dm5mw' HTTP/2 301 date: Thu, 12 Oct 2023 21:54:28 GMT content-type: text/plain;charset=UTF-8 content-length: 0 location: https://discord.com/invite/ktd74dm5mw strict-transport-security: max-age=31536000; includeSubDomains; preload permissions-policy: interest-cohort=() x-content-type-options: nosniff x-frame-options: DENY x-robots-tag: noindex, nofollow, noarchive, nocache, noimageindex, noodp x-xss-protection: 1; mode=block report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Dzzrf%2FgGkfFxtzSAQ46slMVDLcFjsH9fsvVkzHtgUUiZ891rXAa6LvTRpHK%2BdSMSQ54F57hS9z1mZXXklIbONZW1bfBuFjSK9J4XmjjLjsFUulMXvpjfCLkB6PI%3D"}],"group":"cf-nel","max_age":604800} nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} server: cloudflare cf-ray: 815294ddff282511-SJC

Demo of that logic: $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-1.yaml New datasette.yaml file will be written to datasette.yaml $ touch metadata-new-1.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette.yaml $ touch datasette.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette-new.yaml

Started playing with this plugin idea, now tearing myself away to work on something more important: ```python from datasette import hookimpl import click import pathlib

@hookimpl def register_commands(cli): @cli.group() def upgrade(): """ Apply configuration upgrades to an existing Datasette instance """ pass

@upgrade.command()
@click.argument(
    "metadata", type=click.Path(exists=True)
)
@click.option(
    "new_metadata", "-m", "--new-metadata", help="Path to new metadata.yaml file", type=click.Path(exists=False)
)
@click.option(
    "new_datasette", "-c", "--new-datasette", help="Path to new datasette.yaml file", type=click.Path(exists=False)
)
@click.option(
    "output_dir", "-e", "--output-dir", help="Directory to write new files to", type=click.Path(), default="."
)
def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):
    """
    Upgrade an existing metadata.json/yaml file to the new metadata.yaml and
    datasette.yaml split introduced prior to Datasette 1.0.
    """
    print("Upgrading {} to new metadata.yaml format".format(metadata))
    output_dir = pathlib.Path(output_dir)
    if not new_metadata:
        # Pick a filename for the new metadata.yaml file that does not yet exist
        new_metadata = pick_filename("metadata", output_dir)
    if not new_datasette:
        new_datasette = pick_filename("datasette", output_dir)
    print("New metadata.yaml file will be written to {}".format(new_metadata))
    print("New datasette.yaml file will be written to {}".format(new_datasette))

def pick_filename(base, output_dir): options = ["{}.yaml".format(base), "{}-new.yaml".format(base)] i = 0 while True: option = options.pop(0) option_path = output_dir / option if not option_path.exists(): return option_path # If we ran out if not options: i += 1 options = ["{}-new-{}.yaml".format(base, i)] ```

I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.

If people want to keep their comments I'm happy to leave them to copy those over by hand.

Related idea: how about a datasette-upgrade plugin which adds a datasette upgrade command that can be used to automate this process?

Maybe something like this: bash datasette install datasette-upgrade datasette upgrade metadata-to-config metadata.json This would output two new files: metadata.yaml and datasette.yaml. If files with those names existed already in the current directory they would be called metadata-new.yaml and datasette-new.yaml.

The command would tell you what it did: ``` Your metadata.json file has been rewritten as two files:

metadata-new.yaml
datasette.yaml

Start Datasette like this to try them out:

datasette -m metadata-new.yaml -c datasette.yaml

`` The command isdatasette upgrade metadata-to-configbecausemetadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.

Some options for where this could go: - Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides. - On a new page? We could have a "upgrade instructions" page in the documentation. - At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page

I'm leaning towards the third option at the moment.

But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.

I'm going to land this and open a new issue for the upgrade instructions.

It would be nice if we could catch that and turn that into a less intimidating Click exception too.

This looks good and works well. The error from this currently looks like:

datasette -m metadata.json -p 8844 Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette", line 33, in <module> sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')()) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 98, in wrapped return fn(*args, **kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 546, in serve metadata_data = fail_if_plugins_in_metadata(parse_metadata(metadata.read())) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 1282, in fail_if_plugins_in_metadata raise Exception( Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details. With wrapping:

Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.

I think we should link directly to documentation that tells people how to perform this upgrade.

https://lite.datasette.io/?install=datasette-packages#/-/packages confirms that Datasette Lite still works (click-default-group-wheel was originally built to allow it to run) and that it's now using these packages: "click-default-group": "1.2.4", "datasette": "0.64.5", Full list: json { "aiofiles": "23.2.1", "anyio": "3.7.1", "asgi-csrf": "0.9", "asgiref": "3.7.2", "certifi": "2022.12.7", "click": "8.1.3", "click-default-group": "1.2.4", "datasette": "0.64.5", "datasette-packages": "0.2", "h11": "0.12.0", "httpcore": "0.15.0", "httpx": "0.23.0", "hupper": "1.12", "idna": "3.4", "itsdangerous": "2.1.2", "janus": "1.0.0", "Jinja2": "3.1.2", "MarkupSafe": "2.1.2", "mergedeep": "1.3.4", "micropip": "0.3.0", "packaging": "23.0", "Pint": "0.22", "pip": "23.2.1", "pluggy": "1.0.0", "pyparsing": "3.0.9", "python-multipart": "0.0.6", "PyYAML": "6.0", "rfc3986": "1.5.0", "setuptools": "67.6.1", "sniffio": "1.3.0", "typing-extensions": "4.5.0", "uvicorn": "0.23.2" }

Confirmed - I ran this in a fresh virtual environment: bash pip install --no-cache datasette And now: bash pip freeze | grep click click==8.1.7 click-default-group==1.2.4 bash datasette --version datasette, version 0.64.5

Replicated this myself: bash cd /tmp mkdir dddd cd dddd pipenv shell pip install datasette pip freeze | grep click click==8.1.7 click-default-group==1.2.4 click-default-group-wheel==1.2.3 Yeah this is bad, I'll ship a 0.64.5 release.

Datasette Cloud really wants this.

Worth noting that it already sets --cors automatically without you needing to specify it:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/init.py#L374-L374

I wonder if that's actually surprising behaviour that we should change before 1.0.

Maybe I should add --cors and --crossdb to datasette publish cloudrun as well?

Found more when I searched for YAML.

Here's the most interesting: https://github.com/labordata/warehouse/blob/0029a72fc1ceae9091932da6566f891167179012/.github/workflows/build.yml#L59

--extra-options="--crossdb --setting sql_time_limit_ms 100000 --cors --setting facet_time_limit_ms 500 --setting allow_facet off --setting trace_debug 1"

Uses both --cors and --crossdb.

https://github.com/search?q=datasette+publish+extra-options+language%3AShell&type=code&l=Shell shows 17 matches, I'll copy in illustrative examples here:

--extra-options="--setting sql_time_limit_ms 5000" --extra-options="--config default_cache_ttl:3600 --config hash_urls:1" --extra-options "--setting sql_time_limit_ms 3500 --setting default_page_size 20 --setting trace_debug 1" --extra-options="--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000" --extra-options="--setting sql_time_limit_ms 5000" --extra-options "--setting suggest_facets off --setting allow_download on --setting truncate_cells_html 0 --setting max_csv_mb 0 --setting sql_time_limit_ms 2000"

I'd really like to remove --extra-options. I think the new design makes that completely obsolete?

Maybe it doesn't. You still need --extra-options for the --crossdb option for example.

Here's the full help for Cloud Run at the moment: bash datasette publish cloudrun --help ``` Usage: datasette publish cloudrun [OPTIONS] [FILES]...

Publish databases to Datasette running on Cloud Run

Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve --branch TEXT Install datasette from a GitHub branch e.g. main --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --install TEXT Additional packages (e.g. plugins) to install --plugin-secret <TEXT TEXT TEXT>... Secrets to pass to plugins, e.g. --plugin- secret datasette-auth-github client_id xxx --version-note TEXT Additional note to show on /-/versions --secret TEXT Secret used for signing secure values, such as signed cookies --title TEXT Title for metadata --license TEXT License label for metadata --license_url TEXT License URL for metadata --source TEXT Source label for metadata --source_url TEXT Source URL for metadata --about TEXT About label for metadata --about_url TEXT About URL for metadata -n, --name TEXT Application name to use when building --service TEXT Cloud Run service to deploy (or over-write) --spatialite Enable SpatialLite extension --show-files Output the generated Dockerfile and metadata.json --memory TEXT Memory to allocate in Cloud Run, e.g. 1Gi --cpu [1|2|4] Number of vCPUs to allocate in Cloud Run --timeout INTEGER Build timeout in seconds --apt-get-install TEXT Additional packages to apt-get install --max-instances INTEGER Maximum Cloud Run instances --min-instances INTEGER Minimum Cloud Run instances --help Show this message and exit. ```

I think the actual design of this is pretty simple. Current help starts like this:

``` Usage: datasette publish cloudrun [OPTIONS] [FILES]...

Publish databases to Datasette running on Cloud Run

Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve `` The-sand-c` short options are not being used.

So I think -c/--config can point to a JSON or YAML datasette.yaml file, and -s/--setting key value can mirror the new -s/--setting option in datasette serve itself (a shortcut for populating the config file directly from the CLI).

Here's the relevant help section from datasette serve: -m, --metadata FILENAME Path to JSON/YAML file containing license/source metadata -c, --config FILENAME Path to JSON/YAML Datasette configuration file -s, --setting SETTING... nested.key, value setting to use in Datasette configuration

1.0a7 is out with this fix as well now: https://docs.datasette.io/en/1.0a7/changelog.html#a7-2023-09-21

Tests all pass now.

That worked

https://github.com/simonw/datasette/commit/e4f868801a6633400045f59584cfe650961c3fa6 is the latest commit right now and https://latest.datasette.io/-/versions shows that as the deployed version.

Tested that locally with Python 3.9 from pyenv and it worked.

Still fails in Python 3.9: https://github.com/simonw/datasette/actions/runs/6266752548/job/17018363302 plugin_info["name"] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' Test failed: datasette-json-html should not have been loaded

The test fails as expected now. Closing this issue, will solve the remaining problems in: - #2057

https://github.com/simonw/datasette/actions/runs/6267146158/job/17019594849 failed on 3.9 this time.

plugin_info["name"] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' Test failed: datasette-json-html should not have been loaded

The @add_common_publish_arguments_and_options decorator described here is bad. If I update it to support a new config option all plugins that use it will break.

https://github.com/simonw/datasette/blob/f130c7c0a88e50cea4121ea18d1f6db2431b6fab/docs/plugin_hooks.rst#L347-L355

I want to deprecate it and switch to a different, better design to address the same problem.

As soon as datasette publish cloudrun has this I can re-enable this bit of the demo deploy:

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

Which should fix this broken demo from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://todomvc.datasette.io/

I'm going to disable this bit of the deploy for the moment, which will break the demo linked to from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

... which raises the challenge that datasette publish doesn't yet know what to do with a config file!

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L114-L122

Correct usage is now: bash python tests/fixtures.py fixtures.db fixtures-config.json fixtures-metadata.json \ plugins --extra-db-filename extra_database.db Test tables written to fixtures.db - metadata written to fixtures-metadata.json - config written to fixtures-config.json Wrote plugin: plugins/register_output_renderer.py Wrote plugin: plugins/view_name.py Wrote plugin: plugins/my_plugin.py Wrote plugin: plugins/messages_output_renderer.py Wrote plugin: plugins/sleep_sql_function.py Wrote plugin: plugins/my_plugin_2.py Test tables written to extra_database.db

New error: "Error: Metadata should end with .json"

https://github.com/simonw/datasette/actions/runs/6266720924/job/17018265851

That does seem to fix the problem!

Hunch: https://pypi.org/project/importlib-metadata/ may help here.

I think I see the problem - it's from here: https://github.com/simonw/datasette/commit/b2ec8717c3619260a1b535eea20e618bf95aa30b#diff-5dbc88d6e5c3615caf10e32a9d6fc6ff683f5b5814948928cb84c3ab91c038b6L770

The config and metadata Click options are the wrong way round:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L785-L786

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L801

Maybe plugins/ does not exist? It should have been created by this line:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L41-L42

This looks to be the step that is failing:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L50-L60

We're planning a breaking change in 1.0a7: - #2191

Since that's a breaking change I'm going to ship 1.0a7 right now with this fix, then ship that breaking change as 1.0a8 instead.

Release 0.64.4: https://docs.datasette.io/en/stable/changelog.html#v0-64-4

No that's not it actually, it's something else.

Got to this point: bash DATASETTE_LOAD_PLUGINS=datasette-init python -i $(which datasette) plugins That fails and drops me into a debugger:

File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 186, in plugins app = Datasette([], plugins_dir=plugins_dir) File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 405, in __init__ for plugin in get_plugins() File "/Users/simon/Dropbox/Development/datasette/datasette/plugins.py", line 89, in get_plugins plugin_info["name"] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name'

The problem is here: 86 distinfo = plugin_to_distinfo.get(plugin) 87 if distinfo is None: 88 breakpoint() 89 -> assert False 90 if distinfo.name is None: 91 breakpoint() 92 assert False 93 if distinfo: 94 plugin_info["version"] = distinfo.version (Pdb) distinfo (Pdb) plugin <module 'datasette.sql_functions' from '/Users/simon/Dropbox/Development/datasette/datasette/sql_functions.py'> That plugin_to_distinfo is missing some stuff.

So the problem is the get_plugins() function returning plugins with None for their name:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/plugins.py#L61-L91

Relevant code: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/app.py#L1127-L1146

In the debugger: ```

import pdb pdb.pm() /Users/simon/Dropbox/Development/datasette/datasette/app.py(1136)_plugins() -> ps.sort(key=lambda p: p["name"]) (Pdb) ps [{'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['prepare_connection', 'render_cell'], 'version': '1.0.1'}, {'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['startup'], 'version': '0.2'}] ```

In my Python 3.8 environment I ran: bash datasette install datasette-init datasette-json-html And now datasette plugins produces this error: File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 192, in plugins click.echo(json.dumps(app._plugins(all=all), indent=4)) File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1136, in _plugins ps.sort(key=lambda p: p["name"]) TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'

Yes, the new script seems to work. On Python 3.11:

tests/test-datasette-load-plugins.sh echo $? 0 On Python 3.8: tests/test-datasette-load-plugins.sh Test failed: datasette-json-html not found echo $? 1

GPT-4 says:

In the script, you're using a subshell ( ... ) to group commands. If you exit 1 within the subshell, it will only exit the subshell and not the main script. This is why GitHub Actions does not see it as a failure.

It suggested doing this instead:

```bash

!/bin/bash

PLUGINS=$(datasette plugins) if ! echo "$PLUGINS" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q true; then echo "Test failed: datasette-json-html not found" exit 1 fi

PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins) if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q false; then echo "Test failed: datasette-json-html should not have been loaded" exit 1 fi

if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-init")' | grep -q true; then echo "Test failed: datasette-init should have been loaded" exit 1 fi

PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins) if ! echo "$PLUGINS3" | grep -q '[]'; then echo "Test failed: datasette plugins should have returned []" exit 1 fi ```

The script:

https://github.com/simonw/datasette/blob/b0d0a0e5de8bb5b9b6c253e8af451a532266bcf1/tests/test-datasette-load-plugins.sh#L1-L29

I'm not sure why those exit 1 lines did not cause a CI failure.

Here's the failure in CI, which did not cause the workflow to fail even though it should have:

That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the "Test DATASETTE_LOAD_PLUGINS" test shows errors but did not fail the CI run.

I'll imitate certbot:

https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot-ci/setup.py#L9

python 'importlib_resources>=1.3.1; python_version < "3.9"', Looks like 1.3 is the minimum version needed for compatibility with the 3.9 standard library, according to https://github.com/python/importlib_resources/blob/main/README.rst#compatibility

https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot/certbot/_internal/constants.py#L13C29-L16

python if sys.version_info >= (3, 9): # pragma: no cover import importlib.resources as importlib_resources else: # pragma: no cover import importlib_resources

I think I can fix this using https://importlib-resources.readthedocs.io/en/latest/using.html - maybe as a dependency only installed if the Python version is less than 3.9.

Confirmed: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files

importlib.resources.files(package) [...] New in version 3.9.

This broke in Python 3.8:

if plugin.__name__ not in DEFAULT_PLUGINS: try: if (importlib.resources.files(plugin.__name__) / "static").is_dir(): E AttributeError: module 'importlib.resources' has no attribute 'files'

I'm going to release this in 1.0a7, and I'll backport it to a 0.64.4 release too.

The {"units": {"distance": "m", "frequency": "Hz"}} bit is for the units feature which I've half-disabled already and would like to remove before 1.0, though ideally turning that functionality into a plugin instead (if I can figure out how to do that).

This is one of the most interesting illustrative examples in the new code:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L301-L349

Interesting to note that it now has canned queries in it, which include this bit:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L341-L342

It looks like metadata, but in this case it's configuration. That blur between metadata and configuration at the canned query level still feels a little bit odd to me, but I still think we're going in the right direction with it.

Also interesting, from that same file:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L351-L399

There are a few things in that metadata block that are arguably configuration, not metadata - for example:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L360

I think extra_css_urls is definitely configuration, not metadata.

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L369-L395

Most of that stuff is arguably configuration too, with the exception of the roadside_attractions.columns bit which is metadata about those columns.

I tested this locally for permissions like this. datasette.yml:

yaml databases: content: allow: id: root Started Datasette like this: bash datasette --root content.db pottery2.db -c datasette.yml As root I could see this (note the padlock):

http://127.0.0.1:8001/-/metadata returned {} showing that the permissions must have come from the config file instead.

Added a note to that example in the documentation: https://github.com/simonw/datasette/blob/4e6a34179eaedec44c1263275d7592fd83d7e2ac/docs/internals.rst?plain=1#L1320

The one other thing affected by this change is this documentation, which suggests a not-actually-safe pattern: https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/docs/internals.rst#L1292-L1321

My current hunch is that SQLite gets unhappy if multiple threads access the same underlying C object - which sometimes happens with in-memory connections and Datasette presumably because they are faster than file-backed databases.

I'm going to remove the asyncio.gather() code from the table view. I'll ship a 0.x release with that fix too.

Using SQLite In Multi-Threaded Applications

That indicates that there's a SQLite option for "Serialized" mode where it's safe to access anything SQLite provides from multiple threads, but as far as I can tell Python doesn't give you an option to turn that mode on or off for a connection - you can read sqlite3.threadsafety to see if that mode was compiled in or not, but not actually change it.

On my Mac sqlite3.threadsafety returns 1 which means https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety "Multi-thread: In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads." - it would need to return 3 for that serialized mode.

Looking again at this code:

https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/datasette/database.py#L87-L117

check_same_thread=False really stands out here.

Python docs at https://docs.python.org/3/library/sqlite3.html

check_same_thread (bool) -- If True (default), ProgrammingError will be raised if the database connection is used by a thread other than the one that created it. If False, the connection may be accessed in multiple threads; write operations may need to be serialized by the user to avoid data corruption. See threadsafety for more information.

I think I'm playing with fire by allowing multiple threads to access the same connection without doing my own serialization of those requests.

I do do that using the write connection - and in this particular case the bug isn't coming from write queries, it's coming from read queries - but perhaps SQLite has issues with threading for reads, too.

Now that I've confirmed that parallel query execution of the kind introduced in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 can cause hangs (presumably some kind of locking issue) against in-memory databases, some options:

1. Disable parallel execution entirely and rip out related code.
2. Disable parallel execution entirely by leaving that code but having it always behave as if _noparallel=1
3. Continue digging and try and find some way to avoid this problem

The parallel execution work is something I was playing with last year in the hope of speeding up Datasette pages like the table page which need to execute a bunch of queries - one for each facet, plus one for each column to see if it should be suggested as a facet.

I wrote about this at the time here: https://simonwillison.net/2022/May/6/weeknotes/

My hope was that despite Python's GIL this optimization would still help, because the SQLite C module releases the GIL once it gets to SQLite.

But... that didn't hold up. It looked like enough work was happening in Python land with the GIL that the optimization didn't improve things.

Running the nogil fork of Python DID improve things though! I left the code in partly on the hope that the nogil fork would be accepted into Python core.

... which it now has! But it will still be a year or two before it fully lands: https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

So I'm not particularly concerned about dropping the parallel execution. If I do drop it though do I leave the potentially complex code in that relates to it?

I also confirmed that http://127.0.0.1:8064/airtable_refs/airtable_refs?_noparallel=1 does not trigger the bug but http://127.0.0.1:8064/airtable_refs/airtable_refs does.

OK it looks like it found it!

``` 942411ef946e9a34a2094944d3423cddad27efd3 is the first bad commit commit

Author: Simon Willison swillison@gmail.com Date: Tue Apr 26 15:48:56 2022 -0700

Execute some TableView queries in parallel

Use ?_noparallel=1 to opt out (undocumented, useful for benchmark comparisons)

Refs #1723, #1715

datasette/views/table.py | 93 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 67 insertions(+), 26 deletions(-) bisect found first bad commit ``` https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 does look like the cause of this problem.

Turned out I wasn't running the datasette from the current directory, so it was not testing what I intended.

FIxed that with pip install -e . in the datasette/ directory.

Now I'm seeing some passes, which look like this: running '../airtable-export/testit.sh' INFO: Started server process [77810] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls INFO: 127.0.0.1:59439 - "GET /airtable_refs/airtable_refs HTTP/1.1" 200 OK INFO: 127.0.0.1:59440 - "GET /airtable_refs/airtable_refs HTTP/1.1" 200 OK INFO: 127.0.0.1:59441 - "GET /airtable_refs/airtable_refs HTTP/1.1" 200 OK All curl succeeded Killing datasette server with PID 77810 ../airtable-export/testit.sh: line 54: 77810 Killed: 9 datasette pottery2.db -p $port All three curls succeeded. Bisecting: 4 revisions left to test after this (roughly 2 steps) [7463b051cf8d7f856df5eba9f7aa944183ebabe5] Cosmetic tweaks after blacken-docs, refs #1718 running '../airtable-export/testit.sh' INFO: Started server process [77826] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls

I'm now trying this test script: ```bash

!/bin/bash

port=8064

Start datasette server in the background and get its PID

datasette pottery2.db -p $port & server_pid=$!

Wait for a moment to ensure the server has time to start up

sleep 2

Initialize counters and parameters

retry_count=0 max_retries=3 success_count=0 path="/airtable_refs/airtable_refs"

Function to run curl with a timeout

function test_curl { # Run the curl command with a timeout of 3 seconds timeout 3s curl -s "http://localhost:${port}${path}" > /dev/null if [ $? -eq 0 ]; then # Curl was successful ((success_count++)) fi }

Try three parallel curl requests

while [[ $retry_count -lt $max_retries ]]; do # Reset the success counter success_count=0

# Run the curls in parallel
echo "  Running curls"
test_curl
test_curl
test_curl #  & test_curl & test_curl &

# Wait for all curls to finish
#wait

# Check the success count
if [[ $success_count -eq 3 ]]; then
    # All curls succeeded, break out of the loop
    echo "  All curl succeeded"
    break
fi

((retry_count++))

done

Kill the datasette server

echo "Killing datasette server with PID $server_pid" kill -9 $server_pid sleep 2

Print result

if [[ $success_count -eq 3 ]]; then echo "All three curls succeeded." exit 0 else echo "Error: Not all curls succeeded after $retry_count attempts." exit 1 fi I run it like this:bash git bisect reset git bisect start git bisect good 0.59.4 git bisect bad 1.0a6 git bisect run ../airtable-export/testit.sh ``` But... it's not having the desired result, I think because the bug is intermittent so each time I run it the bisect spits out a different commit as the one that is to blame.

I knocked it down to 1 retry just to see what happened.

I tried it with a path of / and everything passed - so it's definitely the path of /airtable_refs/airtable_refs (an in-memory database created by an experimental branch of https://github.com/simonw/airtable-export) that triggers the problem.

Output while it is running looks like this: running '../airtable-export/testit.sh' INFO: Started server process [75649] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls Running curls Running curls Killing datasette server with PID 75649 ../airtable-export/testit.sh: line 54: 75649 Killed: 9 datasette pottery2.db -p $port Error: Not all curls succeeded after 3 attempts. Bisecting: 155 revisions left to test after this (roughly 7 steps) [247e460e08bf823142f7b84058fe44e43626787f] Update beautifulsoup4 requirement (#1703) running '../airtable-export/testit.sh' INFO: Started server process [75722] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls Running curls Running curls Killing datasette server with PID 75722 ../airtable-export/testit.sh: line 54: 75722 Killed: 9 datasette pottery2.db -p $port Error: Not all curls succeeded after 3 attempts. Bisecting: 77 revisions left to test after this (roughly 6 steps) [3ef47a0896c7e63404a34e465b7160c80eaa571d] Link rel=alternate header for tables and rows running '../airtable-export/testit.sh' INFO: Started server process [75818] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls

This worked, including on macOS even though GPT-4 thought timeout would not work there: https://chat.openai.com/share/cc4628e9-5240-4f35-b640-16a9c178b315 ```bash

!/bin/bash

Run the command with a timeout of 5 seconds

timeout 5s datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs

Check the exit code from timeout

if [ $? -eq 124 ]; then echo "Error: Command timed out after 5 seconds." exit 1 fi ```

OK, I can trigger the bug like this:

bash datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs Can I write a bash script that fails (and terminates the process) if it takes longer than X seconds?

I switched that particular implementation to using an on-disk database instead of an in-memory database and could no longer recreate the bug.