I think this actually needs to include a whole section of the documentation about the road to 1.0 - what to expect (planned breaking changes) etc. I can add that to the https://docs.datasette.io/en/stable/contributing.html page perhaps - or even create a Roadmap page.

New methods are documented here: https://docs.datasette.io/en/1.0-dev/internals.html#resolve-database-request

When I do this it's important to update the documentation for resolve_database() and the like:

https://github.com/simonw/datasette/blob/ee64130fa8a5ff4a24791916c696e10cf2375102/docs/internals.rst#L594

Found myself needing an await db.view_exists() method for this, similar to the existing await db.table_exists() one.

Open question: should resolve_table() know how to identify named canned queries too?

I think not, at least for the moment. Feels a bit too specialist to expose in a documented API.

Incomplete implementation of this view: ```python class RowUpdateView(BaseView): name = "row-update"

def __init__(self, datasette):
    self.ds = datasette

async def post(self, request):
    database_route = tilde_decode(request.url_vars["database"])
    table = tilde_decode(request.url_vars["table"])
    try:
        db = self.ds.get_database(route=database_route)
    except KeyError:
        return _error(["Database not found: {}".format(database_route)], 404)

    database_name = db.name
    if not await db.table_exists(table):
        return _error(["Table not found: {}".format(table)], 404)

    pk_values = urlsafe_components(request.url_vars["pks"])

    sql, params, pks = await row_sql_params_pks(db, table, pk_values)
    results = await db.execute(sql, params, truncate=True)
    rows = list(results.rows)
    if not rows:
        return _error([f"Record not found: {pk_values}"], 404)

    # Ensure user has permission to update this row
    if not await self.ds.permission_allowed(
        request.actor, "update-row", resource=(database_name, table)
    ):
        return _error(["Permission denied"], 403)

    body = await request.post_body()
    try:
        data = json.loads(body)
    except json.JSONDecodeError as e:
        return _error(["Invalid JSON: {}".format(e)])
    if not isinstance(data, dict):
        return _error(["JSON must be a dictionary"])

    def update_row(conn):
        sqlite_utils.Database(conn)[table].update(pk_values, updates)

    await db.execute_write_fn(update_row)
    result = {"ok": True}
    if data.get("return"):
        result["row"] = {"row-here": "TODO"}
    return Response.json(result, status=200)

``` This is before the refactor in: - #1896

I guess it is not incorrect when it says the version is 4, though it is confusing. Maybe it doesn't even refer to FTS4/FTS5 versions, but something else? In any case, it's not related to sqlite-utils, but SQLite itself.

Okay, my final observations for the night! I've been pushing and pulling the various levers in utils/__init__.py to see what makes this work without hard-coding in something for arm64 and it seems that if I change /usr/lib/x86_64-linux-gnu/mod_spatialite.so here to just mod_spatialite it's happy.

Unfortunately cannot audit that for x86_64, but maybe that's a solution that'd be cross-arch compatible? It seems like it's the hard-coding of that path that's tripping it up.

(It was actually this comment from back in 2018 in an entirely unrelated repo that nudged me to try this, ha.)

Thanks!

Demo: https://latest.datasette.io/fixtures

Can confirm that my uname -a returns something different at the end:

root:xnu-8792.41.9~2/RELEASE_ARM64_T6000 arm64

I'm in arm64 land, you're in x86_64. I am admittedly very fuzzy on how this factors into Docker these days. Honestly thought this was one of the things Docker was suppose to help address. 🤔

Is it, uh, possible we are on different architectures? 😅 I'm using an Apple M1 Pro.

I jumped into a bash shell of an unmodified python:3.11.0-slim-bullseye container and manually ran apt-get update and installed libsqlite3-mod-spatialite. I don't end up with with mod_spatialite.so in /usr/lib/x86_64-linux-gnu/ — mine is in /usr/lib/aarch64-linux-gnu/.

I swapped that directory in here in a local copy of datasette and poof — it worked!

This search helps too: https://ripgrep.datasette.io/-/ripgrep?pattern=%7B%25+block+nav&literal=on&ignore=on&glob=%21datasette%2F**

You get:

=> [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 0.9s => [internal] load build context 2.3s => => transferring context: 72.38MB 2.3s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s

I get:

=> [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 1.0s => [internal] load build context 0.0s => => transferring context: 705B 0.0s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s Both the image name and the hash are exactly the same. So why are you getting an error while mine works OK?

For my machine: ``` ~ % docker --version Docker version 20.10.12, build e91ed57

~ % uname -a Darwin Simons-MacBook-Pro-2.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 x86_64 ```

This is so weird! What version of Datasette do you get from datasette --version there - and what's your Docker version / operating system version?

Here's the full list of 10 plugin releases for this issue:

• datasette-search-all 1.1.1
• datasette-ripgrep 0.7.1
• datasette-socrata 0.3.1
• datasette-configure-fts 1.1.1
• datasette-edit-templates 0.2
• datasette-copyable 0.3.2
• datasette-public 0.2.1
• datasette-import-table 0.3.1
• datasette-indieauth 1.2.2
• datasette-edit-schema 0.5.2

Other plugins this looks like it will affect:

• [x] datasette-ripgrep https://github.com/simonw/datasette-ripgrep/blob/03446464420130368582022eeb5944993f64ec8f/datasette_ripgrep/templates/ripgrep.html#L37-L42
• [x] datasette-socrata https://github.com/simonw/datasette-socrata/blob/32fb256a461bf0e790eca10bdc7dd9d96c20f7c4/datasette_socrata/templates/datasette_socrata_error.html#L5-L10
• [x] datasette-configure-fts https://github.com/simonw/datasette-configure-fts/blob/eca742e5d4b9190fc22d68bc0a406c575e6d09a0/datasette_configure_fts/templates/configure_fts_database.html#L9-L14
• [x] datasette-edit-templates https://github.com/simonw/datasette-edit-templates/blob/f772aff4a2a4080c949746668a8ec6302dbeb0d9/datasette_edit_templates/templates/edit_template.html#L17-L23
• [x] datasette-copyable https://github.com/simonw/datasette-copyable/blob/204d5c912a8d48c49155c67fba7339d4bb26ab9a/datasette_copyable/templates/copyable.html#L36-L43
• [x] datasette-public https://github.com/simonw/datasette-public/blob/32b6a0ba53bd5714b6b41eddd8705b213c105efc/datasette_public/templates/public_table_change_privacy.html#L5-L11
• [x] datasette-import-table https://github.com/simonw/datasettecloud-datasette/blob/37d0fe525c6649c1aec3d1ee8bc35a684570e87f/templates/import_data.html#L5-L10
• [x] datasette-edit-schema (three places)
• [x] datasette-indieauth https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10

That's all of them!

Interesting! So I tried this locally using your copy of nps-spatialite.db and I got the same error. 🤔

``` ❯ datasette package nps-spatialite.db --spatialite [+] Building 27.5s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 622B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 0.9s => [internal] load build context 2.3s => => transferring context: 72.38MB 2.3s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s => [2/6] COPY . /app 0.1s => [3/6] WORKDIR /app 0.0s => [4/6] RUN apt-get update && apt-get install -y python3-dev gcc libsqlite3-mod-spatialite && rm -rf /var/lib/apt/lists/* 18.5s => [5/6] RUN pip install -U datasette 4.9s => ERROR [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json 0.7s

[6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json:

10 0.681 Traceback (most recent call last):

10 0.681 File "/usr/local/bin/datasette", line 8, in <module>

10 0.681 sys.exit(cli())

10 0.681 ^^^^^

10 0.681 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in call

10 0.682 return self.main(args, *kwargs)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main

10 0.682 rv = self.invoke(ctx)

10 0.682 ^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke

10 0.682 return _process_result(sub_ctx.command.invoke(sub_ctx))

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke

10 0.682 return ctx.invoke(self.callback, **ctx.params)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke

10 0.682 return __callback(args, *kwargs)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/cli.py", line 164, in inspect

10 0.683 inspect_data = loop.run_until_complete(inspect_(files, sqlite_extensions))

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete

10 0.683 return future.result()

10 0.683 ^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/cli.py", line 179, in inspect_

10 0.683 counts = await database.table_counts(limit=3600 * 1000)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 304, in table_counts

10 0.683 for table in await self.table_names():

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 342, in table_names

10 0.683 results = await self.execute(

10 0.683 ^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 267, in execute

10 0.683 results = await self.execute_fn(sql_operation_in_thread)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 213, in execute_fn

10 0.683 return await asyncio.get_event_loop().run_in_executor(

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run

10 0.683 result = self.fn(self.args, *self.kwargs)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 209, in in_thread

10 0.683 self.ds._prepare_connection(conn, self.name)

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/app.py", line 593, in _prepare_connection

10 0.683 conn.execute("SELECT load_extension(?)", [extension])

10 0.683 sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory

executor failed running [/bin/sh -c datasette inspect nps-spatialite.db --inspect-file inspect-data.json]: exit code: 1 ```

For datasette-copyable I want to show breadcrumbs that take database/instance permissions into account, so I'm removing {% block nav %} entirely and replacing it with this:

html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database, table=table) }} {% endblock %}

I can't actually remember where that min-height: 70px came from. I just tried without it and it seems fine - especially since any time you add a newline in the editor it increases its height to fit.

I ran this in the DevTools console:

javascript document.querySelector('.cm-editor').style.minHeight = 'none';

Could you provide full steps to reproduce plus a SpatiaLite database file that triggered this for you? I'm not able to recreate the problem.

Did you use the --spatialite option?

I just tried this:

datasette package nps-spatialite.db

It built the image OK (I didn't see the error you reported), but running the container failed with an error:

``` /tmp % docker run -p 8001:8001 7298e8e6bbfb Usage: datasette serve [OPTIONS] [FILES]... Try 'datasette serve --help' for help.

Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

Read more: https://docs.datasette.io/en/stable/spatialite.html ```

Trying this out locally with this 69MB SpatiaLite file I happened to have lying around (from testing shapefile-to-sqlite a while ago): https://static.simonwillison.net/static/2022/nps-spatialite.db % datasette package nps-spatialite.db --spatialite ... => [2/6] COPY . /app 0.4s => [3/6] WORKDIR /app 0.0s => [4/6] RUN apt-get update && apt-get install -y python3-dev gcc libsqlite3-mod-spatialite && rm -rf /var/lib/apt/lists/* 29.6s => [5/6] RUN pip install -U datasette 12.0s => [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json 2.6s => exporting to image 3.0s => => exporting layers 3.0s => => writing image sha256:4dfef1c373c5c057ef7ac22344f834d522acef24313a1b25d2eba9e500066b8f 0.0s And then:

docker run -p 8001:8001 4dfef1c373c5

This worked fine for me. I ran datasette package using Datasette 0.63.1.

Oh this is with datasette package? That should work. Will investigate.

Which Docker image are you using here? It looks like it's missing SpatiaLite from the image.

One of the big changes still left to do for Datasette 1.0 is to unify the JSON representation with the context psssed to the templates (via an ?_extra= mechanism to add extra context needed by the HTML templates), because a goal for 1.0 is for the template context to be a documented API contract such that custom templates won't break with future releases.

As such I expect to do quite a bit of refactoring and cleanup on how the template context works later on.

Are you tracking the change to send the JSON over to the frontend separately or was that part of this? Something like this is probably pretty close https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25#diff-0c93232bfd5477eeac96382e52769108b41433d960d5277ffcccf2f464e60abdR9

Also datasette-indieauth https://github.com/simonw/datasette-indieauth/blob/a08ce67ddad6098b1240adbeff37d040e4df53b1/datasette_indieauth/templates/indieauth.html#L5-L10

Looks like this issue could affect a bunch of other plugins too: https://cs.github.com/?scopeName=All+repos&scope=&q=%3Cp+class%3D%22crumbs%22%3E+user%3Asimonw

Good catch. Looks like that bug was introduced by this change: https://github.com/simonw/datasette/commit/1a5e5f2aa951e5bd731067a49819efba68fbe8ef

From:

• https://github.com/simonw/datasette/issues/1831

The search all plugin includes this code which interacts poorly with that refactor:

https://github.com/simonw/datasette-search-all/blob/847b55c368a285e4567627029624d7872ee75cac/datasette_search_all/templates/search_all.html#L31-L36

html+jinja {% block nav %} <p class="crumbs"> <a href="{{ urls.instance() }}">home</a> </p> {{ super() }} {% endblock %}

I decided to just go for the view names, not their columns.

Views aren't currently available in the _internal schema.

Just noticed that this isn't including views, which it should.

This function works even if the SQLite JSON functions are not available:

python async def _table_columns(datasette, database_name): internal = datasette.get_database("_internal") result = await internal.execute( "select table_name, name from columns where database_name = ?", [database_name], ) table_columns = {} for row in result.rows: table_columns.setdefault(row["table_name"], []).append(row["name"]) return table_columns