A /-/debug-token page that can take a token and decode it to show you how long until it expires, what actor it represents and the permissions it has will be useful as well.

I'm going to build an initial /-/create-token interface which just bakes a token with the current actor in it and an optional expiry timestamp. I'll try the limited permissions thing later.

In which case the token would need to duplicate the current actor and then add extra constraints. So maybe the token design looks like this: json { "a": { "copy_of": "actor_creating_token"}, "p": { "t": "... the thing designed earlier, with those permissions in it" }, "e": "integer timestamp when token expires" }

... so maybe there's a way to create a token that inherits the exact permissions of the actor that created the token? That could even be a default mode for tokens, with an option to then further restrict permissions if desired.

Interesting challenge: what permissions should users be allowed to grant to tokens?

Clearly a user should not be able to create a token with a permission that the user themselves does not have.

And should there be a permission that allows people to create tokens? I think so.

Datasette currently defaults to having everything public-readable by default, unless a permission plugin changes that default.

In thinking more about this API mechanism, I realized that it might be good to have a mode where Datasette doesn't default to public everything. Maybe datasette --private to start it like that?

Might even be an opportunity to get rid of the current slightly confusing mechanism where permission checks can announce that they should default to true:

https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/views/database.py#L152-L154

The datasette insert concept included plugin support, with the idea of being able to support things like SpatiaLite files: - #1160

I think this API mechanism is going to be a bit less exciting than that - it will be low-level for inserting rows, and if you want to do something fancier you can use a canned query that feeds incoming GeoJSON to a SpatiaLite function instead.

Tweeted about it here: https://twitter.com/simonw/status/1585038766678609921

• Now tested against Python 3.11. (#502)
• New table.search_sql(include_rank=True) option, which adds a rank column to the generated SQL. Thanks, Jacob Chapman. (#480)
• Progress bars now display for newline-delimited JSON files using the --nl option. Thanks, Mischa Untaga. (#485)
• New db.close() method. (#504)
• Conversion functions passed to table.convert(...) can now return lists or dictionaries, which will be inserted into the database as JSON strings. (#495)
• sqlite-utils install and sqlite-utils uninstall commands for installing packages into the same virtual environment as sqlite-utils, described here. (#483)
• New sqlite_utils.utils.flatten() utility function. (#500)
• Documentation on using Just to run tests, linters and build documentation.
• Documentation now covers the Release process for this package.

I was going to suggest using db.table(name) instead of db[name] - but it looks like that method will have the same problem:

https://github.com/simonw/sqlite-utils/blob/defa2974c6d3abc19be28d6b319649b8028dc966/sqlite_utils/db.py#L497-L506

I could change sqlite-utils so db.table(name) always returns a table and you need to call db.view(name) if you want to access a view - that would require bumping to 4.0 though. I'm not convinced that's the best approach here either.

I do care about this, but I'm not hugely experienced with types yet so I'm open to suggestions about how to do it!

Based on the docs here I tried the following too: https://docutils.sourceforge.io/docs/user/smartquotes.html#description

• \--
• \\--
• \\-\\-
• \-\-

But none of them had the desired effect in this particular piece of markup:

the :ref:`insert \--convert <cli_insert_convert>`

I think because this is text inside a :ref: block, not regular text.

Consider the following:

The \--convert and the :ref:`insert \--convert <cli_insert_convert>` and

It's rendered like this:

I've decided not to explicitly document this, since it's consistent with how other parts of the library work already.

There is a case where the function can return a dictionary at the moment: multi=True

python table.convert( "title", lambda v: {"upper": v.upper(), "lower": v.lower()}, multi=True ) But I think this change is still compatible with that. if you don't use multi=True then the return value will be stringified. If you DO use multi=True then something like this could work: python table.convert( "title", lambda v: {"upper": {"str": v.upper()}, "lower": {"str": v.lower()}}, multi=True ) This would result in a upper and lower column, each containing the JSON string {"str": "UPPERCASE"}.

This makes sense to me. There are other places in the codebase where JSON is automatically stringified: https://github.com/simonw/sqlite-utils/blob/c7e4308e6f49d929704163531632e558f9646e4a/sqlite_utils/db.py#L2759-L2766

I don't see why the return value from a convert function shouldn't do the same thing.

Since this will result in previous errors working, I don't think it warrants a major version bump either.

Yeah, table.columns and table.columns_dict are meant to handle this: https://sqlite-utils.datasette.io/en/stable/python-api.html#columns

Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.close

TIL about this: https://til.simonwillison.net/python/os-remove-windows

That fixed it.

Used psutil to confirm that closing a SQLite connection closes the underlying file: https://til.simonwillison.net/python/too-many-open-files-psutil ```pycon

import psutil import sqlite3 for f in psutil.Process().open_files(): print(f) ... sqlite3.connect("/tmp/blah.db") <sqlite3.Connection object at 0x1007264d0> conn = _ for f in psutil.Process().open_files(): print(f) ... popenfile(path='/private/tmp/blah.db', fd=3) conn.close() for f in psutil.Process().open_files(): print(f) ...

```

... or maybe Windows doesn't like attempts to remove a file that the process has opened?

This test reliably fails on Windows with Python 3.11.

I'm going to skip the test for the moment to get back to green CI... but I'll leave this issue open. This is definitely concerning, I just don't have the right local environment to solve this at the moment.

I'm not hugely happy with my fix there:

https://github.com/simonw/sqlite-utils/blob/c5d7ec1dd71fa1dce829bc8bb82b639018befd63/sqlite_utils/db.py#L321-L328

The problem here was that in the case where the os.remove() failed the self.conn property was NOT being set to a valid connection - which caused __repr__ to fail later on.

So now I catch the os.remove() error, set self.conn to a memory connection, then raise the error again.

Now mypy is failing:

sqlite_utils/db.py:474: error: Item "None" of "Optional[Any]" has no attribute "execute" sqlite_utils/db.py:476: error: Item "None" of "Optional[Any]" has no attribute "execute" sqlite_utils/db.py:486: error: Item "None" of "Optional[Any]" has no attribute "executescript" sqlite_utils/db.py:603: error: Item "None" of "Optional[Any]" has no attribute "__enter__" sqlite_utils/db.py:603: error: Item "None" of "Optional[Any]" has no attribute "__exit__" sqlite_utils/db.py:604: error: Item "None" of "Optional[Any]" has no attribute "execute" sqlite_utils/db.py:607: error: Item "None" of "Optional[Any]" has no attribute "execute" sqlite_utils/db.py:1082: error: Item "None" of "Optional[Any]" has no attribute "__enter__" sqlite_utils/db.py:1082: error: Item "None" of "Optional[Any]" has no attribute "__exit__" sqlite_utils/db.py:1083: error: Item "None" of "Optional[Any]" has no attribute "cursor" sqlite_utils/db.py:1155: error: Item "None" of "Optional[Any]" has no attribute "enable_load_extension" sqlite_utils/db.py:1156: error: Item "None" of "Optional[Any]" has no attribute "load_extension" Found 12 errors in 1 file (checked 51 source files)

Failed again, but just noticed this: https://github.com/simonw/sqlite-utils/actions/runs/3323932266/jobs/5494890223

```

  Database(filepath, recreate=True)["t2"].insert({"foo": "bar"})

tests\test_recreate.py:31:

self = <[AttributeError("'Database' object has no attribute 'conn'") raised in repr()] Database object at 0x29fc125aa90> ```

This is the failing test: https://github.com/simonw/sqlite-utils/blob/7b2d1c0ffd0b874e280292b926f328a61cb31e2c/tests/test_recreate.py#L21-L32

I'm going to try a different way of creating the temporary file: https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html

Passes on Windows with other Python versions for some reason.

This pattern should work (for the http server at least): python # Loop until port 8041 serves traffic while True: try: httpx.get("http://localhost:8041/") break except httpx.ConnectError: time.sleep(0.1)

Here's the code that starts those various servers: https://github.com/simonw/datasette/blob/613ad05c095f92653221db267ef53d54d00cdfbb/tests/conftest.py#L104-L177

I don't like those time.sleep(1.5) lines much - I'm going to try polling for readiness instead.

Also: ``` @pytest.mark.serial @pytest.mark.skipif( not hasattr(socket, "AF_UNIX"), reason="Requires socket.AF_UNIX support" ) def test_serve_unix_domain_socket(ds_unix_domain_socket_server): _, uds = ds_unix_domain_socket_server transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport)

  response = client.get("http://localhost/_memory.json")

/home/runner/work/datasette/datasette/tests/test_cli_serve_server.py:35:

/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1039: in get return self.request( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:815: in request return self.send(request, auth=auth, follow_redirects=follow_redirects) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:902: in send response = self._send_handling_auth( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:930: in _send_handling_auth response = self._send_handling_redirects( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:967: in _send_handling_redirects response = self._send_single_request(request) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1003: in _send_single_request response = transport.handle_request(request) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_transports/default.py:217: in handle_request with map_httpcore_exceptions(): /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/contextlib.py:153: in exit self.gen.throw(typ, value, traceback)

@contextlib.contextmanager
def map_httpcore_exceptions() -> typing.Iterator[None]:
    try:
        yield
    except Exception as exc:  # noqa: PIE-786
        mapped_exc = None

        for from_exc, to_exc in HTTPCORE_EXC_MAP.items():
            if not isinstance(exc, from_exc):
                continue
            # We want to map to the most specific exception we can find.
            # Eg if `exc` is an `httpcore.ReadTimeout`, we want to map to
            # `httpx.ReadTimeout`, not just `httpx.TimeoutException`.
            if mapped_exc is None or issubclass(to_exc, mapped_exc):
                mapped_exc = to_exc

        if mapped_exc is None:  # pragma: nocover
            raise

        message = str(exc)

      raise mapped_exc(message) from exc

E httpx.ConnectError: [Errno 2] No such file or directory ```

Opened an issue here: - https://github.com/coleifer/pysqlite3/issues/43

https://latest.datasette.io/-/versions now shows 3.11.

Replicated locally using a fresh virtual environment with Python 3.11 and:

 pytest -k test_query_invalid_function

This broke the deploy of https://latest.datasette.io/ - because it tries to install pysqlite3-binary which doesn't have a 3.11 release yet:

https://github.com/simonw/datasette/blob/2e9751672d4fe329b3c359d5b7b1992283185820/.github/workflows/deploy-latest.yml#L77

I started using that for the latest.datasette.io demo in https://github.com/simonw/datasette/commit/a970276b9999687b96c5e11ea1c817d814f5d267 because I wanted a version of SQLite that supported generated columns.

Those were added in SQLite 3.31.0 - and the SQLite version in the new base image is 3.34.1 - so I don't actually need pysqlite3-binary any more.

And tested datasette package like this:

datasette package fixtures.db -t datasette-package-python-upgrade-3-11 Then: docker run -p 8081:8001 datasette-package-python-upgrade-3-11 And tested it like this: curl http://localhost:8081/-/versions.json | jq Output: { "python": { "version": "3.11.0", "full": "3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]" },

Also tested by running this locally:

datasette publish cloudrun fixtures.db --service issue-1853

https://issue-1853-j7hipcg4aq-uc.a.run.app/-/versions now shows Python 3.11.0.

Ran the upgrade on the Datasette Cloud image first, works fine there.

https://simon.datasette.cloud/-/versions shows me:

{ "python": { "version": "3.11.0", "full": "3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]" },

I ran a very crude benchmark on my laptop using Locust (against the official macOS packages from www.python.org for Python 3.10 and Python 3.11) and saw a substantial speed increase:

533.89 requests/second on 3.11 413.56 requests/second on 3.10

That was from running Locust against this locustfile.py: ```python from locust import HttpUser, task

class CounterOne(HttpUser): @task def hello(self): self.client.get("/-/static/app.css") ``` Using:

locust --headless --users 4 --spawn-rate 4 -H http://127.0.0.1:8001

This is going to tie into Datasette's existing permissions mechanism, so plugins will be able to define their own custom mechanisms for tokens to be attached to a specific identity: https://docs.datasette.io/en/stable/authentication.html

There's only one plugin for API tokens at the moment, which is this one: https://datasette.io/plugins/datasette-auth-tokens

I'm actually planning on adding another, default token mechanism to Datasette itself as part of this work:

• 1852

It may well be that datasette-sandstorm-support needs to add something custom here too.