New token design:

json { "a": "actor-id", "t": "creation timestamp as integer", "d": "intended duration in seconds, or blank if no duration set" } This is in place of the "e": "expiry timestamp" design I've built so far.

I just realized this can't easily affect the datasette create-token command because it doesn't currently accept the --setting option, so it wouldn't know what max_signed_tokens_ttl was.

More to the point: even if it did, someone could abuse their knowledge of the secret to create a signed non-expiring token even on servers that didn't want to support those.

So I actually need to redesign the token format: it needs to store the timestamp when the token was created and the intended duration, NOT the timestamp that the token expires at.

Otherwise it's not possible for servers to enforce max_signed_tokens_ttl - someone could always create a token with a custom expires_at timestamp on it outside of the configured limit.

I'm going for consistency with max_csv_mb and max_returned_rows and allow_signed_tokens and default_cache_ttl.

So max_signed_tokens_ttl.

That's deployed to https://latest.datasette.io/ now - some examples:

• https://latest.datasette.io/fixtures?sql=--+one+kind+of+comment%0D%0Aselect+*+from+searchable
• https://latest.datasette.io/fixtures?sql=%2F+Multi%0D%0A++line+comment+%2F%0D%0Aselect+*+from+searchable
• https://latest.datasette.io/fixtures?sql=%2F+Both+kinds+%2F%0D%0A--+of+comment%0D%0A%2F+and+more+%2F%0D%0A--+and+more+and+more%0D%0Aselect+*+from+searchable

I'm never 100% sure how to tell if a regular expression includes a nasty denial of service attack - are there any inputs that could cause this new regex to execute in quadratic time or similar?

Here are the new tests - each of these should now work: https://github.com/simonw/datasette/blob/55a709c480a1e7401b4ff6208f37a2cf7c682183/tests/test_utils.py#L170-L175

I'm experimenting with this: ```python

Allow SQL to start with a / / or -- comment

comment_re = ( # Start of string, then any amount of whitespace r'^(\s' + # Comment that starts with -- and ends at a newline r'(?:--.?\n\s)' + # Comment that starts with / and ends with / r'|(?:/*[\s\S]?*/)' + # Whitespace r')\s' )

allowed_sql_res = [ re.compile(comment_re + r"select\b"), re.compile(comment_re + r"explain\s+select\b"), re.compile(comment_re + r"explain\s+query\s+plan\s+select\b"), re.compile(comment_re + r"with\b"), re.compile(comment_re + r"explain\s+with\b"), re.compile(comment_re + r"explain\s+query\s+plan\s+with\b"), ] ``` This should allow any number of comments of either type as a suffix to the allowed SQL patterns.

Needs extensive unit tests!

I'm not massively worried if it has a flaw in it though, since this is part of Datasette's defense in depth: if a non-SELECT query sneaks through it still shouldn't be able to cause any damage as the database connection is read-only or immutable.

Yeah we should fix this.

https://www.sqlite.org/lang_comment.html - SQLite also supports -- style comments.

I like how explicit the documentation is here:

SQL comments begin with two consecutive "-" characters (ASCII 0x2d) and extend up to and including the next newline character (ASCII 0x0a) or until the end of input, whichever comes first.

C-style comments begin with "/" and extend up to and including the next "/" character pair or until the end of input, whichever comes first. C-style comments can span multiple lines.

Generally though we should expect that people might try to use render_template(...) without passing a request, so Datasette core should be able to handle this.

From the stack trace in Sentry:

So this happened because a custom plugin tried to render forbidden.html without passing in the request.

This was a hit to an authenticated page where the incoming user WAS logged in but did not have permission to view that specific page.

Code in question: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/app.py#L634-L640

Yeah, every time I see something cool done with triggers, I remember that I need to start using triggers.

That trigger solution is pretty neat!

This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.

@eyeseast Maybe you could do this with triggers? Like you can insert JSON-friendly data into a "raw" table, and create a trigger that transforms that inserted data into the proper table

Here's an example:

```sql -- meant to be updated from a Datasette insert create table points_raw(longitude int, latitude int);

-- the target table with proper spatliate geometries create table points(point geometry);

CREATE TRIGGER insert_points_raw INSERT ON points_raw BEGIN insert into points(point) values (makepoint(new.longitude, new.latitude)) END; ```

You could then POST a new row to points_raw like this: POST /db/points_raw Authorization: Bearer xxx Content-Type: application/json { "row": { "longitude": 27.64356, "latitude": -47.29384 } }

Then SQLite with run the trigger and insert a new row in points with the correct geometry point. Downside is you'd have duplicated data with points_raw, but maybe it could be a TEMP table (or have a cron that deletes all rows from that table every so often?)

The problem with how it is currently is that the transformed fts table will return incorrect results (unless the table was only 1 row or something), even if create_triggers was enabled previously. Maybe the simplest solution is to disable fts on a transformed table rather than try to recreate it? Thoughts?

I guess the issue is here: https://github.com/simonw/datasette/blob/9676b2deb07cff20247ba91dad3e84a4ab0b00d1/datasette/utils/init.py#L209

Here is a working regexp allowing it: diff - re.compile(r"^select\b"), + re.compile(r"^\s*(/\*.+?(?=\*/)\*/\s*)*select"), ^\s*: beginning by 0 or an infinite number of \s (spaces, tabs, newlines...) (/\*.+?(?=\*/)\*/\s*)*: 0 or an infinite number of chars beginning by /* and ending to the next occurrence of */ followed by 0 or an infinite number of \s

You can play with the regexp here: https://regex101.com/r/aESXDL/3

I'm going to delay working on this until after I have some of the write APIs built to try it against: - #1851

Documentation: - https://docs.datasette.io/en/1.0-dev/authentication.html#datasette-create-token - https://docs.datasette.io/en/1.0-dev/cli-reference.html#datasette-create-token

This time I'm suspicious that there are open SQLite files tucked away in thread locals hidden inside my thread pool executor:

https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L24

https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L204-L214

Just saw this error again!

Reads from DATASETTE_SECRET or accepts --secret for the signing secret.

Initial design:

datasette create-token <id>

Or:

datasette create-token <id> --expire-after 10m

This setting will disable the "Token never expires" option:

I'm going to split the remaining work into separate issues:

• [x] #1856
• [ ] #1855

Documentation: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens

I started the documentation for the API tokens mechanism here: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens

I'm going to set a convention that an actor signed in via a token should set "token": "something" as a key. Then the /-/create-token view can reject those actors.

I'm going to set a convention that "token": "something" in an actor means that they were authenticated by a token.

"token": "dstok" for example.

It strikes me that users should NOT be able to use a token to create additional tokens.

The current design actually does allow that, since the dstok_ Bearer token can be used to authenticate calls to the /-/create-token page.

So I think I need a mechanism whereby that page can only allow access to users authenticated by cookie.

Not obvious how to do that though, since Datasette's authentication actor system is designed to abstract that detail away!

Because of that I think this is a better name:

--setting allow_signed_tokens off

It's a bit confusing that a setting called allow_create_tokens also causes incoming dstok_ tokens to be ignored.

Is it confusing enough that I should pick a different name for the setting though?

The effect of this setting will be:

• /-/create-tokens interface is no longer available
• Incoming dstok_ tokens are no longer respected by the following code:

https://github.com/simonw/datasette/blob/b29e487bc3fde6418bf45bda7cfed2e081ff03fb/datasette/default_permissions.py#L52-L72

Built a prototype of the actor_from_request() hook for this and now:

% curl http://127.0.0.1:8001/-/actor.json -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsImUiOm51bGx9.6O1OxgNTFkAU6uw7xNcmXYX949A' {"actor": {"id": "root", "dstok": true}}

Implemented that dstok_ prefix and the thing where only the actor["id"] is copied to the "a" field.

Prefix: dstok_ - for Datasette signed token.

The tokens also need something that can be used to differentiate them from alternative token mechanisms that other plugins might provide.

Maybe a prefix before the signed value.

Prefixes are also useful for scanning to check they were not accidentally committed to source control.

Other options:

• --setting default_api_tokens off
• --setting signed_api_tokens off
• --setting allow_create_token off

These feel inconsistent because they don't use the allow_ prefix - but they're also a bit less ugly to look at.

I like that last one.

Rather than duplicating the entire actor into the "a" field, maybe just copy the actor ID?

Would need to restrict token creation to just actors with an ID set.

On that basis, I think the model described above where tokens mainly work to provide an "act on behalf of this actor" - but with optional additional constraints - is a good one.

This is what we do for Sandstorm essentially and I fully agree it's the right way to do API tokens in multiuser systems.

Constraints will definitely be important though. I know I want a token to submit error reports programmatically, but I wouldn't want that token to convey my right to delete tables and records, Little Bobby Tables is out there somewhere, and he's all grown up now.

Tests failed because I added a view without also adding documentation!

I forgot that the deploy still goes out for branches other than main even if the tests aren't passing: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/.github/workflows/deploy-latest.yml#L34-L38

Another thought about tokens that can act on behalf of the user.

Imagine a user has permission to access a table. They create a token that can create that table... but then their permission is revoked. It would be bad if they could still use that token they created earlier to access that table!

On that basis, I think the model described above where tokens mainly work to provide an "act on behalf of this actor" - but with optional additional constraints - is a good one.

Next step: make these tokens actually do something.

If I'm going to change the naming conventions for settings I should do it before Datasette 1.0.

If I'm going to have a setting to disable this feature I need to decide what it will be called.

Closest existing setting is this one, since it's for a feature that is turned on by default:

datasette mydatabase.db --setting allow_download off

So maybe this?

datasette mydatabase.db --setting allow_signed_api_tokens off

I like allow_signed_api_tokens more than allow_api_tokens because if you install a plugin such as https://datasette.io/plugins/datasette-auth-tokens then API tokens will work even though you disabled this default signed token feature.

allow_signed_api_tokens does feel a bit clumsy/verbose though.

Demo is now live at https://latest-1-0-dev.datasette.io/-/create-token - visit https://latest-1-0-dev.datasette.io/login-as-root first to sign in.

If you click "Create token" for "Token never expires" multiple times you currently get exactly the same token each time, since it's just a signed token containing a copy of your actor dictionary.

I'm not sure if I like that. I could give each token a random ID (maybe using secrets.token_hex()) such that different tokens have different identities, which would be useful for logging and auditing and maybe even revocation at some point in the future.

Demo: