Thanks for the details write-up! This looks like a bug in Datasette itself when run with Docker. Moving this issue there.

Datasette 0.63 is the release that broke this, thanks to this issue:

• https://github.com/simonw/datasette/issues/1809

This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the invoke_startup() hook.

Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago:

https://datasette.io/plugins/datasette-gunicorn

And I just tested and it has the same bug you describe here! Filed:

• https://github.com/simonw/datasette-gunicorn/issues/5

``` The third Datasette 1.0 alpha release adds upsert support to the JSON API, plus the ability to specify finely grained permissions when creating an API token.

• New /db/table/-/upsert API, documented here. upsert is an update-or-replace: existing rows will have specified keys updated, but if no row matches the incoming primary key a brand new row will be inserted instead. (#1878)
• New register_permissions(datasette) plugin hook. Plugins can now register named permissions, which will then be listed in various interfaces that show available permissions. (#1940)
• The /db/-/create API for creating a table now accepts "ignore": true and "replace": true options when called with the "rows" property that creates a new table based on an example set of rows. This means the API can be called multiple times with different rows, setting rules for what should happen if a primary key collides with an existing row. (#1927)
• Arbitrary permissions can now be configured at the instance, database and resource (table, SQL view or canned query) level in Datasette's Metadata JSON and YAML files. The new "permissions" key can be used to specify which actors should have which permissions. See Other permissions in metadata for details. (#1636)
• The /-/create-token page can now be used to create API tokens which are restricted to just a subset of actions, including against specific databases or resources. See API Tokens for details. (#1947)
• Likewise, the datasette create-token CLI command can now create tokens with a subset of permissions. (#1855)
• New datasette.create_token() API method <datasette_create_token>` for programmatically creating signed API tokens. (#1951)
• /db/-/create API now requires actor to have insert-row permission in order to use the "row" or "rows" properties. (#1937) ```

I got this far: ```diff diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index 8b7e83e3..31d41e00 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -54,7 +54,17 @@ def handle_exception(datasette, request, exception): headers = {} if datasette.cors: add_cors_headers(headers) - if request.path.split("?")[0].endswith(".json"): + # Return JSON error under certain conditions + should_return_json = ( + # URL ends in .json + request.path.split("?")[0].endswith(".json") + or + # Hints from incoming request headers + request.headers.get("content-type") == "application/json" + or "application/json" in request.headers.get("accept", "") + ) + breakpoint() + if should_return_json: return Response.json(info, status=status, headers=headers) else: template = datasette.jinja_env.select_template(templates) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index f27d143f..982543a6 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1140,6 +1140,38 @@ async def test_create_table_permissions( assert data["errors"] == expected_errors

+@pytest.mark.asyncio +@pytest.mark.parametrize( + "headers,expect_json", + ( + ({}, False), + ({"Accept": "text/html"}, True), + ({"Accept": "application/json"}, True), + ({"Content-Type": "application/json"}, True), + ({"Accept": "application/json, text/plain, /"}, True), + ({"Content-Type": "application/json"}, True), + ({"accept": "application/json, text/plain, /"}, True), + ({"content-type": "application/json"}, True), + ), +) +async def test_permission_errors_html_and_json(ds_write, headers, expect_json): + request_headers = {"Authorization": "Bearer bad_token"} + request_headers.update(headers) + response = await ds_write.client.post( + "/data/-/create", + json={}, + headers=request_headers, + ) + assert response.status_code == 403 + if expect_json: + data = response.json() + assert data["ok"] is False + assert data["errors"] == ["Permission denied"] + else: + assert response.headers["Content-Type"] == "text/html; charset=utf-8" + assert "Permission denied" in response.text + + @pytest.mark.asyncio @pytest.mark.parametrize( "input,expected_rows_after", ``` Then decided I would punt this until the next milestone.

Highlights:

• /db/table/-/upsert
• ignore and replace for /db/-/create
• register_permissions() plugin hook
• datasette create-token can create restricted tokens
• /-/create-token can too
• datasette --get --token option
• datasette.create_token() API method

Plus some smaller things.

I fixed this issue to help research this further: - https://github.com/simonw/datasette-ripgrep/issues/26

Now this search works:

https://ripgrep.datasette.io/-/ripgrep?pattern=return+_error&literal=on&glob=datasette%2F**

I wish I had this feature! - https://github.com/simonw/datasette-ripgrep/issues/24

Looks like I have both _error() and _errors() functions in there!

One option: if any rows were truncated, show a button you can click to run the query again with truncation disabled - maybe with ?_truncate=off in the URL or similar.

But... still want to truncate if the user runs a query that would return multiple MBs of HTML (assuming the other Datasette query limits don't stop that from happening).

Related: - #1875

Also weird: errors returned by that mechanism look like this: json { "ok": false, "errors": ["list of error messages"] } While errors returned by the rest of Datasette look like this: https://latest.datasette.io/fixtures/no_table.json

json { "ok": false, "error": "Table not found: no_table", "status": 404, "title": null } Related: - #1875

Looks like the code I've written for permission checking on TableCreateView and friends doesn't use the regular raise Forbidden or raise DatasetteError mechanisms - it does its own thing here:

https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/views/database.py#L580-L584

Which uses this:

https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/views/base.py#L547-L548

Having two different patterns to return errors is bad, I should fix that.

Easiest fix would be to look for accept: application/json and/or content-type: application/json headers.

Not bullet-proof, so people might occasionally make JSON requests and get back an HTML error - but the documentation can tell people that they need to send those headers if they want to reliably get back JSON error messages.

I'm happy with this as a solution.

Sniffing for a { is a little bit tricky though, as the post body is lazily loaded on request here:

https://github.com/simonw/datasette/blob/9ad76d279e2c3874ca5070626a25458ce129f126/datasette/utils/asgi.py#L127-L135

This raises a more complicated issue

At some point I'm likely to want to add an HTML interface for creating tables and inserting and updating rows.

The obvious URLs for that are the same as for the JSON API: /db/table/-/insert and suchlike.

Those endpoints are currently POST only - and can return JSON all the time.

If they start accepting form POSTs too they'll need to be able to accept form-encoded data and return HTML instead. That's OK - they can detect incoming JSON thanks to the content-type header an the fact that the request body starts with { - but the should_return_json fix described above could intefere with how errors are returned if I'm not careful.

I think it can still work though: I'll only set should_return_json = True if the endpoint gets a POST with a body starting {, or a content-type JSON header.

I'm going to prototype that up to see what it looks like.

I need a way for those JSON endpoints to communicate back to the handle_exception handler that they are returning JSON, so it knows to behave differently.

Since it gets the request object, one way could be to have view code set request.should_return_json = True so that the handler knows to do something different.

It's a bit of a cludge though!

Most .json errors DO return as JSON, thanks to this:

https://github.com/simonw/datasette/blob/c094dde3ff2bae030f261e6440d4fb082eb860a9/datasette/handle_exception.py#L19-L24

https://github.com/simonw/datasette/blob/c094dde3ff2bae030f261e6440d4fb082eb860a9/datasette/handle_exception.py#L57-L58

But that code triggers when the URL ends with .json - and none of the JSON write API endpoints (things like /db/-/create and /db/table/-/insert) follow that convention.

I'm going to test this using calls to ds.permission_allowed() with an actor with _r block.

I can add extra tests to https://github.com/simonw/datasette/blob/1a3dcf494376e32f7cff110c86a88e5b0a3f3924/tests/test_permissions.py#L605-L616

Also this code should work with non-abbreviations too.

I don't have much experience with Python; however, I wonder if this print statement needs flush=True?

https://github.com/simonw/datasette/blob/fdf7c27b5438f02153c3a7f8ad1b320e4b29e4f4/datasette/cli.py#L621

Hmm, it appears after I kill the process with Ctrl+c:

Non-memory named databases shouldn't show write actions, since those won't persist.

I think the next big step for this feature is for me to actually use it to build a few things.

It would be neat not to show write permissions against immutable databases too - and not hard from a performance perspective since it doesn't involve hundreds more permission checks.

That will need permissions to grow a flag for if they need a mutable database though, which is a bigger job.

Maybe I should have kept _memory listed for instances that are running with --crossdb enabled?

Yeah I think I should.

New interface now live at https://latest.datasette.io/-/create-token

It shouldn't be showing _memory though.

Tests can go here:

https://github.com/simonw/datasette/blob/d98a8effb10ce8fe04a03eae42baa8a9cb0ca3f7/tests/test_auth.py#L143-L160

I shipped a working interface. Could still do with some extra tests.

Documentation for the new method: https://docs.datasette.io/en/latest/internals.html#create-token-actor-id-expires-after-none-restrict-all-none-restrict-database-none-restrict-resource-none

I like that the word restrict reflects the _r in the actor/token.

Maybe this:

```python datasette.create_token("root", expires_after=3600, restrict_all=("view-query", "view-table"))

token = datasette.create_token("root", expires_after=3600, restrict_database={ "fixtures": ("view-query",) })

token = datasette.create_token("root", expires_after=3600, restrict_resource={ "fixtures": { "facetable": ("insert-row", "update-row") } }) ```

The problem with all=(..) is it feels misleading - it's actually restricting the permissions made available to the token.

Likewise, databases= being a dict of restricted permissions isn't completely obvious.

And the nested tables= dictionary feels a bit odd too.

Some sketches: ```python

Token for root user

token = datasette.create_token("root")

Expiring in an hour

token = datasette.create_token("root", expires_after=3600) More complicated is when you want to restrict to specific permissions:python

Limited to view-query and view-table

token = datasette.create_token("root", expires_after=3600, all=("view-query", "view-table"))

I'm not sure about that all= name

Limits within a specific database:

token = datasette.create_token("root", expires_after=3600, databases={ "fixtures": ("view-query",) })

And specific tables:

token = datasette.create_token("root", expires_after=3600, tables={ "fixtures": { "facetable": ("insert-row", "update-row") } }) ```

It's going to look very similar to the CLI tool, at least in terms of capabilities: ``` Usage: datasette create-token [OPTIONS] ID

Create a signed API token for the specified actor ID

Example:

  datasette create-token root --secret mysecret

To allow only "view-database-download" for all databases:

  datasette create-token root --secret mysecret \
      --all view-database-download

To allow "create-table" against a specific database:

  datasette create-token root --secret mysecret \
      --database mydb create-table

To allow "insert-row" against a specific table:

  datasette create-token root --secret myscret \
      --resource mydb mytable insert-row

Restricted actions can be specified multiple times using multiple --all, --database, and --resource options.

Add --debug to see a decoded version of the token. ```

With tilde-encoding for database and table names the HTML looks like this:

html <input type="checkbox" name="table:weird:foo~3Abar:view-table">

Another option: I could set a time limit - say 200ms - on how long I'm willing to spend calculating permissions before displaying this form

First calculate view permissions for tables and databases (and maybe views and canned queries too).

Then see if I can check every permission that I'm going to show as a checkbox on this page. If I get that done within the time limit use that to show the options.

If I run out of time show all options and maybe include a note saying that some of them may not actually be available.

Also: don't show hidden tables.

Another thing to consider in the future: once Datasette can support thousands of tables (see #417) the list on this page will turn into multiple MBs of HTML, which may cause all kinds of problems - not to mention the overhead of all of those table visibility permission checks.

Hopefully by then I'll have a good fix for the permission listings problem: - #1152

And I can apply the same mechanism here.

FYI @simonw, I don't see that message.

I'm going to ignore the permissions issue for the moment - I'll allow people to select any permissions they like in any of the databases or tables that are visible to them (don't want to leak the existence of databases/tables to users who shouldn't be able to see them).

I think the value of getting this working outweights any potential confusion from not using finely grained permission checks to decide if the user should be able to apply a permission or not.

The tokens themselves won't be able to perform insert-row or similar if the user doesn't have the ability to do that, even if they selected that checkbox.

Here's the checkbox prototype: ```diff diff --git a/datasette/templates/create_token.html b/datasette/templates/create_token.html index a94881ed..1795ebaf 100644 --- a/datasette/templates/create_token.html +++ b/datasette/templates/create_token.html @@ -2,11 +2,20 @@

{% block title %}Create an API token{% endblock %}

+{% block extra_head %} +<style type="text/css"> +#restrict-permissions label { + display: inline; + width: 90%; +} +</style> +{% endblock %} + {% block content %}

Create an API token

-

This token will allow API access with the same abilities as your current user.

This token will allow API access with the same abilities as your current user, {{ request.actor.id }}

{% if errors %} {% for error in errors %} @@ -27,8 +36,39 @@ - + +

{% if token %}

Slightly tricky thing here is that it should only show permissions that the user themselves has - on databases and tables that they have permission to access.

I have a nasty feeling this may require looping through everything and running every permission check, which could get very expensive if there are plugins involved that do their own storage check to resolve a permission.

It's that classic permission system problem: how to efficiently iterate through everything the user has permission to do in one go?

Might be that I have to punt on that, and show the user a list of permissions to select that they might not actually have ability for.

Checkbox interface looks like this. It's not beautiful but it's good enough for the moment:

I think checkboxes will work well.

Here's the data I get back from them (as post_vars()):

{'all:debug-menu': 'on', 'all:insert-row': 'on', 'expire_duration': '', 'expire_type': '', 'table:fixtures:delete-row': 'on', 'table:fixtures:drop-table': 'on', 'table:fixtures:view-query': 'on'}

My <select multiple> prototype: ```diff diff --git a/datasette/templates/create_token.html b/datasette/templates/create_token.html index a94881ed..5bd641cc 100644 --- a/datasette/templates/create_token.html +++ b/datasette/templates/create_token.html @@ -6,7 +6,7 @@

Create an API token

-

This token will allow API access with the same abilities as your current user.

This token will allow API access with the same abilities as your current user, {{ request.actor.id }}

{% if errors %} {% for error in errors %} @@ -28,6 +28,36 @@ + +

{% if token %} diff --git a/datasette/views/special.py b/datasette/views/special.py index 30345d14..9d0fcd31 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -231,7 +231,17 @@ class CreateTokenView(BaseView): return await self.render( ["create_token.html"], request, - {"actor": request.actor}, + { + "actor": request.actor, + "all_permissions": self.ds.permissions.keys(), + "database_permissions": [key for key, value in self.ds.permissions.items() if value.takes_database], + "table_permissions": [key for key, value in self.ds.permissions.items() if value.takes_resource], + "databases": self.ds.databases.keys(), + "database_with_tables": [{ + "database": db.name, + "tables": await db.table_names(), + } for db in self.ds.databases.values()], + }, )

 async def post(self, request):

```

Got an option group thing working:

But... it strikes me that any time you're considering a <select multiple> like this a nested list of checkboxes would actually be better - easier for people to use.

I'm experimenting with a <select multiple> for this.

The usability for keyboards is still pretty awful, but it's a niche enough feature that maybe that's OK for the moment?

javascript var select = document.querySelector('select'); var selected = Array.from(temp0.options).filter(o => o.selected).map(o => o.value)

• [x] I should add a --database example to that help text.

https://latest.datasette.io/-/create-token currently looks like this:

As a reminder, the CLI options that this needs to provide an alternative to are:

https://github.com/simonw/datasette/blob/d4b98d3924dec625a99236e65b1b169ff957381f/docs/cli-reference.rst#L619-L638

Most places I use that exception at the moment set their own non-500 status error: ``` datasette % rg DatasetteError datasette/handle_exception.py 7:from .views.base import DatasetteError 33: elif isinstance(exception, DatasetteError):

datasette/filters.py 2:from datasette.views.base import DatasetteError 22: raise DatasetteError("_where= is not allowed", status=403) 141: raise DatasetteError(

datasette/views/table.py 34:from .base import BaseView, DataView, DatasetteError, ureg, _error 178: raise DatasetteError( 192: raise DatasetteError( 390: raise DatasetteError("Cannot use _sort and _sort_desc at the same time") 394: raise DatasetteError(f"Cannot sort table by {sort}") 400: raise DatasetteError(f"Cannot sort table by {sort_desc}")

datasette/views/base.py 39:class DatasetteError(Exception): 219: raise DatasetteError(str(e), title="Invalid SQL", status=400) 222: raise DatasetteError(str(e)) 224: except DatasetteError: 382: raise DatasetteError( 402: raise DatasetteError(str(e), title="Invalid SQL", status=400) 405: raise DatasetteError(str(e)) 407: except DatasetteError:

datasette/views/table2.py 28:from .base import DataView, DatasetteError, ureg 296: raise DatasetteError( 310: raise DatasetteError( 472: raise DatasetteError("Cannot use _sort and _sort_desc at the same time") 476: raise DatasetteError(f"Cannot sort table by {sort}") 482: raise DatasetteError(f"Cannot sort table by {sort_desc}")

datasette/views/database.py 31:from .base import BaseView, DatasetteError, DataView, _error 188: raise DatasetteError("Invalid database", status=404) 190: raise DatasetteError("Cannot download in-memory databases", status=404) 194: raise DatasetteError("Cannot download database", status=404) ```

https://github.com/simonw/datasette/blob/d4b98d3924dec625a99236e65b1b169ff957381f/datasette/views/table.py#L392-L400

When you run --root you need to follow the special link that gets output to the console:

% datasette --root http://127.0.0.1:8001/-/auth-token?token=036d8055cc8000e9667f21c1dd08722a9358c066463873ad9566d23d88765c52 INFO: Started server process [53934] INFO: Waiting for application startup. INFO: Application startup complete. That /-/auth-token?... link is the one that sets the cookie and lets you in.

I may have found the reason. I don't think the --root option is taking effect.

Visited: http://127.0.0.1:8001/-/permissions

Should make sure that every API that returns an object as the top level (that's almost all of them) includes "ok": true to indicate no errors.

I'm going to hide the options for reducing the scope of the token inside a details/summary element.

Also you should need update-row permission to use the "replace": true option - I should add that rule to /-/insert add well.

I landed this already: - #1636

I tested:

{"id": "root", "_r": {"a": "view-table"}}

New documentation: https://docs.datasette.io/en/latest/authentication.html#restricting-the-actions-that-a-token-can-perform

For the UI: I think I'm going to dump a whole bunch of form elements on the page (so you can set up to 3 of each category of limit without any JavaScript), then add JavaScript that hides all but one of the options and gives you a "add another" widget that adds multiple more.

Decided to do the /-/create-token UI in a separate ticket: - #1947

Demo of the new feature: % datasette create-token --secret s root dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjcwOTA1NjgwfQ.pqSWOwCSNp678hEWl9l5o7m1GaM % datasette --get /-/actor.json {"actor": null} % DATASETTE_SECRET=s datasette --get /-/actor.json --token dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjcwOTA1NjgwfQ.pqSWOwCSNp678hEWl9l5o7m1GaM {"actor": {"id": "root", "token": "dstok"}}

Two options:

• --header "Authorization: Bearer XXX" which can be used to send any headers
• --token XXX to specify the token, which is then sent using that header

I like the second option more, simply because there are currently no other headers that affect how Datasette works. --token feels obvious and easy to use.

For the tests for datasette create-token it would be useful if datasette --get had a mechanism for sending an Authorization: Bearer X header.

I'm going to move this code into datasette/cli.py - it's a bit unexpected having it live in default_permissions.py like this (I couldn't find the code when I went looking for it earlier).

Help looks like this:

``` Usage: datasette create-token [OPTIONS] ID

Create a signed API token for the specified actor ID

Example:

  datasette create-token root --secret mysecret

To only allow create-table:

  datasette create-token root --secret mysecret \
      --all create-table

Or to only allow insert-row against a specific table:

  datasette create-token root --secret myscret \
      --resource mydb mytable insert-row

Restricted actions can be specified multiple times using multiple --all, --database, and --resource options.

Add --debug to see a decoded version of the token.

Options: --secret TEXT Secret used for signing the API tokens [required] -e, --expires-after INTEGER Token should expire after this many seconds -a, --all ACTION Restrict token to this action -d, --database DB ACTION Restrict token to this action on this database -r, --resource DB RESOURCE ACTION Restrict token to this action on this database resource (a table, SQL view or named query) --debug Show decoded token --plugins-dir DIRECTORY Path to directory containing custom plugins --help Show this message and exit. ```

I just noticed this in the existing code:

https://github.com/simonw/datasette/blob/c5d30b58a1cd1c66bbddcf3561db005543ecaf25/datasette/default_permissions.py#L195-L203

Hard-coding those action names should not be necessary any more, especially now we have datasette.permissions for looking up metadata about the permissions.

Initial prototype of the create-token command changes:

diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 406dae40..bbe1247e 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -278,17 +278,55 @@ def register_commands(cli): help="Token should expire after this many seconds", type=int, ) + @click.option( + "alls", + "-a", + "--all", + type=str, + multiple=True, + help="Restrict token to this permission", + ) + @click.option( + "databases", + "-d", + "--database", + type=(str, str), + multiple=True, + help="Restrict token to this permission on this database", + ) + @click.option( + "resources", + "-r", + "--resource", + type=(str, str, str), + multiple=True, + help="Restrict token to this permission on this database resource (a table, SQL view or named query)", + ) @click.option( "--debug", help="Show decoded token", is_flag=True, ) - def create_token(id, secret, expires_after, debug): + def create_token(id, secret, expires_after, alls, databases, resources, debug): "Create a signed API token for the specified actor ID" ds = Datasette(secret=secret) bits = {"a": id, "token": "dstok", "t": int(time.time())} if expires_after: bits["d"] = expires_after + if alls or databases or resources: + bits["_r"] = {} + if alls: + bits["_r"]["a"] = list(alls) + if databases: + bits["_r"]["d"] = {} + for database, action in databases: + bits["_r"]["d"].setdefault(database, []).append(action) + if resources: + bits["_r"]["r"] = {} + for database, table, action in resources: + bits["_r"]["r"].setdefault(database, {}).setdefault( + table, [] + ).append(action) token = ds.sign(bits, namespace="token") click.echo("dstok_{}".format(token)) if debug: Still needs tests, plus I'd like it to use abbreviations if available to keep the token length shorter.

I'm going to rename "t" in the magic format to "r" for resource.

I was going to have the CLI command throw an error if you attempt to use a permission that isn't registered with Datasette, but then I remembered that one of the uses for the CLI tool is to create signed tokens that will work against other Datasette instances (via the --secret option) that might have different plugins installed that register different permission names.

So I might have it output warnings instead.

The datasette create-token command will need to be able to do this too.

Right now that command looks like this: ``` % datasette create-token --help Usage: datasette create-token [OPTIONS] ID

Create a signed API token for the specified actor ID

Options: --secret TEXT Secret used for signing the API tokens [required] -e, --expires-after INTEGER Token should expire after this many seconds --debug Show decoded token --help Show this message and exit. % datasette create-token root --secret sec --debug -e 445 dstok_eyJhIjoicm9vdCIsInRva2VuIjoiZHN0b2siLCJ0IjoxNjY4NDA2MjEzLCJkIjo0NDV9.Hd6qRli6xRKkOIRQgZkPO5iN1wM

Decoded:

{ "a": "root", "token": "dstok", "t": 1668406213, "d": 445 } `` (The--debug` bit adds the decoded token.)

Syntax for adding "insert row" for everything, "update row" for all in the "data" database and "delete row" just for the docs / titles table: datasette create-token root --secret sec \ --all insert-row \ --database data update-row \ --table docs titles delete-row The ir / ur / dr options would work too. To add multiple permissions use these options multiple times: datasette create-token root --secret sec \ --all insert-row \ --all delete-row Short versions: -a and -d and -t.

UPDATE: I have decided to use the term resource in the user-facing elements of this feature instead of table, since that can refer to a SQL view and a canned query as well.

So --resource and -r, not -t.

The hardest piece here is the UI. I'm going to implement the CLI command first.

The thing I'm stuck on at the moment is how to implement it such that an allow block for create-table at the root of the metadata will be checked correctly.

Maybe the algorithm when _resolve_metadata_permissions_blocks(datasette, actor, action, resource) is called should do this:

1. If a root permission block matching that action exists, test with that
2. Next, if resource has been passed, check at the database level
3. If the resource included a table/query, check at that level too

So everything is keyed off the incoming action name.

The implementation for this will go here: https://github.com/simonw/datasette/blob/8bf06a76b51bc9ace7cf72cf0cca8f1da7704ea7/datasette/default_permissions.py#L81-L83

Here's the start of the tests (currently marked as xfail):

https://github.com/simonw/datasette/blob/8bf06a76b51bc9ace7cf72cf0cca8f1da7704ea7/tests/test_permissions.py#L652-L689

A bunch of the work for this just landed - in particular the new scheme is now documented (even though it doesn't work yet):

https://docs.datasette.io/en/latest/authentication.html#other-permissions-in-metadata

Documentation for the new hook: https://docs.datasette.io/en/latest/plugin_hooks.html#register-permissions-datasette

This URL is already used for the https://latest.datasette.io/-/permissions tool - but it could include a block on that page that tells you what permissions are available.

This PR ended up bundling part of the implementation of: - #1636

I'm going to be bad an NOT untangle that from this before I merge it.

Codecov Report

Base: 92.00% // Head: 92.03% // Increases project coverage by +0.02% :tada:

Coverage data is based on head (a1317ab) compared to base (e539c1c). Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head a1317ab differs from pull request most recent head 94e5c75. Consider uploading reports for the commit 94e5c75 to get more accurate results

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Actually one last thing: I said that the error would only occur if the permissions differed in some way.

It's this change which triggers the failures: ```diff diff --git a/datasette/app.py b/datasette/app.py index 760063d5..defa9688 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -707,9 +707,12 @@ class Datasette: ) return crumbs

• async def permission_allowed(self, actor, action, resource=None, default=False):
• async def permission_allowed(self, actor, action, resource=None, default=None): """Check permissions using the permissions_allowed plugin hook""" result = None

• Use default from registered permission, if available

• if default is None and action in self.permissions:
• default = self.permissions[action].default for check in pm.hook.permission_allowed( datasette=self, actor=actor, ```

I'm going to revert that last commit, see if I can get the tests running again and then apply the changes a line at a time to figure out which ones broke things.

Here's my first test failure: ``` tests/test_permissions.py .......F

traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

allow = {}, expected_anon = 403, expected_auth = 403, path = '/fixtures/compound_three_primary_keys' padlock_client = <datasette.utils.testing.TestClient object at 0x107c09d20>

@pytest.mark.parametrize(
    "allow,expected_anon,expected_auth",
    [
        (None, 200, 200),
        ({}, 403, 403),
        ({"id": "root"}, 403, 200),
    ],
)
@pytest.mark.parametrize(
    "path",
    (
        "/",
        "/fixtures",
        "/fixtures/compound_three_primary_keys",
        "/fixtures/compound_three_primary_keys/a,a,a",
        "/fixtures/two",  # Query
    ),
)
def test_view_padlock(allow, expected_anon, expected_auth, path, padlock_client):
    padlock_client.ds._metadata_local["allow"] = allow
    fragment = "🔒</h1>"
    anon_response = padlock_client.get(path)

  assert expected_anon == anon_response.status

E assert 403 == 200 E + where 200 = <datasette.utils.testing.TestResponse object at 0x107aea680>.status

/Users/simon/Dropbox/Development/datasette/tests/test_permissions.py:61: AssertionError

entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /Users/simon/Dropbox/Development/datasette/tests/test_permissions.py(61)test_view_padlock() -> assert expected_anon == anon_response.status (Pdb) anon_response <datasette.utils.testing.TestResponse object at 0x107aea680> (Pdb) anon_response.status 200 (Pdb) path '/fixtures/compound_three_primary_keys' (Pdb) padlock_client.ds.metadata *** AttributeError: 'Datasette' object has no attribute 'metadata' (Pdb) padlock_client.ds._metadata_local {'databases': {'fixtures': {'queries': {'two': {'sql': 'select 1 + 1', 'name': 'two'}, 'from_async_hook': {'sql': 'select 2', 'name': 'from_async_hook'}, 'from_hook': {'sql': "select 1, 'null' as actor_id", 'name': 'from_hook'}}, 'source': None, 'source_url': None, 'license': None, 'license_url': None, 'about': None, 'about_url': None}}, 'allow': {}} (Pdb) allow {} `` It looks like I've broken theallowlogic that notices that if there's an"allow": {}` on the root then anonymous users should not be allowed to view any pages.

Idea: a /-/permissions introspection endpoint for listing registered permissions

At some point I should drop it from all of these other projects too: https://cs.github.com/?scopeName=All+repos&scope=&q=user%3Asimonw+%223.6%22+path%3A.github%2Fworkflows%2F*

One way this could work: if plugins request it, a block like this is added to the page:

```html

Then a function could be provided which extracts and parses that data:javascript var data = await datasette.jsonData(); `` Why anawait? Because then I could have it work the exact same way if the data is NOT available on the page - it could trigger afetch()` call for the same stuff. So loading it on the page becomes an optional performance optimization.

Ah, yes! Was just trying to do this and had the same issue. +1 to this!

I'm going to address those ideas for changes to the permission_allowed() in a separate issue.

What would it take for the register_permissions() hook to be something I'm comfortable landing?

I think it's mainly that the list of permissions it provides should Do More Stuff:

• Participate in unit tests, in particular this one:

https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/tests/conftest.py#L79-L102

• That new default option should be respected - maybe if you omit default= from a call to permission_allowed() it could fall back on the default from there?
• Log a warning if you attempt to check a permission that wasn't registered

Then I can use the permissions - in particular their metadata - to help implement his:

• 1636

I don't plan to implement this for Datasette 1.0, but it should be something that can be added later on in a minor version bump.

Draft docs for the new plugin hook: https://datasette--1940.org.readthedocs.build/en/1940/plugin_hooks.html#register-permissions-datasette

Made a draft PR so ReadTheDocs would deploy my new documentation somewhere.

I'm going to try a spike in a branch with datasette.action_allowed(...) and a register_permissions() plugin hook, to see what they look like.

I think action_allowed is my favourite, even though there's a little bit of concept overlap with table_actions and database_actions.

I never really liked those plugin hook names much to be honest, especially since they are inconsistent with menu_links:

https://github.com/simonw/datasette/blob/d67f812b7327c7075732688f3df728807503dc58/datasette/hookspecs.py#L123-L135

Another option:

python if await datasette.actor_can(actor, "insert-data"...)

Asked ChatGPT for some alternative names, I didn't like any of them:

is_permission_granted
has_permission
check_permission
is_action_allowed
check_access_permission
permission_check
validate_permission
check_actor_permission
verify_permission
check_authorization

It's also referenced in this plugin hook: python @hookspec def permission_allowed(datasette, actor, action, resource): """Check if actor is allowed to perform this action - return True, False or None""" But more importantly, in these ones: ```python @hookspec def table_actions(datasette, actor, database, table, request): """Links for the table actions menu"""

@hookspec def database_actions(datasette, actor, database, request): """Links for the database actions menu""" ``` So the word "action" is already used within Datasette to refer to those things - which are almost but not quite the same as actions-as-permissions: many of the things that show up in those menus relate to permissions the user has, but not necessarily all of them.

Should I rename "permission" to "action" elsewhere too? Maybe have a register_actions(...) plugin hook instead of adding register_permissions(...)?

What else could the word "action" mean?

Currently it's used in the codebase to refer to GitHub Actions, and for code like this: python if await self.permission_allowed( actor=actor, action="view-instance", default=True ): Which is already revealing the confusion between "permission" and "action".

One option: python async def action_allowed(self, actor, action, database=None, resource=None): action_allowed fixes the permission v.s. action thing a bit, and is a new name that doesn't clash with the existing method. I dropped default because that's now a property of the permission itself. table is now called resource and database is a separate parameter.

What if I came up with a new method name for this, which could co-exist with the old one while that old one was deprecated?

Extracted a TIL: https://til.simonwillison.net/github/github-code-search-api-uses