issue_comments

I'm going to stash these on the request object after all, so the memory used for the messages gets automatically cleaned up at the end of the request.

Problem with datasette.fetch_and_clear_messages(request, response) is that I want to call it from the template (so we only clear messages if they have been displayed) - but by that point in the code the Response object has not yet been created, so it can't have cookie set on it to clear the list of messages.

Solution: call it datasette.show_messages(request) and have it update internal state on the datasette object such that a later call to write_messages_to_response(request, response) knows to clear the cookie.

datasette.add_message(request, message, type=datasette.INFO)

Then later:

datasette.write_messages_to_response(request, response) Writes the messages as cookies in the response.

datasette.fetch_and_clear_messages(request, response) To display messages and clears them from the response.

A fun thing about this tutorial is that it can start with a classic, basic todo list - and then start growing all kinds of outlandish features to help demonstrate various Datasette plugins and approaches.

Your TODOs on a map. URLs in TODOs that have been unfurled. Tag your TODOs and browse them with facets. Vega graphs showing your progress. Etc etc etc.

It would be neat if this was driven by a method on datasette just because that's already the object passed to plugins as a documented API.

Flask and Django both support "types" of message - info, warning etc. I think I should do the same.

If scope had an immutable correlation ID I could use that with a dict somewhere mapping correlation_id to a messages object.

The problem then is how do I know to clean up the memory used by that dictionary when the request flows out of the system? I guess the code that updates the cookies in the response could do that.

What if I use a mutable key on scope to track messages for the duration of the request? Is that an OK thing to do?

ASGI spec says this: https://asgi.readthedocs.io/en/latest/specs/main.html#middleware

Middleware

It is possible to have ASGI "middleware" - code that plays the role of both server and application, taking in a scope and the send/receive awaitables, potentially modifying them, and then calling an inner application.

When middleware is modifying the scope, it should make a copy of the scope object before mutating it and passing it to the inner application, as changes may leak upstream otherwise. In particular, you should not assume that the copy of the scope you pass down to the application is the one that it ends up using, as there may be other middleware in the way; thus, do not keep a reference to it and try to mutate it outside of the initial ASGI constructor callable that receives scope. Your one and only chance to add to it is before you hand control to the child application.

Here's how the Django stuff works: https://github.com/django/django/blob/master/django/contrib/messages/storage/base.py

Notably the messages are mostly dealt with on the request object, with a piece of middleware that reads from the request and modifies the response (to set or clear cookies) right at the end: https://github.com/django/django/blob/master/django/contrib/messages/middleware.py

Alternative: datasette.add_message(message) and datasette.read_and_clear_messages() - these would need some kind of dark magic to ensure that the message was associated with the current request flowing through the system though, since that datasette object is shared my multiple concurrent requests.

Maybe use a request correlation ID that gets added to the scope? This is all getting a bit messy.

Maybe two utility functions:

add_message(request, message) - adds a Flash message (will be set later)

read_and_clear_messages(request) - reads messages and sets them to be cleared

Problem: the request object isn't created at the very top of the stack - it's actually created within each view. So maybe I need to move its creation up to the top of the routing stuff so that the code that returns the response can see it?

I'm going to build the first version of this with signed cookies.

I'm inclined to do this all on the request object, since it's the object representing the current request as it flows through the application. I need the ability to remember which messages were set and which need to be cleared, so I need to do that on something that is available for the lifetime of the request.

Setting messages just needs access to the response. Reading messages needs access to both request AND response, since it needs to clear the messages that are being displayed.

That's if the messages are persisted exclusively in cookies - which makes sense for Django since it's designed to run as many different load-balanced processes.

Since Datasette is a single process which can access an on-file database, maybe consider storing the flash messages within Datasette memory itself - a sort of session mechanism?

Actually I'm inclined to use cookies now, ala Django: https://docs.djangoproject.com/en/3.0/ref/contrib/messages/

This class stores the message data in a cookie (signed with a secret hash to prevent manipulation) to persist notifications across requests. Old messages are dropped if the cookie data size would exceed 2048 bytes.

I can use the new signed values support from #785 to help build this.

Here's the new default_permissions.py file I can add this permission check to: https://github.com/simonw/datasette/blob/dfdbdf378aba9afb66666f66b78df2f2069d2595/datasette/default_permissions.py#L1-L7

Looks like this (at the moment):

Or I could get fancy and implement my own pm.trace.root.setwriter function which collects data that can be appended to the ?_trace=1 dump.

Easiest way to do this would be with an environment variable.

The skeleton of this page now exists at https://datasette.readthedocs.io/en/latest/authentication.html

I can use a deque with a max length for this: https://docs.python.org/3/library/collections.html#deque-objects

Debugging tool idea: /-/permissions page which shows you the actor and lets you type in the strings for action, resource_type and resource_identifier - then shows you EVERY plugin hook that would have executed and what it would have said, plus when the chain would have terminated.

Bonus: if you're logged in as the root user (or a user that matches some kind of permission check, maybe a check for permissions_debug) you get to see a rolling log of the last 30 permission checks and what the results were across the whole of Datasette. This should make figuring out permissions policies a whole lot easier.

Plugin idea: datasette-allow-all - really simple plugin which just says "yes" to every permission check.

Idea for the authentication piece: I'll have the canned query code execute the following:

python if await datasette.permission_allowed( request.scope.get("actor"), "execute_query", "canned_query", query_name, default=True ): Then I'll add a default plugin to Datasette which implements that plugin hook, looks at the Datasette metadata for that query, and says "No" if the following (and request.scope["actor"] is empty):

json { "databases": { "my-database": { "queries": { "add_twitter_handle": { "sql": "insert into twitter_handles (username) values (:username)", "write": true, "requires_actor": true } } } } } I think I'll support this too:

json "allowed_actors": ["root"] So you can configure queries to only be available to specific {"id": xxx} actors.

This will be the first time the new permission_allowed mechanism from #699 will be exercised in Datasette core.

https://latest.datasette.io/-/actor is now live (it returns null because there's no current way to sign into the latest.datasette.io site - not even with a fake ds_actor cookie because there's no way to know what that site's random secret is).

Some next steps:

• Try out a branch of datasette-auth-github that builds on these new plugin hooks
• Build a datasette-api-tokens plugin which implements Authorization: bearer xxx token support for API access
• Maybe prototype up a datasette-user-accounts plugin which supports username/password accounts and allows an admin user to create/delete them
• Do more work on writable canned queries in #698 and see what they look like if they take advantage of the permissions hook (to restrict some to only allowing authenticated users)

I rebased in #783 so all of this is on master now.

I'm considering this done. I'm going to leave it to plugins to implement a web-based sign-in flow for accounts (at least for the moment).

I should add an entire page to the documentation describing Datasette authentication.

OK, the implementation in PR #783 is in a good state now - it implements the new plugin hooks with tests and documentation, plus it implements this:

$ datasette . --root
http://127.0.0.1:8001/-/auth-token?token=3ca9ee460a6451142389351d19b147bce27d2a785dfb6b5a74f82211be1ede49
...

That URL, when clicked, will set a cookie for the {"id": "root"} user. The cookie is respected and used to populate scope["actor"].

I'm going to merge that pull request and continue working on this stuff on master.

The URL for this will be:

/-/auth-token?token=xxx

The token will be generated by Datasette on startup and will only be valid for a single request, at which point it will be used to set a signed ds_actor cookie and then redirect to the homepage.

That documentation: https://github.com/simonw/datasette/blob/c818de88a9c2683437875f788e325d911c8b767b/docs/config.rst#configuring-the-secret

This is nearly ready to close. I'm going to add documentation for --secret and the DATASETTE_SECRET environment variable.