issue_comments

But then what kind of magic parameters might plugins want to add?

Here's a crazy idea: _scrapedcontent_url - it would look for the url column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function scrapedcontent() would need to optionally be sent the full row dictionary being inserted too.

Brainstorming more potential magic parameters:

• _actor_id
• _actor_name
• _request_ip
• _request_user_agent
• _cookie_cookiename
• _signedcookie_cookiename - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, _signedcookie_cookiename_mynamespace. Might not need special signed cookie support since actor is already usually from a signed cookie.
• _timestamp_unix (not happy with these names yet)
• _timestamp_localtime
• _timestamp_datetime
• _timestamp_utc

python @hookspec def canned_queries(datasette, database, actor): "Return a dictionary of canned query definitions or an awaitable function that returns them"

It would be neat if the queries returned by this hook could be restricted to specific users. I think I can do that by returning an "allow" block as part of the query.

But... what if we allow users to save private queries and we might have thousands of users each with hundreds of saved queries?

For that case it would be good if the plugin hook could take an optional actor parameter.

This would also allow us to dynamically generate a canned query for "return the bookmarks belonging to this actor" or similar!

https://pypi.org/project/datasette/0.45a0/ is the release on PyPI.

And in a fresh virtual environment:

$ pip install datasette==0.45a0 ... $ datasette --version datasette, version 0.45a0 But running pip install datasette still gets 0.44.

This worked!

https://pypi.org/project/datasette/#history

https://github.com/simonw/datasette/releases/tag/0.45a0 is my manually created GitHub prerelease.

https://datasette.readthedocs.io/en/latest/changelog.html#a0-2020-06-18 has the release notes.

A shame Read The Docs doesn't seem to build the docs for these releases -it's not showing the tag in the releases pane here:

Also the new tag isn't an option in the Build menu on https://readthedocs.org/projects/datasette/builds/

Not a big problem though since the "latest" tag on Read The Docs will still carry the in-development documentation.

Problem there is Login CSRF attacks: https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf - I still want to perform CSRF checks on login forms, even though the user may not yet have any cookies.

Maybe I can turn off CSRF checks for cookie-free requests but allow login forms to specifically opt back in to CSRF protection?

So maybe one really easy fix here is to disable CSRF checks entirely for any request that doesn't have any cookies? Also suggested here: https://twitter.com/mrkurt/status/1273682965168603137

New documentation about the alpha/beta releases: https://datasette.readthedocs.io/en/latest/contributing.html#contributing-alpha-beta

Alpha release is running through Travis now: https://travis-ci.org/github/simonw/datasette/builds/699864168

https://pypi.org/project/datasette-render-images/#history worked:

I'm now confident enough that I'll make these changes and ship an alpha of Datasette itself.

Here's the Read The Docs documentation on versioned releases: https://docs.readthedocs.io/en/stable/versions.html

It looks like they do the right thing:

We in fact are parsing your tag names against the rules given by PEP 440. This spec allows “normal” version numbers like 1.4.2 as well as pre-releases. An alpha version or a release candidate are examples of pre-releases and they look like this: 2.0a1.

We only consider non pre-releases for the stable version of your documentation.

https://travis-ci.com/github/simonw/datasette-render-images/builds/172118541 demonstrates that the alpha/beta conditional is working as intended:

One more experiment: I'm going to ship datasette-render-images 0.2 and see if that works correctly - including printing out the new debug section I put in the Travis config here: https://github.com/simonw/datasette-render-images/blob/6b5f22dab75ca364f671f5597556d2665a251bd8/.travis.yml#L35-L39 - which should demonstrate if my conditional for pushing to Docker Hub will work or not.

In the alpha releasing run on Travis that echo statement did NOT execute: https://travis-ci.com/github/simonw/datasette-render-images/builds/172116625

OK, I just shipped 0.2a0 of datasette-render-images - https://pypi.org/project/datasette-render-images/ has no indication of that:

But this page does: https://pypi.org/project/datasette-render-images/#history

And https://pypi.org/project/datasette-render-images/0.2a0/ exists.

In a fresh virtual environment pip install datasette-render-images gets 0.1.

pip install datasette-render-images==0.2a0 gets 0.2a0.

Useful tip from Carlton Gibson: https://twitter.com/carltongibson/status/1273680590672453632

DRF makes ALL views CSRF exempt and then enforces CSRF if you're using Session auth only.

View: https://github.com/encode/django-rest-framework/blob/e18e40d6ae42457f60ca9c68054ad40d15ba8433/rest_framework/views.py#L144 Auth: https://github.com/encode/django-rest-framework/blob/e18e40d6ae42457f60ca9c68054ad40d15ba8433/rest_framework/authentication.py#L130

https://github.com/simonw/datasette-render-images uses Travis and is low-risk for trying this out.

I'm going to try this on a separate repository so I don't accidentally publish a Datasette release I didn't mean to publish!

So maybe this condition is right?

if: (tag IS present) AND NOT (tag =~ [ab])

Travis conditions documentation: https://docs.travis-ci.com/user/conditions-v1

These look useful: branch =~ /^(one|two)-three$/ (tag =~ ^v) AND (branch = master)

So I think if I push a tag of 0.45a0 everything might just work - Travis will build it, push the build to PyPI, PyPI won't treat it as a stable release.

Except... I don't want to push alphas as Docker images - so I need to fix this code:

https://github.com/simonw/datasette/blob/6151c25a5a8d566c109af296244b9267c536bd9a/.travis.yml#L34-L43

I thought I might have to update a regex (my CircleCI configs won't match on a0, example) but it turns out Travis is currently configured to treat ALL tags as potential releases:

https://github.com/simonw/datasette/blob/6151c25a5a8d566c109af296244b9267c536bd9a/.travis.yml#L21-L35

Relevant PEP: https://www.python.org/dev/peps/pep-0440/

Django's implementation dates back 8 years: https://github.com/django/django/commit/40f0ecc56a23d35c2849f8e79276f6d8931412d1

From the PEP:

Implicit pre-release number

Pre releases allow omitting the numeral in which case it is implicitly assumed to be 0. The normal form for this is to include the 0 explicitly. This allows versions such as 1.2a which is normalized to 1.2a0.

I'm going to habitually include the 0.

I'd like this soon, because I want to start experimenting with things like #852 and #842 without shipping those plugin hooks in a full stable release.

I'd be OK with the first version of this not including a plugin hook.

If every magic parameter has a prefix and suffix, like _request_ip and _actor_id, then plugins could register a function for a prefix. Register a function to _actor and actor("id")will be called for _actor_id.

But does it make sense for every magic parameter to be of form _a_b? I think so.

The _actor_id param makes this a bit trickier, because we can't just say "if you see an unknown parameter called X call this function" - our magic parameter logic isn't adding single parameters, it might add a whole family of them.

Yes that can work - and using __missing__ (new in Python 3) is nicer because then the regular dictionary gets checked first: ```python import sqlite3

conn = sqlite3.connect(":memory:")

class Magic(dict): def missing(self, key): return key.upper()

conn.execute("select :name", Magic()).fetchall() Outputs: [('NAME',)] ```

It would be nice if Datasette didn't have to do any additional work to find e.g. _request_ip if that parameter turned out not to be used by the query.

Could I do this with a custom class that implements __getitem__() and then gets passed as SQLite arguments?

I had the same idea again ten days later: #852.

Idea: a mechanism where the asgi_csrf() can take an optional should_protect() callback function which gets called with the scope and decides if the current request should be protected or not. It can then look at headers and paths and suchlike and make its own decisions. Datasette could then provide a should_protect() callback which can interact with plugins.

if you did Origin based CSRF checks, then could the absence of an Origin header be used? https://twitter.com/cnorthwood/status/1273674392757829632

I wonder if it's safe to generically say "Don't do CSRF protection on any request that includes a Authorization: Bearer... header - because it's not possible for a regular browser to send that header since the format is different from the header used in browser-based HTTP basic auth?

datasette-auth-tokens could switch to using asgi_wrapper instead of actor_from_request - then it could add a scope["skip_csrf"] = True scope property to indicate that CSRF should not be protected.

Since asgi_wrapper wraps the CSRF protection middleware changes made to the scope by an asgi_wrapper will be visible to the CSRF middleware:

https://github.com/simonw/datasette/blob/d2aef9f7ef30fa20b1450cd181cf803f44fb4e21/datasette/app.py#L877-L888

The only way I can think of for a view to opt-out of CSRF protection is for them to be able to reconfigure the asgi-csrf middleware to skip specific URL patterns.

Here's the Rails pattern for this: https://gist.github.com/maxivak/a25957942b6c21a41acd

I think there are a couple of steps to this one.

The nature of CSRF is that it's about hijacking existing authentication credentials. If your Datasette site runs without any authentication plugins at all CSRF protection isn't actually useful.

Some POST endpoints should be able to opt-out of CSRF protection entirely. A writable canned query that accepts anonymous poll submissions for example might determine that CSRF is not needed.

If a plugin adds Authorization: Bearer xxx token support that plugin should also be able to specify that CSRF protection can be skipped. https://github.com/simonw/datasette-auth-tokens could do this.

This means I need two new mechanisms:

• A way for wrapped views to indicate "actually don't CSRF protect me". I'm not sure how feasible this is without a major redesign, since the decision to return a 403 forbidden status is made before the wrapped function has even been called.
• A way for authentication plugins like datasette-auth-tokens to say "CSRF protection is not needed for this request". This is a bit tricky too, since right now the actor_from_request hook doesn't have a channel for information other than returning the actor dictionary.

Tweeted about this here: https://twitter.com/simonw/status/1273655053170077701

I have a test that demonstrates this working, but also demonstrates that the CSRF protection from #798 makes this really tricky to work with. I'd like to improve that.

The easiest way to do this would be with a new plugin hook:

def canned_queries(datasette, database):
    """Return a list of canned query definitions
    or an awaitable function that returns them"

Another approach would be to make the whole of metadata.json customizable by plugins.

I think I like the dedicated canned_queries option better. I'm not happy with the way metadata keeps growing - see #493 - so adding a dedicated hook would be more future proof against other changes I might make to the metadata mechanism.

Question about this on Twitter: https://twitter.com/amjithr/status/1273440766862352384