Ah, I see the problem. The truncate is inside a loop I didn't realize was there.

Hmm, while tests pass, this may not work as intended on larger datasets. Looking into it.

Documentation: https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor

OK that fix worked.https://github.com/dogsheep/github-to-sqlite/runs/839764768?check_suite_focus=true

I bet it's datasette-search-all.

https://github-to-sqlite.dogsheep.net/-/plugins

[
    {
        "name": "datasette-json-html",
        "static": false,
        "templates": false,
        "version": "0.6",
        "hooks": [
            "prepare_connection",
            "render_cell"
        ]
    },
    {
        "name": "datasette-render-markdown",
        "static": false,
        "templates": false,
        "version": "1.1.2",
        "hooks": [
            "extra_template_vars",
            "render_cell"
        ]
    },
    {
        "name": "datasette-pretty-json",
        "static": false,
        "templates": false,
        "version": "0.2",
        "hooks": [
            "render_cell"
        ]
    },
    {
        "name": "datasette-search-all",
        "static": false,
        "templates": true,
        "version": "0.2.1",
        "hooks": [
            "asgi_wrapper",
            "extra_template_vars"
        ]
    },
    {
        "name": "datasette-vega",
        "static": true,
        "templates": false,
        "version": "0.6.2",
        "hooks": [
            "extra_css_urls",
            "extra_js_urls"
        ]
    }
]

So it looks like it's the ASGI lifespan change I made in https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69 - It must be incompatible with one of the plugins.

https://console.cloud.google.com/run/detail/us-central1/github-to-sqlite/logs?project=datasette-222320 has some clues.

Same error.

Might be that it's not got enough RAM. I'll try deploying to a larger instance.

This is an excellent fix. Thanks!

Not sure why codecov is complaining. I'm going to merge it as-is.

Codecov Report

Merging #890 into master will decrease coverage by 0.01%.
The diff coverage is 80.00%.

@@            Coverage Diff             @@
##           master     #890      +/-   ##
==========================================
- Coverage   83.42%   83.40%   -0.02%     
==========================================
  Files          27       27              
  Lines        3632     3634       +2     
==========================================
+ Hits         3030     3031       +1     
- Misses        602      603       +1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 57879dc...745af3b. Read the comment docs.

Codecov Report

Merging #848 into master will decrease coverage by 0.60%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #848      +/-   ##
==========================================
- Coverage   83.42%   82.82%   -0.61%     
==========================================
  Files          27       26       -1     
  Lines        3632     3540      -92     
==========================================
- Hits         3030     2932      -98     
- Misses        602      608       +6     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 57879dc...0d100d1. Read the comment docs.

I was able to narrow this down to the fact that lifespan protocol is turned on.

I see the workaround you've used here: https://github.com/simonw/datasette-debug-asgi/commit/72d568d32a3159c763ce908c0b269736935c6987

If so, maybe it's time to update some of the asg_wrapper plugins.

FWIW, this error does NOT happen in datasette 0.45a4.

It only started on 0.45a5

In investigating this I'm not convinced 500 errors are being correctly raised by errors in canned writable queries.

A common error with this (and other) magic parameters is for the database query to result in the following:

You did not supply a value for binding 3.

This is a pretty crufty error. I'm inclined to say that ANY missing or invalid magic parameter should be treated as a None value instead.

If a canned query has a title defined that will be used instead: https://latest.datasette.io/fixtures/neighborhood_search

Demo has updated: https://latest.datasette.io/fixtures/magic_parameters

This is a great idea.

Example in the live demo: https://latest.datasette.io/fixtures/magic_parameters

https://simonwillison.net/2020/Jul/1/datasette-045/

https://datasette.readthedocs.io/en/latest/changelog.html#v0-45

I've been testing the WIP using this in the console:

fetch('/data/add_name.json', {
  method: 'POST',
  body: 'name=XXXfetch',
  credentials: 'omit',
  headers: {'Content-Type': 'application/x-www-form-urlencoded'}
})
.then(response => console.log(response))

Against a canned query configured like this:

databases:
  data:
    queries:
      add_name:
        sql: insert into names (name) values (:name)
        write: true

I haven't got it to work yet. Latest error is this one:

INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
Traceback (most recent call last):
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 975, in route_path
    await response.asgi_send(send)
AttributeError: 'tuple' object has no attribute 'asgi_send'
INFO:     127.0.0.1:49938 - "POST /data/add_name.json HTTP/1.1" 500 Internal Server Error

It looks like I'm going to have to rethink how the BaseView code around tables, formats and hashes is structured in order to fix this. That's a big refactoring! I'm moving this to a new milestone for Datasette 0.46.

Don't forget to update the news in the README.

The latest release of https://github.com/simonw/datasette-auth-tokens (0.2) now supports SQL configuration of tokens.

Tokens get verified by plugins. So far there's only one: https://github.com/simonw/datasette-auth-tokens - which has you hard-coding plugins in a configuration file. I have a issue there to add support for database-backed tokens too: https://github.com/simonw/datasette-auth-tokens/issues/1

Well tests need to be updated.

I need to get tests working on Windows.

Codecov Report

Merging #883 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #883   +/-   ##
=======================================
  Coverage   83.42%   83.42%           
=======================================
  Files          27       27           
  Lines        3632     3632           
=======================================
  Hits         3030     3030           
  Misses        602      602           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 676bb64...251884f. Read the comment docs.

Turns out we should include hidden tables in the result dict, or we're breaking tests. I've committed a refactor https://github.com/simonw/datasette/pull/883/commits/4f06e1bf6fbe4b73be770b87f610bf7c0e6e3ea7

I am calling the API from another origin, so injecting CSRF token into templates wouldn't work.

EDIT:

I'll try the new version, it sounds promising

Bearer tokens sound interesting. Where do tokens come from? An auth provider of my choosing? How do they get verified?

Have you tried the method described here? https://datasette.readthedocs.io/en/latest/internals.html#csrf-protection - I'm happy to bulk out that section of the documentation if that doesn't help solve your problem.

I just closed #835 which should make CSRF protection easier to work with - it won't interfere with requests without cookies or requests with Authentication: Bearer token tokens. See also https://github.com/simonw/asgi-csrf/issues/11

You can try out pip install datasette==0.45a5 to get those features. Hopefully releasing a full 0.45 tomorrow.

Does this mean custom routes get to expose endpoints accepting POST requests? I've tried earlier to add some POST endpoints, but requests were being rejected by Datasette due to CSRF

This case may not be covered without extra work:
https://github.com/simonw/datasette/blob/3ec5b1abf6afa2d22a3378092809a1a8c0249d26/datasette/views/database.py#L122-L123

This can be a plugin hook:

@hookspec
def forbidden(datasette, request, message, send):
    "Custom response for a 403 forbidden error"

If the hook returns a Response object, it will be returned to the user. Plugins are likely to want to return a redirect response.

Maybe the hook can instead use the send argument to respond to the request and return True which means "I've responded to this"?

I'm going to leave send off for the moment - I can add that in the future if it turns out it would have been a good idea.

The response from this will never be a 302 - it will always be a 200 if the response worked or a 400 for bad parameters or a 500 for errors. The body returned will always be in JSON format.

I've just realized Datasette tries to count hidden tables too. There are 5 visible tables, 25 hidden tables, which I haven't realize earlier to consider their effect. I've turned off counting for hidden tables to see if it has any effect.

What's the point of counting FTS tables?

I'm going to add some tests for this.

Done: https://latest.datasette.io/-/patterns

Updated documentation: https://datasette.readthedocs.io/en/latest/pages.html

I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions

Tests needed for this:

• If a user has view table but NOT view database / view instance, can they view the table page?
• If a user has view canned query but NOT view database / view instance, can they view the canned query page?
• If a user has view database but NOT view instance, can they view the database page?

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves.

Hah... but check_permissionis a method onBaseView`. Here are the various permission methods at the moment:

https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14

And on BaseView:

https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70

permission_allowed is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on datasette..

I could rename permission_allowed() to check_permission() and have a complementary check_permissions() method.

This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary permission_allowed() alias which prints a deprecation warning to the console, then remove that at 1.0.

I already have this method on Datasette:

async def permission_allowed(self, actor, action, resource=None, default=False):

What would be a good method name that complements that and indicates "check a list of permissions in order"? Should it even run against the request or should you have to hand it request.actor?

Relevant: #835

https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf

Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form.

Sounds like regular CSRF protection to me.

Codecov Report

Merging #868 into master will increase coverage by 0.11%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #868      +/-   ##
==========================================
+ Coverage   83.31%   83.42%   +0.11%     
==========================================
  Files          27       27              
  Lines        3595     3614      +19     
==========================================
+ Hits         2995     3015      +20     
+ Misses        600      599       -1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8a5f81...ef837b3. Read the comment docs.

No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way.

So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good.

Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.

How about a blog? Pre-configured canned queries that are only available to "root", plus datasette-template-sql and default templates for the index page and blog entry pages.

I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact -p 0 --root combination which isn't going to affect many people at all.

Uninstalling datasette-debug-asgi caused the server to startup correctly again.

Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486

Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding lifespan="on" to the call to uvicorn.run().

When I did that here:

uvicorn.run(ds.app(), host=host, port=port, log_level="info", lifespan="on")

The server failed to start with this error:

INFO:     Started server process [68849]
INFO:     Waiting for application startup.
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File ".../uvicorn/lifespan/on.py", line 48, in main
    await app(scope, self.receive, self.send)
  File ".../uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File ".../datasette_debug_asgi.py", line 9, in wrapped_app
    if scope["path"] == "/-/asgi-scope":
KeyError: 'path'

I'm not convinced that function is ever actually being called - I added a print() statement to it and it's not executing. I don't think the tests cover it.

I already have a AsgiLifespan class:
https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905

It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894

Could that startup function also output the --root login URL, if needed?

On startup Datasette currently outputs:

INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.

So the ASGI lifespan protocol is almost certainly the right way to solve this.

Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic --root link a second after the server starts up somehow.

Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py

This line is interesting: is this a hook I can attach to somehow?

        await self.lifespan.startup()

From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475

The challenge is... can we run our own custom code after that line has executed that has access to server and can hence access server.servers[0].sockets[0].getsockname()[1] to find the port?

Within uvicorn it does this:

            if port == 0:
                port = server.sockets[0].getsockname()[1]

That server variable is later stashed here:

self.servers = [server]

Where self is the instance of class Server - which is the class that Uvicorn instantiates and calls .run() on when we do uvicorn.run() here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409

This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30

From https://github.com/simonw/datasette/issues/840#issuecomment-643454625

Another problem: what to display in the "you are logged in as", since we don't dictate an actor design.

I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.

The default will look for the first available of the following keys:

* display
* name
* username
* login
* id

I could borrow the implementation for this from datasette-auth-github
https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html

{% extends "default:base.html" %}

{% block extra_head %}
<style type="text/css">
.hd .logout {
    float: right;
    text-align: right;
    padding-left: 1em;
}
</style>
{% endblock %}

{% block nav %}
    {{ super() }}
    {% if auth and auth.username %}
        <p class="logout">
            <strong>{{ auth.username }}</strong> &middot; <a href="/-/logout">Log out</a>
        </p>
    {% endif %}
{% endblock %}

Now just need the "Logged in as: XXX <logout>" navigation item.

Step one: a "logout" page at /-/logout - which shows you a single CSRF-protected "logout" button if you do a GET against it and logs you out if you do a POST against it.

Using datasette-glitch and the new https://github.com/simonw/datasette-write - currently running on datasette==0.45a4 - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the /-/write interface.

To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596

I think the fix is to move the "show_messages" variable to here:

https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748

Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends base.html. For example, this code in a plugin:

        return Response.html(
            await datasette.render_template(
                "write.html",
                {"databases": databases, "sql": request.args.get("sql") or ""},
                request=request,
            )
        )

This won't display messages. The reason is that the messages are made available to the template context in the BaseView.render() method here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95

I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with register_routes()) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
https://github.com/simonw/datasette/issues/870#issuecomment-650842381

I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with register_routes()) as part of Datasette 0.45 but it would be OK if it slipped to a later release.

I'm going to create the single Request() instance in the DatasetteRouter class - at the beginning of the route_path method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925

This code is interesting:

https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955

I want to change the signature of that return await view(new_scope, receive, send) method to instead take (request, send) - so I can have a single shared request object that's created just once per HTTP request.

The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared Request instance? Should its .scope be replaced with alternative scopes as it travels through the codebase?

So now the problem is simpler: I need to get BaseView to a state where it can accept a shared request object and it can be used in conjunction with register_routes().

I'm going to ditch that AsgiView class too, by combining it into BaseView.

Maybe I could add a as_request_view method as an alternative to as_asgi:

https://github.com/simonw/datasette/blob/a8bcafc1775c8a8655b365ae22a3d64f6361c74a/datasette/utils/asgi.py#L150-L174

Or I could teach the Router to spot the dispatch_request method and call it directly.

I'm happy enough with https://codecov.io/gh/simonw/datasette that I'm not going to spend any more time on this.

Since AsgiRouter is only used as the super-class of the DatasetteRouter class maybe I should get rid of AsgiRouter entirely - no point in having a Datasette-specific subclass of it if the parent class isn't ever used by anything else.

I could also rename it to just Router which is a nicer name than DatasetteRouter.

The key to all of this may be the DatasetteRouter class. It deals with scope right now but if it internally dealt with request that could be enough to fix #864 by adding logic needed by the .add_message() mechanism.

https://github.com/simonw/datasette/blob/0991ea75cc7b265389aa8362414a305ba532d31a/datasette/app.py#L904-L938

There's a lot of complex logic in the DataView class, which handles conditionally returning content as .json or as HTML or as .csv.

That view subclasses AsgiView which is itself request-aware, so maybe I don't need to reconsider how those classes work - just figure out how to hook them up with register_routes.

Currently:

_timestamp_epoch

The number of seconds since the Unix epoch.

_timestamp_date_utc

The date in UTC, e.g. 2020-06-01

_timestamp_datetime_utc

The ISO 8601 datetime in UTC, e.g. 2020-06-24T18:01:07Z

I'm going to rename them to:

• _now_epoch
• _now_date_utc
• _now_datetime_utc

I have two plugins in progress that use this hook now:

• https://github.com/simonw/datasette-init creates tables and views on startup
• https://github.com/simonw/datasette-glitch outputs the login-as-root secret link on Glitch

https://github.com/simonw/datasette-glitch is my new plugin that outputs the root login link on Glitch when the server starts.

This would be a lot easier if I had extracted out the hash logic to a plugin, see #745.

Documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#startup-datasette

Live demo: https://latest.datasette.io/fixtures/magic_parameters

I can use magic parameters from #842 in this.

I'm going to add some canned queries to the metadata.json used by the live demo that illustrate this feature.

Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters

Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters

Codecov Report

Merging #869 into master will increase coverage by 0.23%.
The diff coverage is 90.62%.

@@            Coverage Diff             @@
##           master     #869      +/-   ##
==========================================
+ Coverage   82.99%   83.23%   +0.23%     
==========================================
  Files          26       27       +1     
  Lines        3547     3609      +62     
==========================================
+ Hits         2944     3004      +60     
- Misses        603      605       +2     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1bb33da...9e693a7. Read the comment docs.

I'm going to rename _request_X to _header_X as that better reflects what it now does.

This is really exciting! Thanks so much for looking into this.

I'm interested in moving CI for this repo over to GitHub Actions, so I'd be fine with you getting this to work as an Action rather than through Travis. If you can get it working in Travis though I'll happily land that and figure out how to convert that to GitHub Actions later on.

Skipping CSRF on Authorization: Bearer xxx headers also makes sense for JWT applications, which tend to send JWTs using that form of header.

Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter _cookie_ds_actor to be resolved and returned as JSON data that the attacker can see.

This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.

https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237

Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter _cookie_ds_actor to be resolved and returned as JSON data that the attacker can see.