issue_comments

Done: https://latest.datasette.io/-/patterns

Updated documentation: https://datasette.readthedocs.io/en/latest/pages.html

I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions

Tests needed for this:

• If a user has view table but NOT view database / view instance, can they view the table page?
• If a user has view canned query but NOT view database / view instance, can they view the canned query page?
• If a user has view database but NOT view instance, can they view the database page?

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves.

Hah... but check_permissionis a method onBaseView`. Here are the various permission methods at the moment:

https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14

And on BaseView:

https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70

permission_allowed is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on datasette..

I could rename permission_allowed() to check_permission() and have a complementary check_permissions() method.

This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary permission_allowed() alias which prints a deprecation warning to the console, then remove that at 1.0.

I already have this method on Datasette:

async def permission_allowed(self, actor, action, resource=None, default=False):

What would be a good method name that complements that and indicates "check a list of permissions in order"? Should it even run against the request or should you have to hand it request.actor?

Relevant: #835

https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf

Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form.

Sounds like regular CSRF protection to me.

No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way.

So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good.

Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.

How about a blog? Pre-configured canned queries that are only available to "root", plus datasette-template-sql and default templates for the index page and blog entry pages.

I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact -p 0 --root combination which isn't going to affect many people at all.

Uninstalling datasette-debug-asgi caused the server to startup correctly again.

Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486

Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding lifespan="on" to the call to uvicorn.run().

When I did that here:

uvicorn.run(ds.app(), host=host, port=port, log_level="info", lifespan="on")

The server failed to start with this error:

INFO:     Started server process [68849]
INFO:     Waiting for application startup.
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File ".../uvicorn/lifespan/on.py", line 48, in main
    await app(scope, self.receive, self.send)
  File ".../uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File ".../datasette_debug_asgi.py", line 9, in wrapped_app
    if scope["path"] == "/-/asgi-scope":
KeyError: 'path'

I'm not convinced that function is ever actually being called - I added a print() statement to it and it's not executing. I don't think the tests cover it.

I already have a AsgiLifespan class:
https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905

It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894

Could that startup function also output the --root login URL, if needed?

On startup Datasette currently outputs:

INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.

So the ASGI lifespan protocol is almost certainly the right way to solve this.

Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic --root link a second after the server starts up somehow.

Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py

This line is interesting: is this a hook I can attach to somehow?

        await self.lifespan.startup()

From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475

The challenge is... can we run our own custom code after that line has executed that has access to server and can hence access server.servers[0].sockets[0].getsockname()[1] to find the port?

Within uvicorn it does this:

            if port == 0:
                port = server.sockets[0].getsockname()[1]

That server variable is later stashed here:

self.servers = [server]

Where self is the instance of class Server - which is the class that Uvicorn instantiates and calls .run() on when we do uvicorn.run() here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409

This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30

From https://github.com/simonw/datasette/issues/840#issuecomment-643454625

Another problem: what to display in the "you are logged in as", since we don't dictate an actor design.

I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.

The default will look for the first available of the following keys:

* display
* name
* username
* login
* id

I could borrow the implementation for this from datasette-auth-github
https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html

{% extends "default:base.html" %}

{% block extra_head %}
<style type="text/css">
.hd .logout {
    float: right;
    text-align: right;
    padding-left: 1em;
}
</style>
{% endblock %}

{% block nav %}
    {{ super() }}
    {% if auth and auth.username %}
        <p class="logout">
            <strong>{{ auth.username }}</strong> &middot; <a href="/-/logout">Log out</a>
        </p>
    {% endif %}
{% endblock %}

Now just need the "Logged in as: XXX <logout>" navigation item.

Step one: a "logout" page at /-/logout - which shows you a single CSRF-protected "logout" button if you do a GET against it and logs you out if you do a POST against it.

Using datasette-glitch and the new https://github.com/simonw/datasette-write - currently running on datasette==0.45a4 - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the /-/write interface.

To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596

I think the fix is to move the "show_messages" variable to here:

https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748

Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends base.html. For example, this code in a plugin:

        return Response.html(
            await datasette.render_template(
                "write.html",
                {"databases": databases, "sql": request.args.get("sql") or ""},
                request=request,
            )
        )

This won't display messages. The reason is that the messages are made available to the template context in the BaseView.render() method here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95

I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with register_routes()) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
https://github.com/simonw/datasette/issues/870#issuecomment-650842381

I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with register_routes()) as part of Datasette 0.45 but it would be OK if it slipped to a later release.

I'm going to create the single Request() instance in the DatasetteRouter class - at the beginning of the route_path method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925

This code is interesting:

https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955

I want to change the signature of that return await view(new_scope, receive, send) method to instead take (request, send) - so I can have a single shared request object that's created just once per HTTP request.

The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared Request instance? Should its .scope be replaced with alternative scopes as it travels through the codebase?

So now the problem is simpler: I need to get BaseView to a state where it can accept a shared request object and it can be used in conjunction with register_routes().

I'm going to ditch that AsgiView class too, by combining it into BaseView.

Maybe I could add a as_request_view method as an alternative to as_asgi:

https://github.com/simonw/datasette/blob/a8bcafc1775c8a8655b365ae22a3d64f6361c74a/datasette/utils/asgi.py#L150-L174

Or I could teach the Router to spot the dispatch_request method and call it directly.

I'm happy enough with https://codecov.io/gh/simonw/datasette that I'm not going to spend any more time on this.

Since AsgiRouter is only used as the super-class of the DatasetteRouter class maybe I should get rid of AsgiRouter entirely - no point in having a Datasette-specific subclass of it if the parent class isn't ever used by anything else.

I could also rename it to just Router which is a nicer name than DatasetteRouter.

The key to all of this may be the DatasetteRouter class. It deals with scope right now but if it internally dealt with request that could be enough to fix #864 by adding logic needed by the .add_message() mechanism.

https://github.com/simonw/datasette/blob/0991ea75cc7b265389aa8362414a305ba532d31a/datasette/app.py#L904-L938

There's a lot of complex logic in the DataView class, which handles conditionally returning content as .json or as HTML or as .csv.

That view subclasses AsgiView which is itself request-aware, so maybe I don't need to reconsider how those classes work - just figure out how to hook them up with register_routes.

Currently:

_timestamp_epoch

The number of seconds since the Unix epoch.

_timestamp_date_utc

The date in UTC, e.g. 2020-06-01

_timestamp_datetime_utc

The ISO 8601 datetime in UTC, e.g. 2020-06-24T18:01:07Z

I'm going to rename them to:

• _now_epoch
• _now_date_utc
• _now_datetime_utc

I have two plugins in progress that use this hook now:

• https://github.com/simonw/datasette-init creates tables and views on startup
• https://github.com/simonw/datasette-glitch outputs the login-as-root secret link on Glitch

https://github.com/simonw/datasette-glitch is my new plugin that outputs the root login link on Glitch when the server starts.

This would be a lot easier if I had extracted out the hash logic to a plugin, see #745.

Documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#startup-datasette

Live demo: https://latest.datasette.io/fixtures/magic_parameters

I can use magic parameters from #842 in this.

I'm going to add some canned queries to the metadata.json used by the live demo that illustrate this feature.

Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters

Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters

I'm going to rename _request_X to _header_X as that better reflects what it now does.

This is really exciting! Thanks so much for looking into this.

I'm interested in moving CI for this repo over to GitHub Actions, so I'd be fine with you getting this to work as an Action rather than through Travis. If you can get it working in Travis though I'll happily land that and figure out how to convert that to GitHub Actions later on.

Skipping CSRF on Authorization: Bearer xxx headers also makes sense for JWT applications, which tend to send JWTs using that form of header.

Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter _cookie_ds_actor to be resolved and returned as JSON data that the attacker can see.

This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.

https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237

Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter _cookie_ds_actor to be resolved and returned as JSON data that the attacker can see.

Maybe I should ship a default _scope_headers_... parameter instead, which reads from a dictionary of scope["headers"] - https://asgi-scope.now.sh/ shows what those look like.

{'client': ('148.64.98.14', 0),
 'headers': [[b'host', b'asgi-scope.now.sh'],
             [b'x-forwarded-for', b'148.64.98.14'],
             [b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'],
             [b'x-forwarded-host', b'asgi-scope.now.sh'],
             [b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'],
             [b'x-real-ip', b'148.64.98.14'],
             [b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'],
             [b'upgrade-insecure-requests', b'1'],
             [b'x-vercel-trace', b'sfo1'],
             [b'x-forwarded-proto', b'https'],
             [b'accept-language', b'en-US,en;q=0.5'],
             [b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko'
              b'/20100101 Firefox/77.0'],
             [b'x-vercel-forwarded-for', b'148.64.98.14'],
             [b'accept-encoding', b'gzip, deflate, br'],
             [b'dnt', b'1'],
             [b'te', b'trailers']],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/',
 'query_string': b'',
 'raw_path': b'/',
 'root_path': '',
 'scheme': 'https',
 'server': ('asgi-scope.now.sh', 80),
 'type': 'http'}

I'm going to have _request_X actually mean "find the first value for X in scope["headers"]" - with underscores converted to hyphens.

_request_ip is actually quite hard to implement - should it take into account things like the x-forwarded-for header?

It probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment.

As such I think I'll leave that for a plugin.

Here's the relevant code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/datasette/app.py#L1057-L1070
And the relevant test code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/test_plugins.py#L567-L573

https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/plugins/my_plugin.py#L162-L196

I mentioned this issue here: https://simonwillison.net/2020/Jun/26/weeknotes-plugins-sqlite-generate/

Repositories created by following the REAME in https://github.com/simonw/datasette-template and https://github.com/simonw/click-app have a main branch instead of master so I have a few examples live now. https://github.com/simonw/datasette-saved-queries is one example.

I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters

But then what kind of magic parameters might plugins want to add?

Here's a crazy idea: _scrapedcontent_url - it would look for the url column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function scrapedcontent() would need to optionally be sent the full row dictionary being inserted too.

Brainstorming more potential magic parameters:

• _actor_id
• _actor_name
• _request_ip
• _request_user_agent
• _cookie_cookiename
• _signedcookie_cookiename - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, _signedcookie_cookiename_mynamespace. Might not need special signed cookie support since actor is already usually from a signed cookie.
• _timestamp_unix (not happy with these names yet)
• _timestamp_localtime
• _timestamp_datetime
• _timestamp_utc

Another magic parameter that would be useful would be _random. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.

Maybe _random_chars_128 to create a 128 character long random string (using os.urandom(64).hex()).

This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately _random_chars:128 wouldn't work because these parameters are used in a SQLite query where : has special meaning: insert into blah (secret) values (:_random_chars:128) wouldn't make sense.

Actually this is already supported by the proposed design - _random_chars_128 would become random("chars_128") so the random() function could split off the 128 itself.

Thanks for the bug report. Yes I think #838 may be the same issue. Will investigate.

I really need to get myself a Windows 10 development environment working so I can dig into this kind of bug properly. I have a gaming PC lying around that I could re-task for that.

The ideal fix here would be to rework my BaseView subclass mechanism to work with register_routes() so that those views don't have any special privileges above plugin-provided views.

Urgh, fixing this is going to be a bit of a pain.

Here's where I added that custom dispatch_request() method - it was to implement flash messaging in #790: https://github.com/simonw/datasette/blame/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/views/base.py#L85

If I want this to be made available to register_routes() views as well, I'm going to have to move the logic somewhere else. In particular I need to make sure that the request object is created once and used throughout the whole request cycle.

Currently register_routes() view functions get their own separate request object which is created here:

https://github.com/simonw/datasette/blob/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/app.py#L1057-L1068

So I'm going to have to refactor this quite a bit to get that shared request object which can be passed both to register_routes views and to my various BaseView subclasses.

So there are two sides to supporting this:

• Being able to sensibly introspect composite foreign keys
• Being able to define composite foreign keys when creating a table

Relevant discussion: https://github.com/simonw/sqlite-generate/issues/8#issuecomment-648438056

Here's what's missing:

In [11]: db.conn.execute('PRAGMA foreign_key_list(song)').fetchall()                                                                       
Out[11]: 
[(0,
  0,
  'album',
  'songartist',
  'albumartist',
  'NO ACTION',
  'NO ACTION',
  'NONE'),
 (0, 1, 'album', 'songalbum', 'albumname', 'NO ACTION', 'NO ACTION', 'NONE')]

Compare with this code here:
https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/sqlite_utils/db.py#L563-L579

The first two columns returned by PRAGMA foreign_key_list(table) are id and seq - these show when two foreign key records are part of the same compound foreign key. sqlite-utils entirely ignores those at the moment.

New docs: https://github.com/simonw/sqlite-utils/blob/2.10.1/docs/python-api.rst#introspection

Should go in this section https://sqlite-utils.readthedocs.io/en/stable/python-api.html#introspection - under .columns_dict and before .foreign_keys.

I wonder if this is a SQLite caching issue then?

Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb

I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030

I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.

Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys

If it's the table page then the problem may well be #862.

Related: #862 - a time limit on the total time spent considering suggested facets for a table.

Should this be controlled be a separate configuration setting? I'm inclined to say no - I think instead I'll set the limit to be 10 * whatever facet_suggest_time_limit_ms is.

sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

Looks like that will take 35 minutes to run (it's not a particularly fast tool).

I generated a 600MB database using sqlite-generate just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.

Those tables only had 4 columns each though.

You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

https://github.com/simonw/sqlite-generate is now ready to be used - see also https://pypi.org/project/sqlite-generate/

I started building that tool here: https://github.com/simonw/sqlite-generate

(I built a new cookiecutter template for that too, https://github.com/simonw/click-app)

I think this is a separate Click utility. I'm going to call it sqlite-generate.

https://datasette.readthedocs.io/en/latest/testing_plugins.html

Now split into four pages:

• https://datasette.readthedocs.io/en/latest/plugins.html
• https://datasette.readthedocs.io/en/latest/writing_plugins.html
• https://datasette.readthedocs.io/en/latest/plugin_hooks.html
• https://datasette.readthedocs.io/en/latest/internals.html

Still need to add the "Testing plugins" page, then I can close this issue.

I should also do #855, documenting the new datasette-plugin cookiecutter template. That can go in writing_plugins.rst.

Maybe add this to the plugins.rst page near the top:

.. toctree::
   :caption: See also
   :maxdepth: 1

   plugin_hooks
   internals

I'm going to break out the plugin hooks first in a single commit to make for a cleaner commit history (since that way git can hopefully detect that the content moved).

So the new plan is NOT to have a plugins/ folder, but instead have several top-level pages:

• Plugins (exists)
• Writing plugins
• Plugin hooks
• Testing plugins
• Internals for plugins (exists)

I'll write a little script which generates a 300MB SQLite file with a bunch of tables with lots of randomly generated rows in to help test this.

Having a tool like that which can generate larger databases with different gnarly performance characteristics will be useful for other performance work too.

This makes a lot of sense. I implemented the mechanism for the index page because I have my own instance of Datasette that was running slow, but it had a dozen database files attached to it. I've not run into this with a single giant database file but it absolutely makes sense that the same optimization would be necessary for the database page there too.

This is also relevant to #639, and may mean I can close that ticket in place of this one. I'm going to get this at least to a proof-of-concept stage first though.

I'm going to take this in a different direction.

I'm not happy with how metadata.(json|yaml) keeps growing new features. Rather than having a single plugin hook for all of metadata.json I'm going to split out the feature that shows actual real metadata for tables and databases - source, license etc - into its own plugin-powered mechanism.

So I'm going to close this ticket and spin up a new one for that.

I think I want the "Plugin hooks" page to be top-level, parallel to "Plugins" and "Internals for Plugins". It's the page of documentation refer to most often so I don't want to have to click down a hierarchy from the side navigation to find it.

The tutorial can start by showing how to use the new cookiecutter template from #642.

Maybe I should also refactor the plugin documentation, as contemplated in #687.

Shipped this today! https://github.com/simonw/datasette-plugin is a cookiecutter template for creating new plugins.

@psychemedia sorry I missed your comment before.

Niche Museums is definitely the best example of custom templates at the moment: https://github.com/simonw/museums/tree/master/templates

I want to comprehensively document the variables made available to custom templates before shipping Datasette 1.0 - just filed that as #857.