issue_comments

3,241 rows sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date)

issue

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
652106227 https://github.com/simonw/datasette/issues/876#issuecomment-652106227 https://api.github.com/repos/simonw/datasette/issues/876 MDEyOklzc3VlQ29tbWVudDY1MjEwNjIyNw== simonw 9599 2020-06-30T23:49:55Z 2020-06-30T23:50:04Z OWNER

Done: https://latest.datasette.io/-/patterns

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add log out link to the pattern portfolio 647879783
652105722 https://github.com/simonw/datasette/issues/879#issuecomment-652105722 https://api.github.com/repos/simonw/datasette/issues/879 MDEyOklzc3VlQ29tbWVudDY1MjEwNTcyMg== simonw 9599 2020-06-30T23:48:06Z 2020-06-30T23:48:06Z OWNER

Updated documentation: https://datasette.readthedocs.io/en/latest/pages.html

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page documentation still talks about hashes in URLs 648569227
652103895 https://github.com/simonw/datasette/issues/832#issuecomment-652103895 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MjEwMzg5NQ== simonw 9599 2020-06-30T23:41:22Z 2020-06-30T23:41:22Z OWNER

I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651999516 https://github.com/simonw/datasette/issues/832#issuecomment-651999516 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5OTUxNg== simonw 9599 2020-06-30T19:33:49Z 2020-06-30T21:34:59Z OWNER

Tests needed for this:

  • If a user has view table but NOT view database / view instance, can they view the table page?
  • If a user has view canned query but NOT view database / view instance, can they view the canned query page?
  • If a user has view database but NOT view instance, can they view the database page?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651995453 https://github.com/simonw/datasette/issues/832#issuecomment-651995453 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5NTQ1Mw== simonw 9599 2020-06-30T19:25:13Z 2020-06-30T19:25:26Z OWNER

I'm going to put the new check_permissions() method on BaseView as well. If I want that method to be available to plugins I can do so by turning that BaseView class into a documented API that plugins are encouraged to use themselves.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651994978 https://github.com/simonw/datasette/issues/832#issuecomment-651994978 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5NDk3OA== simonw 9599 2020-06-30T19:24:12Z 2020-06-30T19:24:12Z OWNER

Hah... but check_permissionis a method onBaseView`. Here are the various permission methods at the moment:

https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14

And on BaseView:

https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651993977 https://github.com/simonw/datasette/issues/832#issuecomment-651993977 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5Mzk3Nw== simonw 9599 2020-06-30T19:22:06Z 2020-06-30T19:22:06Z OWNER

permission_allowed is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on datasette..

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651993537 https://github.com/simonw/datasette/issues/832#issuecomment-651993537 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5MzUzNw== simonw 9599 2020-06-30T19:21:15Z 2020-06-30T19:21:15Z OWNER

I could rename permission_allowed() to check_permission() and have a complementary check_permissions() method.

This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary permission_allowed() alias which prints a deprecation warning to the console, then remove that at 1.0.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651992737 https://github.com/simonw/datasette/issues/832#issuecomment-651992737 https://api.github.com/repos/simonw/datasette/issues/832 MDEyOklzc3VlQ29tbWVudDY1MTk5MjczNw== simonw 9599 2020-06-30T19:19:33Z 2020-06-30T19:20:02Z OWNER

I already have this method on Datasette:

async def permission_allowed(self, actor, action, resource=None, default=False):

What would be a good method name that complements that and indicates "check a list of permissions in order"? Should it even run against the request or should you have to hand it request.actor?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having view-table permission but NOT view-database should still grant access to /db/table 636722501
651984989 https://github.com/simonw/datasette/issues/877#issuecomment-651984989 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MTk4NDk4OQ== simonw 9599 2020-06-30T19:03:25Z 2020-06-30T19:03:25Z OWNER

Relevant: #835

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
651984355 https://github.com/simonw/datasette/issues/877#issuecomment-651984355 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MTk4NDM1NQ== simonw 9599 2020-06-30T19:02:15Z 2020-06-30T19:02:15Z OWNER

https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf

Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form.

Sounds like regular CSRF protection to me.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
650340914 https://github.com/simonw/datasette/pull/868#issuecomment-650340914 https://api.github.com/repos/simonw/datasette/issues/868 MDEyOklzc3VlQ29tbWVudDY1MDM0MDkxNA== codecov[bot] 22429695 2020-06-26T18:53:02Z 2020-06-30T03:51:22Z NONE

Codecov Report

Merging #868 into master will increase coverage by 0.11%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #868      +/-   ##
==========================================
+ Coverage   83.31%   83.42%   +0.11%     
==========================================
  Files          27       27              
  Lines        3595     3614      +19     
==========================================
+ Hits         2995     3015      +20     
+ Misses        600      599       -1     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/utils/__init__.py</td> <td>93.93% <0.00%> (+0.05%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/app.py</td> <td>96.47% <0.00%> (+0.19%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/views/special.py</td> <td>81.17% <0.00%> (+3.39%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8a5f81...ef837b3. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
initial windows ci setup 646448486
651302221 https://github.com/simonw/datasette/issues/805#issuecomment-651302221 https://api.github.com/repos/simonw/datasette/issues/805 MDEyOklzc3VlQ29tbWVudDY1MTMwMjIyMQ== simonw 9599 2020-06-29T19:02:45Z 2020-06-29T19:05:26Z OWNER

No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way.

So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good.

Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries live demo on Glitch 632724154
651301202 https://github.com/simonw/datasette/issues/805#issuecomment-651301202 https://api.github.com/repos/simonw/datasette/issues/805 MDEyOklzc3VlQ29tbWVudDY1MTMwMTIwMg== simonw 9599 2020-06-29T19:00:37Z 2020-06-29T19:00:37Z OWNER

How about a blog? Pre-configured canned queries that are only available to "root", plus datasette-template-sql and default templates for the index page and blog entry pages.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries live demo on Glitch 632724154
651293559 https://github.com/simonw/datasette/issues/875#issuecomment-651293559 https://api.github.com/repos/simonw/datasette/issues/875 MDEyOklzc3VlQ29tbWVudDY1MTI5MzU1OQ== simonw 9599 2020-06-29T18:43:50Z 2020-06-29T18:43:50Z OWNER

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Logged in as: XXX - logout" navigation item 647103735
651203178 https://github.com/simonw/datasette/issues/873#issuecomment-651203178 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MTIwMzE3OA== simonw 9599 2020-06-29T15:44:38Z 2020-06-29T15:44:54Z OWNER

I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact -p 0 --root combination which isn't going to affect many people at all.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
651193594 https://github.com/simonw/datasette/issues/873#issuecomment-651193594 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MTE5MzU5NA== simonw 9599 2020-06-29T15:27:46Z 2020-06-29T15:27:46Z OWNER

Uninstalling datasette-debug-asgi caused the server to startup correctly again.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
651193131 https://github.com/simonw/datasette/issues/873#issuecomment-651193131 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MTE5MzEzMQ== simonw 9599 2020-06-29T15:27:00Z 2020-06-29T15:27:00Z OWNER

Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486

Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding lifespan="on" to the call to uvicorn.run().

When I did that here:

uvicorn.run(ds.app(), host=host, port=port, log_level="info", lifespan="on")

The server failed to start with this error:

INFO:     Started server process [68849]
INFO:     Waiting for application startup.
ERROR:    Exception in 'lifespan' protocol
Traceback (most recent call last):
  File ".../uvicorn/lifespan/on.py", line 48, in main
    await app(scope, self.receive, self.send)
  File ".../uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File ".../datasette_debug_asgi.py", line 9, in wrapped_app
    if scope["path"] == "/-/asgi-scope":
KeyError: 'path'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650910137 https://github.com/simonw/datasette/issues/873#issuecomment-650910137 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkxMDEzNw== simonw 9599 2020-06-29T05:16:32Z 2020-06-29T05:16:32Z OWNER

I'm not convinced that function is ever actually being called - I added a print() statement to it and it's not executing. I don't think the tests cover it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650909476 https://github.com/simonw/datasette/issues/873#issuecomment-650909476 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwOTQ3Ng== simonw 9599 2020-06-29T05:14:08Z 2020-06-29T05:14:08Z OWNER

I already have a AsgiLifespan class:
https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905

It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894

Could that startup function also output the --root login URL, if needed?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650909136 https://github.com/simonw/datasette/issues/873#issuecomment-650909136 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwOTEzNg== simonw 9599 2020-06-29T05:12:58Z 2020-06-29T05:12:58Z OWNER

On startup Datasette currently outputs:

INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.

So the ASGI lifespan protocol is almost certainly the right way to solve this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650908854 https://github.com/simonw/datasette/issues/873#issuecomment-650908854 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwODg1NA== simonw 9599 2020-06-29T05:12:04Z 2020-06-29T05:12:04Z OWNER

Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic --root link a second after the server starts up somehow.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650908534 https://github.com/simonw/datasette/issues/873#issuecomment-650908534 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwODUzNA== simonw 9599 2020-06-29T05:11:06Z 2020-06-29T05:11:06Z OWNER

Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650907323 https://github.com/simonw/datasette/issues/873#issuecomment-650907323 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwNzMyMw== simonw 9599 2020-06-29T05:07:16Z 2020-06-29T05:07:16Z OWNER

This line is interesting: is this a hook I can attach to somehow?

        await self.lifespan.startup()

From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650906533 https://github.com/simonw/datasette/issues/873#issuecomment-650906533 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwNjUzMw== simonw 9599 2020-06-29T05:04:44Z 2020-06-29T05:04:44Z OWNER

The challenge is... can we run our own custom code after that line has executed that has access to server and can hence access server.servers[0].sockets[0].getsockname()[1] to find the port?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650906318 https://github.com/simonw/datasette/issues/873#issuecomment-650906318 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwNjMxOA== simonw 9599 2020-06-29T05:04:04Z 2020-06-29T05:04:12Z OWNER

Within uvicorn it does this:

            if port == 0:
                port = server.sockets[0].getsockname()[1]

That server variable is later stashed here:

self.servers = [server]

Where self is the instance of class Server - which is the class that Uvicorn instantiates and calls .run() on when we do uvicorn.run() here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650905399 https://github.com/simonw/datasette/issues/873#issuecomment-650905399 https://api.github.com/repos/simonw/datasette/issues/873 MDEyOklzc3VlQ29tbWVudDY1MDkwNTM5OQ== simonw 9599 2020-06-29T05:01:03Z 2020-06-29T05:01:03Z OWNER

This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette -p 0 --root" gives the wrong URL 647095487
650899265 https://github.com/simonw/datasette/issues/875#issuecomment-650899265 https://api.github.com/repos/simonw/datasette/issues/875 MDEyOklzc3VlQ29tbWVudDY1MDg5OTI2NQ== simonw 9599 2020-06-29T04:34:32Z 2020-06-29T04:34:32Z OWNER

From https://github.com/simonw/datasette/issues/840#issuecomment-643454625

Another problem: what to display in the "you are logged in as", since we don't dictate an actor design.

I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins.

The default will look for the first available of the following keys:

* display
* name
* username
* login
* id
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Logged in as: XXX - logout" navigation item 647103735
650898808 https://github.com/simonw/datasette/issues/875#issuecomment-650898808 https://api.github.com/repos/simonw/datasette/issues/875 MDEyOklzc3VlQ29tbWVudDY1MDg5ODgwOA== simonw 9599 2020-06-29T04:32:31Z 2020-06-29T04:33:30Z OWNER

I could borrow the implementation for this from datasette-auth-github
https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html

{% extends "default:base.html" %}

{% block extra_head %}
<style type="text/css">
.hd .logout {
    float: right;
    text-align: right;
    padding-left: 1em;
}
</style>
{% endblock %}

{% block nav %}
    {{ super() }}
    {% if auth and auth.username %}
        <p class="logout">
            <strong>{{ auth.username }}</strong> &middot; <a href="/-/logout">Log out</a>
        </p>
    {% endif %}
{% endblock %}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Logged in as: XXX - logout" navigation item 647103735
650895874 https://github.com/simonw/datasette/issues/840#issuecomment-650895874 https://api.github.com/repos/simonw/datasette/issues/840 MDEyOklzc3VlQ29tbWVudDY1MDg5NTg3NA== simonw 9599 2020-06-29T04:18:59Z 2020-06-29T04:19:11Z OWNER

Now just need the "Logged in as: XXX <logout>" navigation item.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Log out mechanism for clearing ds_actor cookie 637966833
650891502 https://github.com/simonw/datasette/issues/840#issuecomment-650891502 https://api.github.com/repos/simonw/datasette/issues/840 MDEyOklzc3VlQ29tbWVudDY1MDg5MTUwMg== simonw 9599 2020-06-29T03:58:08Z 2020-06-29T03:58:08Z OWNER

Step one: a "logout" page at /-/logout - which shows you a single CSRF-protected "logout" button if you do a GET against it and logs you out if you do a POST against it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Log out mechanism for clearing ds_actor cookie 637966833
650891257 https://github.com/simonw/datasette/issues/805#issuecomment-650891257 https://api.github.com/repos/simonw/datasette/issues/805 MDEyOklzc3VlQ29tbWVudDY1MDg5MTI1Nw== simonw 9599 2020-06-29T03:56:48Z 2020-06-29T03:56:48Z OWNER

Using datasette-glitch and the new https://github.com/simonw/datasette-write - currently running on datasette==0.45a4 - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the /-/write interface.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries live demo on Glitch 632724154
650847013 https://github.com/simonw/datasette/issues/864#issuecomment-650847013 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY1MDg0NzAxMw== simonw 9599 2020-06-29T00:41:55Z 2020-06-29T00:41:55Z OWNER

To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
650846625 https://github.com/simonw/datasette/issues/864#issuecomment-650846625 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY1MDg0NjYyNQ== simonw 9599 2020-06-29T00:39:47Z 2020-06-29T00:39:47Z OWNER

I think the fix is to move the "show_messages" variable to here:

https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
650846473 https://github.com/simonw/datasette/issues/864#issuecomment-650846473 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY1MDg0NjQ3Mw== simonw 9599 2020-06-29T00:39:04Z 2020-06-29T00:39:04Z OWNER

Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends base.html. For example, this code in a plugin:

        return Response.html(
            await datasette.render_template(
                "write.html",
                {"databases": databases, "sql": request.args.get("sql") or ""},
                request=request,
            )
        )

This won't display messages. The reason is that the messages are made available to the template context in the BaseView.render() method here:
https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
650842514 https://github.com/simonw/datasette/issues/864#issuecomment-650842514 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY1MDg0MjUxNA== simonw 9599 2020-06-29T00:12:59Z 2020-06-29T00:12:59Z OWNER

I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with register_routes()) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
https://github.com/simonw/datasette/issues/870#issuecomment-650842381

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
650842381 https://github.com/simonw/datasette/issues/870#issuecomment-650842381 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDg0MjM4MQ== simonw 9599 2020-06-29T00:12:07Z 2020-06-29T00:12:07Z OWNER

I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with register_routes()) as part of Datasette 0.45 but it would be OK if it slipped to a later release.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650838972 https://github.com/simonw/datasette/issues/870#issuecomment-650838972 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgzODk3Mg== simonw 9599 2020-06-28T23:46:40Z 2020-06-28T23:46:40Z OWNER

I'm going to create the single Request() instance in the DatasetteRouter class - at the beginning of the route_path method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650838691 https://github.com/simonw/datasette/issues/870#issuecomment-650838691 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgzODY5MQ== simonw 9599 2020-06-28T23:44:12Z 2020-06-28T23:44:25Z OWNER

This code is interesting:

https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955

I want to change the signature of that return await view(new_scope, receive, send) method to instead take (request, send) - so I can have a single shared request object that's created just once per HTTP request.

The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared Request instance? Should its .scope be replaced with alternative scopes as it travels through the codebase?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650834666 https://github.com/simonw/datasette/issues/870#issuecomment-650834666 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgzNDY2Ng== simonw 9599 2020-06-28T23:07:19Z 2020-06-28T23:07:19Z OWNER

So now the problem is simpler: I need to get BaseView to a state where it can accept a shared request object and it can be used in conjunction with register_routes().

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650834251 https://github.com/simonw/datasette/issues/870#issuecomment-650834251 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgzNDI1MQ== simonw 9599 2020-06-28T23:03:28Z 2020-06-28T23:03:28Z OWNER

I'm going to ditch that AsgiView class too, by combining it into BaseView.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650820068 https://github.com/simonw/datasette/issues/870#issuecomment-650820068 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgyMDA2OA== simonw 9599 2020-06-28T20:52:09Z 2020-06-28T20:53:00Z OWNER

Maybe I could add a as_request_view method as an alternative to as_asgi:

https://github.com/simonw/datasette/blob/a8bcafc1775c8a8655b365ae22a3d64f6361c74a/datasette/utils/asgi.py#L150-L174

Or I could teach the Router to spot the dispatch_request method and call it directly.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650819895 https://github.com/simonw/datasette/issues/847#issuecomment-650819895 https://api.github.com/repos/simonw/datasette/issues/847 MDEyOklzc3VlQ29tbWVudDY1MDgxOTg5NQ== simonw 9599 2020-06-28T20:50:21Z 2020-06-28T20:50:21Z OWNER

I'm happy enough with https://codecov.io/gh/simonw/datasette that I'm not going to spend any more time on this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Take advantage of .coverage being a SQLite database 638259643
650818309 https://github.com/simonw/datasette/issues/870#issuecomment-650818309 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgxODMwOQ== simonw 9599 2020-06-28T20:36:28Z 2020-06-28T20:36:52Z OWNER

Since AsgiRouter is only used as the super-class of the DatasetteRouter class maybe I should get rid of AsgiRouter entirely - no point in having a Datasette-specific subclass of it if the parent class isn't ever used by anything else.

I could also rename it to just Router which is a nicer name than DatasetteRouter.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650818086 https://github.com/simonw/datasette/issues/870#issuecomment-650818086 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgxODA4Ng== simonw 9599 2020-06-28T20:34:33Z 2020-06-28T20:34:33Z OWNER

The key to all of this may be the DatasetteRouter class. It deals with scope right now but if it internally dealt with request that could be enough to fix #864 by adding logic needed by the .add_message() mechanism.

https://github.com/simonw/datasette/blob/0991ea75cc7b265389aa8362414a305ba532d31a/datasette/app.py#L904-L938

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650815278 https://github.com/simonw/datasette/issues/870#issuecomment-650815278 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDgxNTI3OA== simonw 9599 2020-06-28T20:09:07Z 2020-06-28T20:11:21Z OWNER

There's a lot of complex logic in the DataView class, which handles conditionally returning content as .json or as HTML or as .csv.

That view subclasses AsgiView which is itself request-aware, so maybe I don't need to reconsider how those classes work - just figure out how to hook them up with register_routes.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
650812444 https://github.com/simonw/datasette/issues/871#issuecomment-650812444 https://api.github.com/repos/simonw/datasette/issues/871 MDEyOklzc3VlQ29tbWVudDY1MDgxMjQ0NA== simonw 9599 2020-06-28T19:43:27Z 2020-06-28T19:43:27Z OWNER

Currently:

_timestamp_epoch

The number of seconds since the Unix epoch.

_timestamp_date_utc

The date in UTC, e.g. 2020-06-01

_timestamp_datetime_utc

The ISO 8601 datetime in UTC, e.g. 2020-06-24T18:01:07Z

I'm going to rename them to:

  • _now_epoch
  • _now_date_utc
  • _now_datetime_utc
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename the _timestamp magic parameters to _now 646840273
650811919 https://github.com/simonw/datasette/issues/834#issuecomment-650811919 https://api.github.com/repos/simonw/datasette/issues/834 MDEyOklzc3VlQ29tbWVudDY1MDgxMTkxOQ== simonw 9599 2020-06-28T19:38:50Z 2020-06-28T19:38:50Z OWNER

I have two plugins in progress that use this hook now:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
startup() plugin hook 637342551
650784162 https://github.com/simonw/datasette/issues/805#issuecomment-650784162 https://api.github.com/repos/simonw/datasette/issues/805 MDEyOklzc3VlQ29tbWVudDY1MDc4NDE2Mg== simonw 9599 2020-06-28T15:48:32Z 2020-06-28T15:48:32Z OWNER

https://github.com/simonw/datasette-glitch is my new plugin that outputs the root login link on Glitch when the server starts.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries live demo on Glitch 632724154
650696054 https://github.com/simonw/datasette/issues/870#issuecomment-650696054 https://api.github.com/repos/simonw/datasette/issues/870 MDEyOklzc3VlQ29tbWVudDY1MDY5NjA1NA== simonw 9599 2020-06-28T04:52:41Z 2020-06-28T04:52:41Z OWNER

This would be a lot easier if I had extracted out the hash logic to a plugin, see #745.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor default views to use register_routes 646737558
643657067 https://github.com/simonw/datasette/issues/834#issuecomment-643657067 https://api.github.com/repos/simonw/datasette/issues/834 MDEyOklzc3VlQ29tbWVudDY0MzY1NzA2Nw== simonw 9599 2020-06-13T17:59:42Z 2020-06-28T04:01:52Z OWNER

Documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#startup-datasette

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
startup() plugin hook 637342551
650684635 https://github.com/simonw/datasette/issues/842#issuecomment-650684635 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDY4NDYzNQ== simonw 9599 2020-06-28T03:30:31Z 2020-06-28T03:30:31Z OWNER

Live demo: https://latest.datasette.io/fixtures/magic_parameters

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650681496 https://github.com/simonw/datasette/issues/805#issuecomment-650681496 https://api.github.com/repos/simonw/datasette/issues/805 MDEyOklzc3VlQ29tbWVudDY1MDY4MTQ5Ng== simonw 9599 2020-06-28T03:11:51Z 2020-06-28T03:11:51Z OWNER

I can use magic parameters from #842 in this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Writable canned queries live demo on Glitch 632724154
650679100 https://github.com/simonw/datasette/issues/842#issuecomment-650679100 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDY3OTEwMA== simonw 9599 2020-06-28T03:00:44Z 2020-06-28T03:00:44Z OWNER

I'm going to add some canned queries to the metadata.json used by the live demo that illustrate this feature.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650678951 https://github.com/simonw/datasette/issues/842#issuecomment-650678951 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDY3ODk1MQ== simonw 9599 2020-06-28T02:59:52Z 2020-06-28T02:59:52Z OWNER

Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters

Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650600176 https://github.com/simonw/datasette/pull/869#issuecomment-650600176 https://api.github.com/repos/simonw/datasette/issues/869 MDEyOklzc3VlQ29tbWVudDY1MDYwMDE3Ng== codecov[bot] 22429695 2020-06-27T18:41:31Z 2020-06-28T02:54:21Z NONE

Codecov Report

Merging #869 into master will increase coverage by 0.23%.
The diff coverage is 90.62%.

@@            Coverage Diff             @@
##           master     #869      +/-   ##
==========================================
+ Coverage   82.99%   83.23%   +0.23%     
==========================================
  Files          26       27       +1     
  Lines        3547     3609      +62     
==========================================
+ Hits         2944     3004      +60     
- Misses        603      605       +2     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/plugins.py</td> <td>82.35% <ø> (ø)</td> <td></td> </tr> <tr> <td>datasette/views/database.py</td> <td>96.45% <86.36%> (-1.88%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/default_magic_parameters.py</td> <td>91.17% <91.17%> (ø)</td> <td></td> </tr> <tr> <td>datasette/app.py</td> <td>96.07% <100.00%> (+0.81%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/hookspecs.py</td> <td>100.00% <100.00%> (ø)</td> <td></td> </tr> <tr> <td>datasette/utils/__init__.py</td> <td>93.87% <100.00%> (+0.02%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1bb33da...9e693a7. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 646734280
650648434 https://github.com/simonw/datasette/issues/842#issuecomment-650648434 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDY0ODQzNA== simonw 9599 2020-06-27T23:27:35Z 2020-06-27T23:37:38Z OWNER

I'm going to rename _request_X to _header_X as that better reflects what it now does.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650600606 https://github.com/simonw/datasette/pull/868#issuecomment-650600606 https://api.github.com/repos/simonw/datasette/issues/868 MDEyOklzc3VlQ29tbWVudDY1MDYwMDYwNg== simonw 9599 2020-06-27T18:44:28Z 2020-06-27T18:44:28Z OWNER

This is really exciting! Thanks so much for looking into this.

I'm interested in moving CI for this repo over to GitHub Actions, so I'd be fine with you getting this to work as an Action rather than through Travis. If you can get it working in Travis though I'll happily land that and figure out how to convert that to GitHub Actions later on.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
initial windows ci setup 646448486
650598710 https://github.com/simonw/datasette/issues/835#issuecomment-650598710 https://api.github.com/repos/simonw/datasette/issues/835 MDEyOklzc3VlQ29tbWVudDY1MDU5ODcxMA== simonw 9599 2020-06-27T18:32:22Z 2020-06-27T18:32:22Z OWNER

Skipping CSRF on Authorization: Bearer xxx headers also makes sense for JWT applications, which tend to send JWTs using that form of header.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for skipping CSRF checks on API posts 637363686
650593122 https://github.com/simonw/datasette/issues/842#issuecomment-650593122 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDU5MzEyMg== simonw 9599 2020-06-27T18:03:02Z 2020-06-27T18:03:10Z OWNER

Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter _cookie_ds_actor to be resolved and returned as JSON data that the attacker can see.

This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch.

https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650458857 https://github.com/simonw/datasette/issues/842#issuecomment-650458857 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDQ1ODg1Nw== simonw 9599 2020-06-27T00:11:04Z 2020-06-27T00:11:04Z OWNER

Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter _cookie_ds_actor to be resolved and returned as JSON data that the attacker can see.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650455793 https://github.com/simonw/datasette/issues/842#issuecomment-650455793 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDQ1NTc5Mw== simonw 9599 2020-06-26T23:57:30Z 2020-06-27T00:00:16Z OWNER

Maybe I should ship a default _scope_headers_... parameter instead, which reads from a dictionary of scope["headers"] - https://asgi-scope.now.sh/ shows what those look like.

{'client': ('148.64.98.14', 0),
 'headers': [[b'host', b'asgi-scope.now.sh'],
             [b'x-forwarded-for', b'148.64.98.14'],
             [b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'],
             [b'x-forwarded-host', b'asgi-scope.now.sh'],
             [b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'],
             [b'x-real-ip', b'148.64.98.14'],
             [b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'],
             [b'upgrade-insecure-requests', b'1'],
             [b'x-vercel-trace', b'sfo1'],
             [b'x-forwarded-proto', b'https'],
             [b'accept-language', b'en-US,en;q=0.5'],
             [b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko'
              b'/20100101 Firefox/77.0'],
             [b'x-vercel-forwarded-for', b'148.64.98.14'],
             [b'accept-encoding', b'gzip, deflate, br'],
             [b'dnt', b'1'],
             [b'te', b'trailers']],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/',
 'query_string': b'',
 'raw_path': b'/',
 'root_path': '',
 'scheme': 'https',
 'server': ('asgi-scope.now.sh', 80),
 'type': 'http'}

I'm going to have _request_X actually mean "find the first value for X in scope["headers"]" - with underscores converted to hyphens.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
650455353 https://github.com/simonw/datasette/issues/842#issuecomment-650455353 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY1MDQ1NTM1Mw== simonw 9599 2020-06-26T23:55:40Z 2020-06-26T23:55:40Z OWNER

_request_ip is actually quite hard to implement - should it take into account things like the x-forwarded-for header?

It probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment.

As such I think I'll leave that for a plugin.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
649931714 https://github.com/simonw/datasette/issues/867#issuecomment-649931714 https://api.github.com/repos/simonw/datasette/issues/867 MDEyOklzc3VlQ29tbWVudDY0OTkzMTcxNA== simonw 9599 2020-06-26T03:12:51Z 2020-06-26T03:12:51Z OWNER

Here's the relevant code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/datasette/app.py#L1057-L1070
And the relevant test code:
https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/test_plugins.py#L567-L573

https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/plugins/my_plugin.py#L162-L196

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
register_routes() should support non-async view functions too 645975649
649908756 https://github.com/simonw/datasette/issues/849#issuecomment-649908756 https://api.github.com/repos/simonw/datasette/issues/849 MDEyOklzc3VlQ29tbWVudDY0OTkwODc1Ng== simonw 9599 2020-06-26T02:09:09Z 2020-06-26T02:09:09Z OWNER

I mentioned this issue here: https://simonwillison.net/2020/Jun/26/weeknotes-plugins-sqlite-generate/

Repositories created by following the REAME in https://github.com/simonw/datasette-template and https://github.com/simonw/click-app have a main branch instead of master so I have a few examples live now. https://github.com/simonw/datasette-saved-queries is one example.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename master branch to main 639072811
649014757 https://github.com/simonw/datasette/issues/842#issuecomment-649014757 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0OTAxNDc1Nw== simonw 9599 2020-06-24T19:15:46Z 2020-06-24T19:31:52Z OWNER

I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646271834 https://github.com/simonw/datasette/issues/842#issuecomment-646271834 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjI3MTgzNA== simonw 9599 2020-06-18T19:49:41Z 2020-06-24T18:49:22Z OWNER

But then what kind of magic parameters might plugins want to add?

Here's a crazy idea: _scrapedcontent_url - it would look for the url column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function scrapedcontent() would need to optionally be sent the full row dictionary being inserted too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
646270702 https://github.com/simonw/datasette/issues/842#issuecomment-646270702 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0NjI3MDcwMg== simonw 9599 2020-06-18T19:47:19Z 2020-06-24T18:48:48Z OWNER

Brainstorming more potential magic parameters:

  • _actor_id
  • _actor_name
  • _request_ip
  • _request_user_agent
  • _cookie_cookiename
  • _signedcookie_cookiename - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, _signedcookie_cookiename_mynamespace. Might not need special signed cookie support since actor is already usually from a signed cookie.
  • _timestamp_unix (not happy with these names yet)
  • _timestamp_localtime
  • _timestamp_datetime
  • _timestamp_utc
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
649000075 https://github.com/simonw/datasette/issues/842#issuecomment-649000075 https://api.github.com/repos/simonw/datasette/issues/842 MDEyOklzc3VlQ29tbWVudDY0OTAwMDA3NQ== simonw 9599 2020-06-24T18:46:36Z 2020-06-24T18:47:37Z OWNER

Another magic parameter that would be useful would be _random. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one.

Maybe _random_chars_128 to create a 128 character long random string (using os.urandom(64).hex()).

This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately _random_chars:128 wouldn't work because these parameters are used in a SQLite query where : has special meaning: insert into blah (secret) values (:_random_chars:128) wouldn't make sense.

Actually this is already supported by the proposed design - _random_chars_128 would become random("chars_128") so the random() function could split off the 128 itself.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Magic parameters for canned queries 638212085
648998264 https://github.com/simonw/datasette/issues/865#issuecomment-648998264 https://api.github.com/repos/simonw/datasette/issues/865 MDEyOklzc3VlQ29tbWVudDY0ODk5ODI2NA== simonw 9599 2020-06-24T18:43:02Z 2020-06-24T18:43:02Z OWNER

Thanks for the bug report. Yes I think #838 may be the same issue. Will investigate.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url doesn't seem to work when adding criteria and clicking "apply" 644582921
648997857 https://github.com/simonw/datasette/issues/858#issuecomment-648997857 https://api.github.com/repos/simonw/datasette/issues/858 MDEyOklzc3VlQ29tbWVudDY0ODk5Nzg1Nw== simonw 9599 2020-06-24T18:42:10Z 2020-06-24T18:42:10Z OWNER

I really need to get myself a Windows 10 development environment working so I can dig into this kind of bug properly. I have a gaming PC lying around that I could re-task for that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
publish heroku does not work on Windows 10 642388564
648818707 https://github.com/simonw/datasette/pull/866#issuecomment-648818707 https://api.github.com/repos/simonw/datasette/issues/866 MDEyOklzc3VlQ29tbWVudDY0ODgxODcwNw== codecov[bot] 22429695 2020-06-24T13:26:14Z 2020-06-24T13:26:14Z NONE

Codecov Report

Merging #866 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #866   +/-   ##
=======================================
  Coverage   82.99%   82.99%           
=======================================
  Files          26       26           
  Lines        3547     3547           
=======================================
  Hits         2944     2944           
  Misses        603      603           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a5b7d3...fb64dda. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update pytest-asyncio requirement from <0.13,>=0.10 to >=0.10,<0.15 644610729
648800356 https://github.com/simonw/datasette/issues/838#issuecomment-648800356 https://api.github.com/repos/simonw/datasette/issues/838 MDEyOklzc3VlQ29tbWVudDY0ODgwMDM1Ng== tballison 6739646 2020-06-24T12:51:48Z 2020-06-24T12:51:48Z NONE

But also want to say thanks for a great tool

+1!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Incorrect URLs when served behind a proxy with base_url set 637395097
648799963 https://github.com/simonw/datasette/issues/865#issuecomment-648799963 https://api.github.com/repos/simonw/datasette/issues/865 MDEyOklzc3VlQ29tbWVudDY0ODc5OTk2Mw== tballison 6739646 2020-06-24T12:51:01Z 2020-06-24T12:51:01Z NONE

This seems to be a duplicate of: https://github.com/simonw/datasette/issues/838

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url doesn't seem to work when adding criteria and clicking "apply" 644582921
648669523 https://github.com/simonw/datasette/issues/859#issuecomment-648669523 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0ODY2OTUyMw== abdusco 3243482 2020-06-24T08:13:23Z 2020-06-24T10:30:36Z CONTRIBUTOR

I tried setting cache_size_kb=0 then cache_size_kb=100000, still getting this behavior. I even changed Database::table_counts and lowered time limit to 1

table_count = (
    await self.execute(
        "select count(*) from [{}]".format(table),
        custom_time_limit=1,
    )
).rows[0][0]
counts[table] = table_count

I feel like 10 seconds is a magic number, like a processing timeout and datasette gives up and returns the page.
Index page loads instantly, table page, query page, as well. But when I return to database page after some time, it loads in 10s.

EDIT:

It's always like 10 + 0.3s, like 10s wait and timeout then 300ms to render the page

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
648580556 https://github.com/simonw/datasette/issues/864#issuecomment-648580556 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY0ODU4MDU1Ng== simonw 9599 2020-06-24T04:40:49Z 2020-06-24T04:40:49Z OWNER

The ideal fix here would be to rework my BaseView subclass mechanism to work with register_routes() so that those views don't have any special privileges above plugin-provided views.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
648580236 https://github.com/simonw/datasette/issues/864#issuecomment-648580236 https://api.github.com/repos/simonw/datasette/issues/864 MDEyOklzc3VlQ29tbWVudDY0ODU4MDIzNg== simonw 9599 2020-06-24T04:39:39Z 2020-06-24T04:39:39Z OWNER

Urgh, fixing this is going to be a bit of a pain.

Here's where I added that custom dispatch_request() method - it was to implement flash messaging in #790: https://github.com/simonw/datasette/blame/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/views/base.py#L85

If I want this to be made available to register_routes() views as well, I'm going to have to move the logic somewhere else. In particular I need to make sure that the request object is created once and used throughout the whole request cycle.

Currently register_routes() view functions get their own separate request object which is created here:

https://github.com/simonw/datasette/blob/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/app.py#L1057-L1068

So I'm going to have to refactor this quite a bit to get that shared request object which can be passed both to register_routes views and to my various BaseView subclasses.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.add_message() doesn't work inside plugins 644309017
648442511 https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648442511 https://api.github.com/repos/simonw/sqlite-utils/issues/117 MDEyOklzc3VlQ29tbWVudDY0ODQ0MjUxMQ== simonw 9599 2020-06-23T21:39:41Z 2020-06-23T21:39:41Z OWNER

So there are two sides to supporting this:

  • Being able to sensibly introspect composite foreign keys
  • Being able to define composite foreign keys when creating a table
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for compound (composite) foreign keys 644161221
648440634 https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440634 https://api.github.com/repos/simonw/sqlite-utils/issues/117 MDEyOklzc3VlQ29tbWVudDY0ODQ0MDYzNA== simonw 9599 2020-06-23T21:35:16Z 2020-06-23T21:35:16Z OWNER

Relevant discussion: https://github.com/simonw/sqlite-generate/issues/8#issuecomment-648438056

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for compound (composite) foreign keys 644161221
648440525 https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440525 https://api.github.com/repos/simonw/sqlite-utils/issues/117 MDEyOklzc3VlQ29tbWVudDY0ODQ0MDUyNQ== simonw 9599 2020-06-23T21:35:01Z 2020-06-23T21:35:01Z OWNER

Here's what's missing:

In [11]: db.conn.execute('PRAGMA foreign_key_list(song)').fetchall()                                                                       
Out[11]: 
[(0,
  0,
  'album',
  'songartist',
  'albumartist',
  'NO ACTION',
  'NO ACTION',
  'NONE'),
 (0, 1, 'album', 'songalbum', 'albumname', 'NO ACTION', 'NO ACTION', 'NONE')]

Compare with this code here:
https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/sqlite_utils/db.py#L563-L579

The first two columns returned by PRAGMA foreign_key_list(table) are id and seq - these show when two foreign key records are part of the same compound foreign key. sqlite-utils entirely ignores those at the moment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for compound (composite) foreign keys 644161221
648434885 https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648434885 https://api.github.com/repos/simonw/sqlite-utils/issues/116 MDEyOklzc3VlQ29tbWVudDY0ODQzNDg4NQ== simonw 9599 2020-06-23T21:21:33Z 2020-06-23T21:21:33Z OWNER

New docs: https://github.com/simonw/sqlite-utils/blob/2.10.1/docs/python-api.rst#introspection

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation for table.pks introspection property 644122661
648403834 https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648403834 https://api.github.com/repos/simonw/sqlite-utils/issues/116 MDEyOklzc3VlQ29tbWVudDY0ODQwMzgzNA== simonw 9599 2020-06-23T20:36:29Z 2020-06-23T20:36:29Z OWNER

Should go in this section https://sqlite-utils.readthedocs.io/en/stable/python-api.html#introspection - under .columns_dict and before .foreign_keys.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation for table.pks introspection property 644122661
648296323 https://github.com/simonw/datasette/issues/694#issuecomment-648296323 https://api.github.com/repos/simonw/datasette/issues/694 MDEyOklzc3VlQ29tbWVudDY0ODI5NjMyMw== kwladyka 3903726 2020-06-23T17:10:51Z 2020-06-23T17:10:51Z NONE

@simonw

Did you find the reason? I had similar situation and I check this on millions ways. I am sure app doesn't consume such memory.

I was trying the app with:
docker run --rm -it -p 80:80 -m 128M foo

I was watching app with docker stats. Even limited memory by CMD ["java", "-Xms60M", "-Xmx60M", "-jar", "api.jar"].
Checked memory usage by app in code and print bash commands. The app definitely doesn't use this memory. Also doesn't write files.

Only one solution is to change memory to 512M.

It is definitely something wrong with cloud run.

I even did special app for testing this. It looks like when I cross very small amount of code / memory / app size in random when, then memory needs grow +hundreds. Nothing make sense here. Especially it works everywhere expect cloud run.

Please let me know if you discovered something more.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish cloudrun --memory option 576582604
648234787 https://github.com/simonw/datasette/issues/859#issuecomment-648234787 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0ODIzNDc4Nw== simonw 9599 2020-06-23T15:22:51Z 2020-06-23T15:22:51Z OWNER

I wonder if this is a SQLite caching issue then?

Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
648232645 https://github.com/simonw/datasette/issues/859#issuecomment-648232645 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0ODIzMjY0NQ== abdusco 3243482 2020-06-23T15:19:53Z 2020-06-23T15:19:53Z CONTRIBUTOR

The issue seems to appear sporadically, like when I return to database page after a while, during which some records have been added to the database.

I've just visited database, page first visit took ~10s, consecutive visits took 0.3s.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
648163272 https://github.com/simonw/datasette/issues/859#issuecomment-648163272 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0ODE2MzI3Mg== simonw 9599 2020-06-23T13:52:23Z 2020-06-23T13:52:23Z OWNER

I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030

I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647925594 https://github.com/simonw/datasette/issues/859#issuecomment-647925594 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkyNTU5NA== abdusco 3243482 2020-06-23T05:55:21Z 2020-06-23T06:28:29Z CONTRIBUTOR

Hmm, not seeing the problem now.
I've removed the commented out sections in database.py and restarted the process. Database page now loads in <250ms.

I have couple of workers that check some pages regularly and scrape new content and save to the DB. Could it be that datasette tries to recount tables every time database size changes? Normally it keeps a count cache, but as DB gets updated so often (new content every 5 min or so) it's practically recounting every time I go to the database page?

EDIT:
It turns out it doesn't hold cache with mutable databases.

I'll update the issue with more findings and a better way to reproduce the problem if I encounter it again.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647936117 https://github.com/simonw/datasette/issues/859#issuecomment-647936117 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkzNjExNw== abdusco 3243482 2020-06-23T06:25:17Z 2020-06-23T06:25:17Z CONTRIBUTOR

sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

Looks like that will take 35 minutes to run (it's not a particularly fast tool).

Try chunking write operations into batches every 1000 records or so.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647935300 https://github.com/simonw/datasette/issues/859#issuecomment-647935300 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkzNTMwMA== abdusco 3243482 2020-06-23T06:23:01Z 2020-06-23T06:23:01Z CONTRIBUTOR

You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

Ah that was a typo, I meant 50k.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647923666 https://github.com/simonw/datasette/issues/859#issuecomment-647923666 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkyMzY2Ng== abdusco 3243482 2020-06-23T05:49:31Z 2020-06-23T05:49:31Z CONTRIBUTOR

I think I should mention that having FTS on all tables mean I have 5 visible, 25 hidden (FTS) tables displayed on database page.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647922203 https://github.com/simonw/datasette/issues/859#issuecomment-647922203 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkyMjIwMw== abdusco 3243482 2020-06-23T05:44:58Z 2020-06-23T05:44:58Z CONTRIBUTOR

I'm seeing the problem on database page. Index page and table page runs quite fast.

  • Tables have <10 columns (id, url, title, body_html, date, author, meta (for keeping unstructured json)). I've added index on date columns (using sqlite-utils) in addition to the index present on id columns.
  • All tables have FTS enabled on text and varchar columns (title, body_html vs) to speed up searching.
  • There are couple of tables related with foreign keys (think a thread in a forum and posts in that thread, related with thread_id)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647894903 https://github.com/simonw/datasette/issues/859#issuecomment-647894903 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0Nzg5NDkwMw== simonw 9599 2020-06-23T04:07:59Z 2020-06-23T04:07:59Z OWNER

Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys

If it's the table page then the problem may well be #862.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647893140 https://github.com/simonw/datasette/issues/596#issuecomment-647893140 https://api.github.com/repos/simonw/datasette/issues/596 MDEyOklzc3VlQ29tbWVudDY0Nzg5MzE0MA== simonw 9599 2020-06-23T03:59:51Z 2020-06-23T03:59:51Z OWNER

Related: #862 - a time limit on the total time spent considering suggested facets for a table.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle really wide tables better 507454958
647892930 https://github.com/simonw/datasette/issues/862#issuecomment-647892930 https://api.github.com/repos/simonw/datasette/issues/862 MDEyOklzc3VlQ29tbWVudDY0Nzg5MjkzMA== simonw 9599 2020-06-23T03:58:48Z 2020-06-23T03:58:48Z OWNER

Should this be controlled be a separate configuration setting? I'm inclined to say no - I think instead I'll set the limit to be 10 * whatever facet_suggest_time_limit_ms is.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Set an upper limit on total facet suggestion time for a page 643510821
647890619 https://github.com/simonw/datasette/issues/859#issuecomment-647890619 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0Nzg5MDYxOQ== simonw 9599 2020-06-23T03:48:21Z 2020-06-23T03:48:21Z OWNER
sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

Looks like that will take 35 minutes to run (it's not a particularly fast tool).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647890378 https://github.com/simonw/datasette/issues/859#issuecomment-647890378 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0Nzg5MDM3OA== simonw 9599 2020-06-23T03:47:19Z 2020-06-23T03:47:19Z OWNER

I generated a 600MB database using sqlite-generate just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms.

Those tables only had 4 columns each though.

You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647889674 https://github.com/simonw/datasette/issues/861#issuecomment-647889674 https://api.github.com/repos/simonw/datasette/issues/861 MDEyOklzc3VlQ29tbWVudDY0Nzg4OTY3NA== simonw 9599 2020-06-23T03:44:17Z 2020-06-23T03:44:17Z OWNER

https://github.com/simonw/sqlite-generate is now ready to be used - see also https://pypi.org/project/sqlite-generate/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Script to generate larger SQLite test files 642652808
647822757 https://github.com/simonw/datasette/issues/861#issuecomment-647822757 https://api.github.com/repos/simonw/datasette/issues/861 MDEyOklzc3VlQ29tbWVudDY0NzgyMjc1Nw== simonw 9599 2020-06-22T23:40:43Z 2020-06-22T23:40:43Z OWNER

I started building that tool here: https://github.com/simonw/sqlite-generate

(I built a new cookiecutter template for that too, https://github.com/simonw/click-app)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Script to generate larger SQLite test files 642652808
647803394 https://github.com/simonw/datasette/issues/838#issuecomment-647803394 https://api.github.com/repos/simonw/datasette/issues/838 MDEyOklzc3VlQ29tbWVudDY0NzgwMzM5NA== ChristopherWilks 6289012 2020-06-22T22:36:34Z 2020-06-22T22:36:34Z NONE

I also am seeing the same issue with an Apache setup (same even w/o ProxyPassReverse, though I typically use it as @tsibley stated).

But also want to say thanks for a great tool (this issue not withstanding)!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Incorrect URLs when served behind a proxy with base_url set 637395097
647266979 https://github.com/simonw/datasette/issues/861#issuecomment-647266979 https://api.github.com/repos/simonw/datasette/issues/861 MDEyOklzc3VlQ29tbWVudDY0NzI2Njk3OQ== simonw 9599 2020-06-22T04:26:25Z 2020-06-22T04:26:25Z OWNER

I think this is a separate Click utility. I'm going to call it sqlite-generate.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Script to generate larger SQLite test files 642652808

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 775.645ms · About: github-to-sqlite