github

Custom SQL query returning 101 rows (hide)

html_urlissue_urlidnode_idusercreated_atupdated_atauthor_associationbodyreactionsissue
https://github.com/simonw/datasette/issues/876#issuecomment-652106227 https://api.github.com/repos/simonw/datasette/issues/876 652106227 MDEyOklzc3VlQ29tbWVudDY1MjEwNjIyNw== 9599 2020-06-30T23:49:55Z 2020-06-30T23:50:04Z OWNER Done: https://latest.datasette.io/-/patterns <img width="1004" alt="Datasette__Pattern_Portfolio" src="https://user-images.githubusercontent.com/9599/86187862-b680cb00-baf1-11ea-8905-8c257dd67665.png">
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647879783
https://github.com/simonw/datasette/issues/879#issuecomment-652105722 https://api.github.com/repos/simonw/datasette/issues/879 652105722 MDEyOklzc3VlQ29tbWVudDY1MjEwNTcyMg== 9599 2020-06-30T23:48:06Z 2020-06-30T23:48:06Z OWNER Updated documentation: https://datasette.readthedocs.io/en/latest/pages.html
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
648569227
https://github.com/simonw/datasette/issues/832#issuecomment-652103895 https://api.github.com/repos/simonw/datasette/issues/832 652103895 MDEyOklzc3VlQ29tbWVudDY1MjEwMzg5NQ== 9599 2020-06-30T23:41:22Z 2020-06-30T23:41:22Z OWNER I don't think this needs any additional documentation - the new behaviour matches how the permissions are documented here: https://datasette.readthedocs.io/en/0.44/authentication.html#built-in-permissions
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
636722501
https://github.com/simonw/datasette/issues/832#issuecomment-651999516 https://api.github.com/repos/simonw/datasette/issues/832 651999516 MDEyOklzc3VlQ29tbWVudDY1MTk5OTUxNg== 9599 2020-06-30T19:33:49Z 2020-06-30T21:34:59Z OWNER Tests needed for this: - If a user has view table but NOT view database / view instance, can they view the table page? - If a user has view canned query but NOT view database / view instance, can they view the canned query page? - If a user has view database but NOT view instance, can they view the database page?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
636722501
https://github.com/simonw/datasette/issues/832#issuecomment-651995453 https://api.github.com/repos/simonw/datasette/issues/832 651995453 MDEyOklzc3VlQ29tbWVudDY1MTk5NTQ1Mw== 9599 2020-06-30T19:25:13Z 2020-06-30T19:25:26Z OWNER I'm going to put the new `check_permissions()` method on `BaseView` as well. If I want that method to be available to plugins I can do so by turning that `BaseView` class into a documented API that plugins are encouraged to use themselves.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
636722501
https://github.com/simonw/datasette/issues/832#issuecomment-651994978 https://api.github.com/repos/simonw/datasette/issues/832 651994978 MDEyOklzc3VlQ29tbWVudDY1MTk5NDk3OA== 9599 2020-06-30T19:24:12Z 2020-06-30T19:24:12Z OWNER Hah... but check_permission` is a method on `BaseView`. Here are the various permission methods at the moment: https://github.com/simonw/datasette/blob/6c2634583627bfab750c115cb13850252821d637/datasette/default_permissions.py#L5-L14 And on BaseView: https://github.com/simonw/datasette/blob/a8a5f813722f72703a7aae41135ccc40635cc02f/datasette/views/base.py#L65-L70
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
636722501
https://github.com/simonw/datasette/issues/832#issuecomment-651993977 https://api.github.com/repos/simonw/datasette/issues/832 651993977 MDEyOklzc3VlQ29tbWVudDY1MTk5Mzk3Nw== 9599 2020-06-30T19:22:06Z 2020-06-30T19:22:06Z OWNER `permission_allowed` is already the name of the pugin hook. It's actually a bit confusing that it's also the name of a method on `datasette.`.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
636722501
https://github.com/simonw/datasette/issues/832#issuecomment-651993537 https://api.github.com/repos/simonw/datasette/issues/832 651993537 MDEyOklzc3VlQ29tbWVudDY1MTk5MzUzNw== 9599 2020-06-30T19:21:15Z 2020-06-30T19:21:15Z OWNER I could rename `permission_allowed()` to `check_permission()` and have a complementary `check_permissions()` method. This is a breaking change but we're pre-1.0 so I think that's OK. I could even set up a temporary `permission_allowed()` alias which prints a deprecation warning to the console, then remove that at 1.0.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
636722501
https://github.com/simonw/datasette/issues/832#issuecomment-651992737 https://api.github.com/repos/simonw/datasette/issues/832 651992737 MDEyOklzc3VlQ29tbWVudDY1MTk5MjczNw== 9599 2020-06-30T19:19:33Z 2020-06-30T19:20:02Z OWNER I already have this method on Datasette: ```python async def permission_allowed(self, actor, action, resource=None, default=False): ``` What would be a good method name that complements that and indicates "check a list of permissions in order"? Should it even run against the request or should you have to hand it `request.actor`?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
636722501
https://github.com/simonw/datasette/issues/877#issuecomment-651984989 https://api.github.com/repos/simonw/datasette/issues/877 651984989 MDEyOklzc3VlQ29tbWVudDY1MTk4NDk4OQ== 9599 2020-06-30T19:03:25Z 2020-06-30T19:03:25Z OWNER Relevant: #835
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
648421105
https://github.com/simonw/datasette/issues/877#issuecomment-651984355 https://api.github.com/repos/simonw/datasette/issues/877 651984355 MDEyOklzc3VlQ29tbWVudDY1MTk4NDM1NQ== 9599 2020-06-30T19:02:15Z 2020-06-30T19:02:15Z OWNER https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html#login-csrf > Login CSRF can be mitigated by creating pre-sessions (sessions before a user is authenticated) and including tokens in login form. Sounds like regular CSRF protection to me.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
648421105
https://github.com/simonw/datasette/pull/868#issuecomment-650340914 https://api.github.com/repos/simonw/datasette/issues/868 650340914 MDEyOklzc3VlQ29tbWVudDY1MDM0MDkxNA== 22429695 2020-06-26T18:53:02Z 2020-06-30T03:51:22Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=h1) Report > Merging [#868](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/a8a5f813722f72703a7aae41135ccc40635cc02f&el=desc) will **increase** coverage by `0.11%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/868/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #868 +/- ## ========================================== + Coverage 83.31% 83.42% +0.11% ========================================== Files 27 27 Lines 3595 3614 +19 ========================================== + Hits 2995 3015 +20 + Misses 600 599 -1 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.93% <0.00%> (+0.05%)` | :arrow_up: | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.47% <0.00%> (+0.19%)` | :arrow_up: | | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `81.17% <0.00%> (+3.39%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=footer). Last update [a8a5f81...ef837b3](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646448486
https://github.com/simonw/datasette/issues/805#issuecomment-651302221 https://api.github.com/repos/simonw/datasette/issues/805 651302221 MDEyOklzc3VlQ29tbWVudDY1MTMwMjIyMQ== 9599 2020-06-29T19:02:45Z 2020-06-29T19:05:26Z OWNER No I prefer the idea that logged out users can still perform some writes, in a not-likely-to-attract-abuse way. So a root-user-can-configure-polls, logged-out-users-can-vote-in-them demo would be good. Or... crazy idea: a collaborative drawing program? A grid of cells of emoji, anyone can add an emoji to a cell. Would involve a bit of JavaScript. I could use https://github.com/joeattardi/emoji-button for this.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
632724154
https://github.com/simonw/datasette/issues/805#issuecomment-651301202 https://api.github.com/repos/simonw/datasette/issues/805 651301202 MDEyOklzc3VlQ29tbWVudDY1MTMwMTIwMg== 9599 2020-06-29T19:00:37Z 2020-06-29T19:00:37Z OWNER How about a blog? Pre-configured canned queries that are only available to `"root"`, plus datasette-template-sql and default templates for the index page and blog entry pages.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
632724154
https://github.com/simonw/datasette/issues/875#issuecomment-651293559 https://api.github.com/repos/simonw/datasette/issues/875 651293559 MDEyOklzc3VlQ29tbWVudDY1MTI5MzU1OQ== 9599 2020-06-29T18:43:50Z 2020-06-29T18:43:50Z OWNER <img width="470" alt="_memory_" src="https://user-images.githubusercontent.com/9599/86043599-ca4b0500-b9fd-11ea-9378-a053e96cfde7.png">
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647103735
https://github.com/simonw/datasette/issues/873#issuecomment-651203178 https://api.github.com/repos/simonw/datasette/issues/873 651203178 MDEyOklzc3VlQ29tbWVudDY1MTIwMzE3OA== 9599 2020-06-29T15:44:38Z 2020-06-29T15:44:54Z OWNER I'm having real trouble figuring out how to gain access to the port that was used to start the server. I'm treating this as a very low priority - it only affects the exact `-p 0 --root` combination which isn't going to affect many people at all.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-651193594 https://api.github.com/repos/simonw/datasette/issues/873 651193594 MDEyOklzc3VlQ29tbWVudDY1MTE5MzU5NA== 9599 2020-06-29T15:27:46Z 2020-06-29T15:27:46Z OWNER Uninstalling `datasette-debug-asgi` caused the server to startup correctly again.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-651193131 https://api.github.com/repos/simonw/datasette/issues/873 651193131 MDEyOklzc3VlQ29tbWVudDY1MTE5MzEzMQ== 9599 2020-06-29T15:27:00Z 2020-06-29T15:27:00Z OWNER Aha! Yes it's not being called, and the reason is this: https://github.com/encode/starlette/issues/486 Short version: by default an exception raised during that phase is silently swallowed! You can avoid the swallowing by adding `lifespan="on"` to the call to `uvicorn.run()`. When I did that here: `uvicorn.run(ds.app(), host=host, port=port, log_level="info", lifespan="on")` The server failed to start with this error: ``` INFO: Started server process [68849] INFO: Waiting for application startup. ERROR: Exception in 'lifespan' protocol Traceback (most recent call last): File ".../uvicorn/lifespan/on.py", line 48, in main await app(scope, self.receive, self.send) File ".../uvicorn/middleware/proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File ".../datasette_debug_asgi.py", line 9, in wrapped_app if scope["path"] == "/-/asgi-scope": KeyError: 'path' ```
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650910137 https://api.github.com/repos/simonw/datasette/issues/873 650910137 MDEyOklzc3VlQ29tbWVudDY1MDkxMDEzNw== 9599 2020-06-29T05:16:32Z 2020-06-29T05:16:32Z OWNER I'm not convinced that function is ever actually being called - I added a `print()` statement to it and it's not executing. I don't think the tests cover it.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650909476 https://api.github.com/repos/simonw/datasette/issues/873 650909476 MDEyOklzc3VlQ29tbWVudDY1MDkwOTQ3Ng== 9599 2020-06-29T05:14:08Z 2020-06-29T05:14:08Z OWNER I already have a `AsgiLifespan` class: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L896-L905 It runs this function: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/app.py#L890-L894 Could that startup function also output the `--root` login URL, if needed?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650909136 https://api.github.com/repos/simonw/datasette/issues/873 650909136 MDEyOklzc3VlQ29tbWVudDY1MDkwOTEzNg== 9599 2020-06-29T05:12:58Z 2020-06-29T05:12:58Z OWNER On startup Datasette currently outputs: ``` INFO: Waiting for application startup. INFO: ASGI 'lifespan' protocol appears unsupported. INFO: Application startup complete. ``` So the ASGI lifespan protocol is almost certainly the right way to solve this.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650908854 https://api.github.com/repos/simonw/datasette/issues/873 650908854 MDEyOklzc3VlQ29tbWVudDY1MDkwODg1NA== 9599 2020-06-29T05:12:04Z 2020-06-29T05:12:04Z OWNER Can I detect the port the server is running on from within the regular Datasette ASGI code? If so I could use that ability and maybe output the magic `--root` link a second after the server starts up somehow.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650908534 https://api.github.com/repos/simonw/datasette/issues/873 650908534 MDEyOklzc3VlQ29tbWVudDY1MDkwODUzNA== 9599 2020-06-29T05:11:06Z 2020-06-29T05:11:06Z OWNER Uvicorn's lifespan stuff isn't easy to figure out, but this test suite holds some clues: https://github.com/encode/uvicorn/blob/master/tests/test_lifespan.py
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650907323 https://api.github.com/repos/simonw/datasette/issues/873 650907323 MDEyOklzc3VlQ29tbWVudDY1MDkwNzMyMw== 9599 2020-06-29T05:07:16Z 2020-06-29T05:07:16Z OWNER This line is interesting: is this a hook I can attach to somehow? ```python await self.lifespan.startup() ``` From https://github.com/encode/uvicorn/blob/a75fe1381f6b1f78901691c71894f3cf487b5d30/uvicorn/main.py#L475
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650906533 https://api.github.com/repos/simonw/datasette/issues/873 650906533 MDEyOklzc3VlQ29tbWVudDY1MDkwNjUzMw== 9599 2020-06-29T05:04:44Z 2020-06-29T05:04:44Z OWNER The challenge is... can we run our own custom code after that line has executed that has access to `server` and can hence access `server.servers[0].sockets[0].getsockname()[1]` to find the port?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650906318 https://api.github.com/repos/simonw/datasette/issues/873 650906318 MDEyOklzc3VlQ29tbWVudDY1MDkwNjMxOA== 9599 2020-06-29T05:04:04Z 2020-06-29T05:04:12Z OWNER Within uvicorn it does this: ```python if port == 0: port = server.sockets[0].getsockname()[1] ``` That `server` variable is later stashed here: ``` self.servers = [server] ``` Where `self` is the instance of `class Server` - which is the class that Uvicorn instantiates and calls `.run()` on when we do `uvicorn.run()` here: https://github.com/simonw/datasette/blob/35aee82c60b2c9a0185b934db5528c8bd11830f2/datasette/cli.py#L409
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/873#issuecomment-650905399 https://api.github.com/repos/simonw/datasette/issues/873 650905399 MDEyOklzc3VlQ29tbWVudDY1MDkwNTM5OQ== 9599 2020-06-29T05:01:03Z 2020-06-29T05:01:03Z OWNER This is a bit tricky to fix. This change to uvicorn is relevant: https://github.com/encode/uvicorn/commit/a75fe1381f6b1f78901691c71894f3cf487b5d30
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647095487
https://github.com/simonw/datasette/issues/875#issuecomment-650899265 https://api.github.com/repos/simonw/datasette/issues/875 650899265 MDEyOklzc3VlQ29tbWVudDY1MDg5OTI2NQ== 9599 2020-06-29T04:34:32Z 2020-06-29T04:34:32Z OWNER From https://github.com/simonw/datasette/issues/840#issuecomment-643454625 > Another problem: what to display in the "you are logged in as", since we don't dictate an actor design. > > I'm going to use a includes template for this that can easily be over-ridden by administrators or by plugins. > > The default will look for the first available of the following keys: > > * display > * name > * username > * login > * id
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647103735
https://github.com/simonw/datasette/issues/875#issuecomment-650898808 https://api.github.com/repos/simonw/datasette/issues/875 650898808 MDEyOklzc3VlQ29tbWVudDY1MDg5ODgwOA== 9599 2020-06-29T04:32:31Z 2020-06-29T04:33:30Z OWNER I could borrow the implementation for this from `datasette-auth-github` https://github.com/simonw/datasette-auth-github/blob/182298b034ecb647971b65057d1d3e7b7fbbb482/datasette_auth_github/templates/base.html ```html+jinja {% extends "default:base.html" %} {% block extra_head %} <style type="text/css"> .hd .logout { float: right; text-align: right; padding-left: 1em; } </style> {% endblock %} {% block nav %} {{ super() }} {% if auth and auth.username %} <p class="logout"> <strong>{{ auth.username }}</strong> &middot; <a href="/-/logout">Log out</a> </p> {% endif %} {% endblock %} ```
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
647103735
https://github.com/simonw/datasette/issues/840#issuecomment-650895874 https://api.github.com/repos/simonw/datasette/issues/840 650895874 MDEyOklzc3VlQ29tbWVudDY1MDg5NTg3NA== 9599 2020-06-29T04:18:59Z 2020-06-29T04:19:11Z OWNER Now just need the "Logged in as: XXX &lt;logout&gt;" navigation item.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
637966833
https://github.com/simonw/datasette/issues/840#issuecomment-650891502 https://api.github.com/repos/simonw/datasette/issues/840 650891502 MDEyOklzc3VlQ29tbWVudDY1MDg5MTUwMg== 9599 2020-06-29T03:58:08Z 2020-06-29T03:58:08Z OWNER Step one: a "logout" page at `/-/logout` - which shows you a single CSRF-protected "logout" button if you do a GET against it and logs you out if you do a POST against it.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
637966833
https://github.com/simonw/datasette/issues/805#issuecomment-650891257 https://api.github.com/repos/simonw/datasette/issues/805 650891257 MDEyOklzc3VlQ29tbWVudDY1MDg5MTI1Nw== 9599 2020-06-29T03:56:48Z 2020-06-29T03:56:48Z OWNER Using `datasette-glitch` and the new https://github.com/simonw/datasette-write - currently running on `datasette==0.45a4` - works on Glitch. The console shows a login link which gives you a cookie which allows you access to the `/-/write` interface.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
632724154
https://github.com/simonw/datasette/issues/864#issuecomment-650847013 https://api.github.com/repos/simonw/datasette/issues/864 650847013 MDEyOklzc3VlQ29tbWVudDY1MDg0NzAxMw== 9599 2020-06-29T00:41:55Z 2020-06-29T00:41:55Z OWNER To test this I'll need a plugin test that renders a custom template. Here's an example I can imitate: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/tests/test_plugins.py#L588-L596
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644309017
https://github.com/simonw/datasette/issues/864#issuecomment-650846625 https://api.github.com/repos/simonw/datasette/issues/864 650846625 MDEyOklzc3VlQ29tbWVudDY1MDg0NjYyNQ== 9599 2020-06-29T00:39:47Z 2020-06-29T00:39:47Z OWNER I think the fix is to move the `"show_messages"` variable to here: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/app.py#L735-L748
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644309017
https://github.com/simonw/datasette/issues/864#issuecomment-650846473 https://api.github.com/repos/simonw/datasette/issues/864 650846473 MDEyOklzc3VlQ29tbWVudDY1MDg0NjQ3Mw== 9599 2020-06-29T00:39:04Z 2020-06-29T00:39:04Z OWNER Re-opening: plugins may get to set messages but they don't display them, even if they render a template that extends `base.html`. For example, this code in a plugin: ```python return Response.html( await datasette.render_template( "write.html", {"databases": databases, "sql": request.args.get("sql") or ""}, request=request, ) ) ``` This won't display messages. The reason is that the messages are made available to the template context in the `BaseView.render()` method here: https://github.com/simonw/datasette/blob/7ac4936cec87f5a591e5d2680f0acefc3d35a705/datasette/views/base.py#L87-L95
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644309017
https://github.com/simonw/datasette/issues/864#issuecomment-650842514 https://api.github.com/repos/simonw/datasette/issues/864 650842514 MDEyOklzc3VlQ29tbWVudDY1MDg0MjUxNA== 9599 2020-06-29T00:12:59Z 2020-06-29T00:12:59Z OWNER > I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release. https://github.com/simonw/datasette/issues/870#issuecomment-650842381
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644309017
https://github.com/simonw/datasette/issues/870#issuecomment-650842381 https://api.github.com/repos/simonw/datasette/issues/870 650842381 MDEyOklzc3VlQ29tbWVudDY1MDg0MjM4MQ== 9599 2020-06-29T00:12:07Z 2020-06-29T00:12:07Z OWNER I've made enough progress on this to be able to solve the messages issue in #864. I may still complete this overall goal (registering internal views with `register_routes()`) as part of Datasette 0.45 but it would be OK if it slipped to a later release.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/870#issuecomment-650838972 https://api.github.com/repos/simonw/datasette/issues/870 650838972 MDEyOklzc3VlQ29tbWVudDY1MDgzODk3Mg== 9599 2020-06-28T23:46:40Z 2020-06-28T23:46:40Z OWNER I'm going to create the single `Request()` instance in the `DatasetteRouter` class - at the beginning of the `route_path` method: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L905-L925
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/870#issuecomment-650838691 https://api.github.com/repos/simonw/datasette/issues/870 650838691 MDEyOklzc3VlQ29tbWVudDY1MDgzODY5MQ== 9599 2020-06-28T23:44:12Z 2020-06-28T23:44:25Z OWNER This code is interesting: https://github.com/simonw/datasette/blob/3bc2461c77ecba3e1a95301dd440a9bef56b1283/datasette/app.py#L948-L955 I want to change the signature of that `return await view(new_scope, receive, send)` method to instead take `(request, send)` - so I can have a single shared request object that's created just once per HTTP request. The problem is the scope modification: I have code that modifies the scope, but how should that impact a shared `Request` instance? Should its `.scope` be replaced with alternative scopes as it travels through the codebase?
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/870#issuecomment-650834666 https://api.github.com/repos/simonw/datasette/issues/870 650834666 MDEyOklzc3VlQ29tbWVudDY1MDgzNDY2Ng== 9599 2020-06-28T23:07:19Z 2020-06-28T23:07:19Z OWNER So now the problem is simpler: I need to get `BaseView` to a state where it can accept a shared `request` object and it can be used in conjunction with `register_routes()`.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/870#issuecomment-650834251 https://api.github.com/repos/simonw/datasette/issues/870 650834251 MDEyOklzc3VlQ29tbWVudDY1MDgzNDI1MQ== 9599 2020-06-28T23:03:28Z 2020-06-28T23:03:28Z OWNER I'm going to ditch that `AsgiView` class too, by combining it into `BaseView`.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/870#issuecomment-650820068 https://api.github.com/repos/simonw/datasette/issues/870 650820068 MDEyOklzc3VlQ29tbWVudDY1MDgyMDA2OA== 9599 2020-06-28T20:52:09Z 2020-06-28T20:53:00Z OWNER Maybe I could add a `as_request_view` method as an alternative to `as_asgi`: https://github.com/simonw/datasette/blob/a8bcafc1775c8a8655b365ae22a3d64f6361c74a/datasette/utils/asgi.py#L150-L174 Or I could teach the `Router` to spot the `dispatch_request` method and call it directly.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/847#issuecomment-650819895 https://api.github.com/repos/simonw/datasette/issues/847 650819895 MDEyOklzc3VlQ29tbWVudDY1MDgxOTg5NQ== 9599 2020-06-28T20:50:21Z 2020-06-28T20:50:21Z OWNER I'm happy enough with https://codecov.io/gh/simonw/datasette that I'm not going to spend any more time on this.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638259643
https://github.com/simonw/datasette/issues/870#issuecomment-650818309 https://api.github.com/repos/simonw/datasette/issues/870 650818309 MDEyOklzc3VlQ29tbWVudDY1MDgxODMwOQ== 9599 2020-06-28T20:36:28Z 2020-06-28T20:36:52Z OWNER Since `AsgiRouter` is only used as the super-class of the `DatasetteRouter` class maybe I should get rid of `AsgiRouter` entirely - no point in having a Datasette-specific subclass of it if the parent class isn't ever used by anything else. I could also rename it to just `Router` which is a nicer name than `DatasetteRouter`.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/870#issuecomment-650818086 https://api.github.com/repos/simonw/datasette/issues/870 650818086 MDEyOklzc3VlQ29tbWVudDY1MDgxODA4Ng== 9599 2020-06-28T20:34:33Z 2020-06-28T20:34:33Z OWNER The key to all of this may be the `DatasetteRouter` class. It deals with `scope` right now but if it internally dealt with `request` that could be enough to fix #864 by adding logic needed by the `.add_message()` mechanism. https://github.com/simonw/datasette/blob/0991ea75cc7b265389aa8362414a305ba532d31a/datasette/app.py#L904-L938
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/870#issuecomment-650815278 https://api.github.com/repos/simonw/datasette/issues/870 650815278 MDEyOklzc3VlQ29tbWVudDY1MDgxNTI3OA== 9599 2020-06-28T20:09:07Z 2020-06-28T20:11:21Z OWNER There's a lot of complex logic in the `DataView` class, which handles conditionally returning content as `.json` or as HTML or as `.csv`. That view subclasses `AsgiView` which is itself request-aware, so maybe I don't need to reconsider how those classes work - just figure out how to hook them up with `register_routes`.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/871#issuecomment-650812444 https://api.github.com/repos/simonw/datasette/issues/871 650812444 MDEyOklzc3VlQ29tbWVudDY1MDgxMjQ0NA== 9599 2020-06-28T19:43:27Z 2020-06-28T19:43:27Z OWNER Currently: > `_timestamp_epoch` > > The number of seconds since the Unix epoch. > > `_timestamp_date_utc` > > The date in UTC, e.g. `2020-06-01` > > `_timestamp_datetime_utc` > > The ISO 8601 datetime in UTC, e.g. `2020-06-24T18:01:07Z` I'm going to rename them to: - `_now_epoch` - `_now_date_utc` - `_now_datetime_utc`
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646840273
https://github.com/simonw/datasette/issues/834#issuecomment-650811919 https://api.github.com/repos/simonw/datasette/issues/834 650811919 MDEyOklzc3VlQ29tbWVudDY1MDgxMTkxOQ== 9599 2020-06-28T19:38:50Z 2020-06-28T19:38:50Z OWNER I have two plugins in progress that use this hook now: - https://github.com/simonw/datasette-init creates tables and views on startup - https://github.com/simonw/datasette-glitch outputs the login-as-root secret link on Glitch
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
637342551
https://github.com/simonw/datasette/issues/805#issuecomment-650784162 https://api.github.com/repos/simonw/datasette/issues/805 650784162 MDEyOklzc3VlQ29tbWVudDY1MDc4NDE2Mg== 9599 2020-06-28T15:48:32Z 2020-06-28T15:48:32Z OWNER https://github.com/simonw/datasette-glitch is my new plugin that outputs the root login link on Glitch when the server starts.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
632724154
https://github.com/simonw/datasette/issues/870#issuecomment-650696054 https://api.github.com/repos/simonw/datasette/issues/870 650696054 MDEyOklzc3VlQ29tbWVudDY1MDY5NjA1NA== 9599 2020-06-28T04:52:41Z 2020-06-28T04:52:41Z OWNER This would be a lot easier if I had extracted out the hash logic to a plugin, see #745.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646737558
https://github.com/simonw/datasette/issues/834#issuecomment-643657067 https://api.github.com/repos/simonw/datasette/issues/834 643657067 MDEyOklzc3VlQ29tbWVudDY0MzY1NzA2Nw== 9599 2020-06-13T17:59:42Z 2020-06-28T04:01:52Z OWNER Documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#startup-datasette
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
637342551
https://github.com/simonw/datasette/issues/842#issuecomment-650684635 https://api.github.com/repos/simonw/datasette/issues/842 650684635 MDEyOklzc3VlQ29tbWVudDY1MDY4NDYzNQ== 9599 2020-06-28T03:30:31Z 2020-06-28T03:30:31Z OWNER Live demo: https://latest.datasette.io/fixtures/magic_parameters
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/805#issuecomment-650681496 https://api.github.com/repos/simonw/datasette/issues/805 650681496 MDEyOklzc3VlQ29tbWVudDY1MDY4MTQ5Ng== 9599 2020-06-28T03:11:51Z 2020-06-28T03:11:51Z OWNER I can use magic parameters from #842 in this.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
632724154
https://github.com/simonw/datasette/issues/842#issuecomment-650679100 https://api.github.com/repos/simonw/datasette/issues/842 650679100 MDEyOklzc3VlQ29tbWVudDY1MDY3OTEwMA== 9599 2020-06-28T03:00:44Z 2020-06-28T03:00:44Z OWNER I'm going to add some canned queries to the `metadata.json` used by the live demo that illustrate this feature.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/842#issuecomment-650678951 https://api.github.com/repos/simonw/datasette/issues/842 650678951 MDEyOklzc3VlQ29tbWVudDY1MDY3ODk1MQ== 9599 2020-06-28T02:59:52Z 2020-06-28T02:59:52Z OWNER Documentation: https://datasette.readthedocs.io/en/latest/sql_queries.html#magic-parameters Plugin hook documentation: https://datasette.readthedocs.io/en/latest/plugin_hooks.html#plugin-hook-register-magic-parameters
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/pull/869#issuecomment-650600176 https://api.github.com/repos/simonw/datasette/issues/869 650600176 MDEyOklzc3VlQ29tbWVudDY1MDYwMDE3Ng== 22429695 2020-06-27T18:41:31Z 2020-06-28T02:54:21Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=h1) Report > Merging [#869](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c&el=desc) will **increase** coverage by `0.23%`. > The diff coverage is `90.62%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/869/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #869 +/- ## ========================================== + Coverage 82.99% 83.23% +0.23% ========================================== Files 26 27 +1 Lines 3547 3609 +62 ========================================== + Hits 2944 3004 +60 - Misses 603 605 +2 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.45% <86.36%> (-1.88%)` | :arrow_down: | | [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <91.17%> (ø)` | | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.07% <100.00%> (+0.81%)` | :arrow_up: | | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/869/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.87% <100.00%> (+0.02%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=footer). Last update [1bb33da...9e693a7](https://codecov.io/gh/simonw/datasette/pull/869?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646734280
https://github.com/simonw/datasette/issues/842#issuecomment-650648434 https://api.github.com/repos/simonw/datasette/issues/842 650648434 MDEyOklzc3VlQ29tbWVudDY1MDY0ODQzNA== 9599 2020-06-27T23:27:35Z 2020-06-27T23:37:38Z OWNER I'm going to rename `_request_X` to `_header_X` as that better reflects what it now does.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/pull/868#issuecomment-650600606 https://api.github.com/repos/simonw/datasette/issues/868 650600606 MDEyOklzc3VlQ29tbWVudDY1MDYwMDYwNg== 9599 2020-06-27T18:44:28Z 2020-06-27T18:44:28Z OWNER This is really exciting! Thanks so much for looking into this. I'm interested in moving CI for this repo over to GitHub Actions, so I'd be fine with you getting this to work as an Action rather than through Travis. If you can get it working in Travis though I'll happily land that and figure out how to convert that to GitHub Actions later on.
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
646448486
https://github.com/simonw/datasette/issues/835#issuecomment-650598710 https://api.github.com/repos/simonw/datasette/issues/835 650598710 MDEyOklzc3VlQ29tbWVudDY1MDU5ODcxMA== 9599 2020-06-27T18:32:22Z 2020-06-27T18:32:22Z OWNER Skipping CSRF on `Authorization: Bearer xxx` headers also makes sense for JWT applications, which tend to send JWTs using that form of header.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
637363686
https://github.com/simonw/datasette/issues/842#issuecomment-650593122 https://api.github.com/repos/simonw/datasette/issues/842 650593122 MDEyOklzc3VlQ29tbWVudDY1MDU5MzEyMg== 9599 2020-06-27T18:03:02Z 2020-06-27T18:03:10Z OWNER > Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see. This is an open security hole in https://github.com/simonw/datasette/commit/94c1315f0030fd58ce46a9294052c5c9d9d181c7 - it's useful for testing, but I need to remove it before I land that branch. https://github.com/simonw/datasette/blob/94c1315f0030fd58ce46a9294052c5c9d9d181c7/datasette/views/database.py#L231-L237
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/842#issuecomment-650458857 https://api.github.com/repos/simonw/datasette/issues/842 650458857 MDEyOklzc3VlQ29tbWVudDY1MDQ1ODg1Nw== 9599 2020-06-27T00:11:04Z 2020-06-27T00:11:04Z OWNER Security thought: make sure it's not possible to accidentally open up a security hole where an attacker can send a GET request that causes the magic parameter `_cookie_ds_actor` to be resolved and returned as JSON data that the attacker can see.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/842#issuecomment-650455793 https://api.github.com/repos/simonw/datasette/issues/842 650455793 MDEyOklzc3VlQ29tbWVudDY1MDQ1NTc5Mw== 9599 2020-06-26T23:57:30Z 2020-06-27T00:00:16Z OWNER Maybe I should ship a default `_scope_headers_...` parameter instead, which reads from a dictionary of `scope["headers"]` - https://asgi-scope.now.sh/ shows what those look like. ``` {'client': ('148.64.98.14', 0), 'headers': [[b'host', b'asgi-scope.now.sh'], [b'x-forwarded-for', b'148.64.98.14'], [b'x-vercel-id', b'sw72x-1593215573008-024e4e603806'], [b'x-forwarded-host', b'asgi-scope.now.sh'], [b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/' b'webp,*/*;q=0.8'], [b'x-real-ip', b'148.64.98.14'], [b'x-vercel-deployment-url', b'asgi-scope-9eyeojbek.now.sh'], [b'upgrade-insecure-requests', b'1'], [b'x-vercel-trace', b'sfo1'], [b'x-forwarded-proto', b'https'], [b'accept-language', b'en-US,en;q=0.5'], [b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko' b'/20100101 Firefox/77.0'], [b'x-vercel-forwarded-for', b'148.64.98.14'], [b'accept-encoding', b'gzip, deflate, br'], [b'dnt', b'1'], [b'te', b'trailers']], 'http_version': '1.1', 'method': 'GET', 'path': '/', 'query_string': b'', 'raw_path': b'/', 'root_path': '', 'scheme': 'https', 'server': ('asgi-scope.now.sh', 80), 'type': 'http'} ``` I'm going to have `_request_X` actually mean "find the first value for X in `scope["headers"`]" - with underscores converted to hyphens.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/842#issuecomment-650455353 https://api.github.com/repos/simonw/datasette/issues/842 650455353 MDEyOklzc3VlQ29tbWVudDY1MDQ1NTM1Mw== 9599 2020-06-26T23:55:40Z 2020-06-26T23:55:40Z OWNER `_request_ip` is actually quite hard to implement - should it take into account things like the `x-forwarded-for` header? It probably should - but that means it now needs a bunch of extra configuration to tell it which of those headers can be trusted in the current environment. As such I think I'll leave that for a plugin.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/867#issuecomment-649931714 https://api.github.com/repos/simonw/datasette/issues/867 649931714 MDEyOklzc3VlQ29tbWVudDY0OTkzMTcxNA== 9599 2020-06-26T03:12:51Z 2020-06-26T03:12:51Z OWNER Here's the relevant code: https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/datasette/app.py#L1057-L1070 And the relevant test code: https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/test_plugins.py#L567-L573 https://github.com/simonw/datasette/blob/1bb33dab49fd25f77b9f8e7ab7ee23b3d64c123c/tests/plugins/my_plugin.py#L162-L196
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
645975649
https://github.com/simonw/datasette/issues/849#issuecomment-649908756 https://api.github.com/repos/simonw/datasette/issues/849 649908756 MDEyOklzc3VlQ29tbWVudDY0OTkwODc1Ng== 9599 2020-06-26T02:09:09Z 2020-06-26T02:09:09Z OWNER I mentioned this issue here: https://simonwillison.net/2020/Jun/26/weeknotes-plugins-sqlite-generate/ Repositories created by following the REAME in https://github.com/simonw/datasette-template and https://github.com/simonw/click-app have a `main` branch instead of `master` so I have a few examples live now. https://github.com/simonw/datasette-saved-queries is one example.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
639072811
https://github.com/simonw/datasette/issues/842#issuecomment-649014757 https://api.github.com/repos/simonw/datasette/issues/842 649014757 MDEyOklzc3VlQ29tbWVudDY0OTAxNDc1Nw== 9599 2020-06-24T19:15:46Z 2020-06-24T19:31:52Z OWNER I'm building this documentation-first - here's the documentation so far: https://github.com/simonw/datasette/blob/6fc8bd9c473f4a25e0a076f24c7e5a9b2f353bb8/docs/sql_queries.rst#magic-parameters
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/842#issuecomment-646271834 https://api.github.com/repos/simonw/datasette/issues/842 646271834 MDEyOklzc3VlQ29tbWVudDY0NjI3MTgzNA== 9599 2020-06-18T19:49:41Z 2020-06-24T18:49:22Z OWNER But then what kind of magic parameters might plugins want to add? Here's a crazy idea: `_scrapedcontent_url` - it would look for the `url` column on the data being inserted, scrape the content from it and insert that. This does suggest that the magic resolving function `scrapedcontent()` would need to optionally be sent the full row dictionary being inserted too.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/842#issuecomment-646270702 https://api.github.com/repos/simonw/datasette/issues/842 646270702 MDEyOklzc3VlQ29tbWVudDY0NjI3MDcwMg== 9599 2020-06-18T19:47:19Z 2020-06-24T18:48:48Z OWNER Brainstorming more potential magic parameters: * `_actor_id` * `_actor_name` * `_request_ip` * `_request_user_agent` * `_cookie_cookiename` * `_signedcookie_cookiename` - reading signed cookies would be cool, not sure how to specify namespace though, maybe always use the same one? Or have the namespace come last, `_signedcookie_cookiename_mynamespace`. Might not need special signed cookie support since `actor` is already usually from a signed cookie. * `_timestamp_unix` (not happy with these names yet) * `_timestamp_localtime` * `_timestamp_datetime` * `_timestamp_utc`
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/842#issuecomment-649000075 https://api.github.com/repos/simonw/datasette/issues/842 649000075 MDEyOklzc3VlQ29tbWVudDY0OTAwMDA3NQ== 9599 2020-06-24T18:46:36Z 2020-06-24T18:47:37Z OWNER Another magic parameter that would be useful would be `_random`. Consider https://github.com/simonw/datasette-auth-tokens/issues/1 for example - I'd like to be able to provide a writable canned query which can create new authentication tokens in the database, but ideally it would automatically populate a secure random secret for each one. Maybe `_random_chars_128` to create a 128 character long random string (using `os.urandom(64).hex()`). This would be the first example of a magic parameter where part of the parameter name is used to configure the resulting value. Maybe neater to separate that with a different character? Unfortunately `_random_chars:128` wouldn't work because these parameters are used in a SQLite query where `:` has special meaning: `insert into blah (secret) values (:_random_chars:128)` wouldn't make sense. Actually this is already supported by the proposed design - `_random_chars_128` would become `random("chars_128")` so the `random()` function could split off the 128 itself.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
638212085
https://github.com/simonw/datasette/issues/865#issuecomment-648998264 https://api.github.com/repos/simonw/datasette/issues/865 648998264 MDEyOklzc3VlQ29tbWVudDY0ODk5ODI2NA== 9599 2020-06-24T18:43:02Z 2020-06-24T18:43:02Z OWNER Thanks for the bug report. Yes I think #838 may be the same issue. Will investigate.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644582921
https://github.com/simonw/datasette/issues/858#issuecomment-648997857 https://api.github.com/repos/simonw/datasette/issues/858 648997857 MDEyOklzc3VlQ29tbWVudDY0ODk5Nzg1Nw== 9599 2020-06-24T18:42:10Z 2020-06-24T18:42:10Z OWNER I really need to get myself a Windows 10 development environment working so I can dig into this kind of bug properly. I have a gaming PC lying around that I could re-task for that.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642388564
https://github.com/simonw/datasette/pull/866#issuecomment-648818707 https://api.github.com/repos/simonw/datasette/issues/866 648818707 MDEyOklzc3VlQ29tbWVudDY0ODgxODcwNw== 22429695 2020-06-24T13:26:14Z 2020-06-24T13:26:14Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=h1) Report > Merging [#866](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=desc) into [master](https://codecov.io/gh/simonw/datasette/commit/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f&el=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/866/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #866 +/- ## ======================================= Coverage 82.99% 82.99% ======================================= Files 26 26 Lines 3547 3547 ======================================= Hits 2944 2944 Misses 603 603 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=footer). Last update [1a5b7d3...fb64dda](https://codecov.io/gh/simonw/datasette/pull/866?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644610729
https://github.com/simonw/datasette/issues/838#issuecomment-648800356 https://api.github.com/repos/simonw/datasette/issues/838 648800356 MDEyOklzc3VlQ29tbWVudDY0ODgwMDM1Ng== 6739646 2020-06-24T12:51:48Z 2020-06-24T12:51:48Z NONE >But also want to say thanks for a great tool +1!
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
637395097
https://github.com/simonw/datasette/issues/865#issuecomment-648799963 https://api.github.com/repos/simonw/datasette/issues/865 648799963 MDEyOklzc3VlQ29tbWVudDY0ODc5OTk2Mw== 6739646 2020-06-24T12:51:01Z 2020-06-24T12:51:01Z NONE This seems to be a duplicate of: https://github.com/simonw/datasette/issues/838
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644582921
https://github.com/simonw/datasette/issues/859#issuecomment-648669523 https://api.github.com/repos/simonw/datasette/issues/859 648669523 MDEyOklzc3VlQ29tbWVudDY0ODY2OTUyMw== 3243482 2020-06-24T08:13:23Z 2020-06-24T10:30:36Z CONTRIBUTOR I tried setting `cache_size_kb=0` then `cache_size_kb=100000`, still getting this behavior. I even changed `Database::table_counts` and lowered time limit to 1 ```py table_count = ( await self.execute( "select count(*) from [{}]".format(table), custom_time_limit=1, ) ).rows[0][0] counts[table] = table_count ``` I feel like 10 seconds is a magic number, like a processing timeout and datasette gives up and returns the page. Index page loads instantly, table page, query page, as well. But when I return to database page after some time, it loads in 10s. EDIT: It's always like 10 + 0.3s, like 10s wait and timeout then 300ms to render the page
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/864#issuecomment-648580556 https://api.github.com/repos/simonw/datasette/issues/864 648580556 MDEyOklzc3VlQ29tbWVudDY0ODU4MDU1Ng== 9599 2020-06-24T04:40:49Z 2020-06-24T04:40:49Z OWNER The ideal fix here would be to rework my `BaseView` subclass mechanism to work with `register_routes()` so that those views don't have any special privileges above plugin-provided views.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644309017
https://github.com/simonw/datasette/issues/864#issuecomment-648580236 https://api.github.com/repos/simonw/datasette/issues/864 648580236 MDEyOklzc3VlQ29tbWVudDY0ODU4MDIzNg== 9599 2020-06-24T04:39:39Z 2020-06-24T04:39:39Z OWNER Urgh, fixing this is going to be a bit of a pain. Here's where I added that custom `dispatch_request()` method - it was to implement flash messaging in #790: https://github.com/simonw/datasette/blame/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/views/base.py#L85 If I want this to be made available to `register_routes()` views as well, I'm going to have to move the logic somewhere else. In particular I need to make sure that the `request` object is created once and used throughout the whole request cycle. Currently `register_routes()` view functions get their own separate request object which is created here: https://github.com/simonw/datasette/blob/1a5b7d318fa923edfcefd3df8f64dae2e9c49d3f/datasette/app.py#L1057-L1068 So I'm going to have to refactor this quite a bit to get that shared request object which can be passed both to `register_routes` views and to my various `BaseView` subclasses.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644309017
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648442511 https://api.github.com/repos/simonw/sqlite-utils/issues/117 648442511 MDEyOklzc3VlQ29tbWVudDY0ODQ0MjUxMQ== 9599 2020-06-23T21:39:41Z 2020-06-23T21:39:41Z OWNER So there are two sides to supporting this: - Being able to sensibly introspect composite foreign keys - Being able to define composite foreign keys when creating a table
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644161221
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440634 https://api.github.com/repos/simonw/sqlite-utils/issues/117 648440634 MDEyOklzc3VlQ29tbWVudDY0ODQ0MDYzNA== 9599 2020-06-23T21:35:16Z 2020-06-23T21:35:16Z OWNER Relevant discussion: https://github.com/simonw/sqlite-generate/issues/8#issuecomment-648438056
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644161221
https://github.com/simonw/sqlite-utils/issues/117#issuecomment-648440525 https://api.github.com/repos/simonw/sqlite-utils/issues/117 648440525 MDEyOklzc3VlQ29tbWVudDY0ODQ0MDUyNQ== 9599 2020-06-23T21:35:01Z 2020-06-23T21:35:01Z OWNER Here's what's missing: ``` In [11]: db.conn.execute('PRAGMA foreign_key_list(song)').fetchall() Out[11]: [(0, 0, 'album', 'songartist', 'albumartist', 'NO ACTION', 'NO ACTION', 'NONE'), (0, 1, 'album', 'songalbum', 'albumname', 'NO ACTION', 'NO ACTION', 'NONE')] ``` Compare with this code here: https://github.com/simonw/sqlite-utils/blob/d0cdaaaf00249230e847be3a3b393ee2689fbfe4/sqlite_utils/db.py#L563-L579 The first two columns returned by `PRAGMA foreign_key_list(table)` are `id` and `seq` - these show when two foreign key records are part of the same compound foreign key. `sqlite-utils` entirely ignores those at the moment.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644161221
https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648434885 https://api.github.com/repos/simonw/sqlite-utils/issues/116 648434885 MDEyOklzc3VlQ29tbWVudDY0ODQzNDg4NQ== 9599 2020-06-23T21:21:33Z 2020-06-23T21:21:33Z OWNER New docs: https://github.com/simonw/sqlite-utils/blob/2.10.1/docs/python-api.rst#introspection
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644122661
https://github.com/simonw/sqlite-utils/issues/116#issuecomment-648403834 https://api.github.com/repos/simonw/sqlite-utils/issues/116 648403834 MDEyOklzc3VlQ29tbWVudDY0ODQwMzgzNA== 9599 2020-06-23T20:36:29Z 2020-06-23T20:36:29Z OWNER Should go in this section https://sqlite-utils.readthedocs.io/en/stable/python-api.html#introspection - under `.columns_dict` and before `.foreign_keys`.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
644122661
https://github.com/simonw/datasette/issues/694#issuecomment-648296323 https://api.github.com/repos/simonw/datasette/issues/694 648296323 MDEyOklzc3VlQ29tbWVudDY0ODI5NjMyMw== 3903726 2020-06-23T17:10:51Z 2020-06-23T17:10:51Z NONE @simonw Did you find the reason? I had similar situation and I check this on millions ways. I am sure app doesn't consume such memory. I was trying the app with: `docker run --rm -it -p 80:80 -m 128M foo` I was watching app with `docker stats`. Even limited memory by `CMD ["java", "-Xms60M", "-Xmx60M", "-jar", "api.jar"]`. Checked memory usage by app in code and print bash commands. The app definitely doesn't use this memory. Also doesn't write files. Only one solution is to change memory to 512M. It is definitely something wrong with `cloud run`. I even did special app for testing this. It looks like when I cross very small amount of code / memory / app size in random when, then memory needs grow +hundreds. Nothing make sense here. Especially it works everywhere expect cloud run. Please let me know if you discovered something more.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
576582604
https://github.com/simonw/datasette/issues/859#issuecomment-648234787 https://api.github.com/repos/simonw/datasette/issues/859 648234787 MDEyOklzc3VlQ29tbWVudDY0ODIzNDc4Nw== 9599 2020-06-23T15:22:51Z 2020-06-23T15:22:51Z OWNER I wonder if this is a SQLite caching issue then? Datasette has a configuration option for this but I haven't spent much time experimenting with it so I don't know how much of an impact it can have: https://datasette.readthedocs.io/en/stable/config.html#cache-size-kb
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-648232645 https://api.github.com/repos/simonw/datasette/issues/859 648232645 MDEyOklzc3VlQ29tbWVudDY0ODIzMjY0NQ== 3243482 2020-06-23T15:19:53Z 2020-06-23T15:19:53Z CONTRIBUTOR The issue seems to appear sporadically, like when I return to database page after a while, during which some records have been added to the database. I've just visited database, page first visit took ~10s, consecutive visits took 0.3s.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-648163272 https://api.github.com/repos/simonw/datasette/issues/859 648163272 MDEyOklzc3VlQ29tbWVudDY0ODE2MzI3Mg== 9599 2020-06-23T13:52:23Z 2020-06-23T13:52:23Z OWNER I'm chunking inserts at 100 at a time right now: https://github.com/simonw/sqlite-utils/blob/4d9a3204361d956440307a57bd18c829a15861db/sqlite_utils/db.py#L1030 I think the performance is more down to using Faker to create the test data - generating millions of entirely fake, randomized records takes a fair bit of time.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-647925594 https://api.github.com/repos/simonw/datasette/issues/859 647925594 MDEyOklzc3VlQ29tbWVudDY0NzkyNTU5NA== 3243482 2020-06-23T05:55:21Z 2020-06-23T06:28:29Z CONTRIBUTOR Hmm, not seeing the problem now. I've removed the commented out sections in `database.py` and restarted the process. Database page now loads in <250ms. I have couple of workers that check some pages regularly and scrape new content and save to the DB. Could it be that datasette tries to recount tables every time database size changes? Normally it keeps a count cache, but as DB gets updated so often (new content every 5 min or so) it's practically recounting every time I go to the database page? EDIT: It turns out it doesn't hold cache with mutable databases. I'll update the issue with more findings and a better way to reproduce the problem if I encounter it again.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-647936117 https://api.github.com/repos/simonw/datasette/issues/859 647936117 MDEyOklzc3VlQ29tbWVudDY0NzkzNjExNw== 3243482 2020-06-23T06:25:17Z 2020-06-23T06:25:17Z CONTRIBUTOR > > > ``` > sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50 > ``` > > Looks like that will take 35 minutes to run (it's not a particularly fast tool). Try chunking write operations into batches every 1000 records or so.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-647935300 https://api.github.com/repos/simonw/datasette/issues/859 647935300 MDEyOklzc3VlQ29tbWVudDY0NzkzNTMwMA== 3243482 2020-06-23T06:23:01Z 2020-06-23T06:23:01Z CONTRIBUTOR > You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes. Ah that was a typo, I meant 50k.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-647923666 https://api.github.com/repos/simonw/datasette/issues/859 647923666 MDEyOklzc3VlQ29tbWVudDY0NzkyMzY2Ng== 3243482 2020-06-23T05:49:31Z 2020-06-23T05:49:31Z CONTRIBUTOR I think I should mention that having FTS on all tables mean I have 5 visible, 25 hidden (FTS) tables displayed on database page.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-647922203 https://api.github.com/repos/simonw/datasette/issues/859 647922203 MDEyOklzc3VlQ29tbWVudDY0NzkyMjIwMw== 3243482 2020-06-23T05:44:58Z 2020-06-23T05:44:58Z CONTRIBUTOR I'm seeing the problem on database page. Index page and table page runs quite fast. - Tables have <10 columns (`id`, `url`, `title`, `body_html`, `date`, `author`, `meta` (for keeping unstructured json)). I've added index on `date` columns (using `sqlite-utils`) in addition to the index present on `id` columns. - All tables have FTS enabled on `text` and `varchar` columns (`title`, `body_html` vs) to speed up searching. - There are couple of tables related with foreign keys (think a thread in a forum and posts in that thread, related with `thread_id`)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-647894903 https://api.github.com/repos/simonw/datasette/issues/859 647894903 MDEyOklzc3VlQ29tbWVudDY0Nzg5NDkwMw== 9599 2020-06-23T04:07:59Z 2020-06-23T04:07:59Z OWNER Just to check: are you seeing the problem on this page: https://latest.datasette.io/fixtures (the database page) - or this page (the table page): https://latest.datasette.io/fixtures/compound_three_primary_keys If it's the table page then the problem may well be #862.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/596#issuecomment-647893140 https://api.github.com/repos/simonw/datasette/issues/596 647893140 MDEyOklzc3VlQ29tbWVudDY0Nzg5MzE0MA== 9599 2020-06-23T03:59:51Z 2020-06-23T03:59:51Z OWNER Related: #862 - a time limit on the total time spent considering suggested facets for a table.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
507454958
https://github.com/simonw/datasette/issues/862#issuecomment-647892930 https://api.github.com/repos/simonw/datasette/issues/862 647892930 MDEyOklzc3VlQ29tbWVudDY0Nzg5MjkzMA== 9599 2020-06-23T03:58:48Z 2020-06-23T03:58:48Z OWNER Should this be controlled be a separate configuration setting? I'm inclined to say no - I think instead I'll set the limit to be 10 * whatever `facet_suggest_time_limit_ms` is.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
643510821
https://github.com/simonw/datasette/issues/859#issuecomment-647890619 https://api.github.com/repos/simonw/datasette/issues/859 647890619 MDEyOklzc3VlQ29tbWVudDY0Nzg5MDYxOQ== 9599 2020-06-23T03:48:21Z 2020-06-23T03:48:21Z OWNER sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50 Looks like that will take 35 minutes to run (it's not a particularly fast tool).
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/859#issuecomment-647890378 https://api.github.com/repos/simonw/datasette/issues/859 647890378 MDEyOklzc3VlQ29tbWVudDY0Nzg5MDM3OA== 9599 2020-06-23T03:47:19Z 2020-06-23T03:47:19Z OWNER I generated a 600MB database using [sqlite-generate](https://github.com/simonw/sqlite-generate) just now - with 100 tables at 100,00 rows and 3 tables at 1,000,000 rows - and performance of the database page was fine, 250ms. Those tables only had 4 columns each though. You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642572841
https://github.com/simonw/datasette/issues/861#issuecomment-647889674 https://api.github.com/repos/simonw/datasette/issues/861 647889674 MDEyOklzc3VlQ29tbWVudDY0Nzg4OTY3NA== 9599 2020-06-23T03:44:17Z 2020-06-23T03:44:17Z OWNER https://github.com/simonw/sqlite-generate is now ready to be used - see also https://pypi.org/project/sqlite-generate/
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642652808
https://github.com/simonw/datasette/issues/861#issuecomment-647822757 https://api.github.com/repos/simonw/datasette/issues/861 647822757 MDEyOklzc3VlQ29tbWVudDY0NzgyMjc1Nw== 9599 2020-06-22T23:40:43Z 2020-06-22T23:40:43Z OWNER I started building that tool here: https://github.com/simonw/sqlite-generate (I built a new cookiecutter template for that too, https://github.com/simonw/click-app)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642652808
https://github.com/simonw/datasette/issues/838#issuecomment-647803394 https://api.github.com/repos/simonw/datasette/issues/838 647803394 MDEyOklzc3VlQ29tbWVudDY0NzgwMzM5NA== 6289012 2020-06-22T22:36:34Z 2020-06-22T22:36:34Z NONE I also am seeing the same issue with an Apache setup (same even w/o `ProxyPassReverse`, though I typically use it as @tsibley stated). But also want to say thanks for a great tool (this issue not withstanding)!
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
637395097
https://github.com/simonw/datasette/issues/861#issuecomment-647266979 https://api.github.com/repos/simonw/datasette/issues/861 647266979 MDEyOklzc3VlQ29tbWVudDY0NzI2Njk3OQ== 9599 2020-06-22T04:26:25Z 2020-06-22T04:26:25Z OWNER I think this is a separate Click utility. I'm going to call it `sqlite-generate`.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
642652808
https://github.com/simonw/datasette/issues/687#issuecomment-647258199 https://api.github.com/repos/simonw/datasette/issues/687 647258199 MDEyOklzc3VlQ29tbWVudDY0NzI1ODE5OQ== 9599 2020-06-22T03:55:20Z 2020-06-22T03:55:20Z OWNER https://datasette.readthedocs.io/en/latest/testing_plugins.html
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
572896293
Powered by Datasette · Query took 55.628ms · About: github-to-sqlite