issue_comments

42 rows where issue = 324188953 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

author_association

issue

  • Port Datasette to ASGI · 42
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
504857097 https://github.com/simonw/datasette/issues/272#issuecomment-504857097 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDg1NzA5Nw== simonw 9599 2019-06-24T04:54:15Z 2019-06-24T04:54:15Z OWNER

I wrote about this on my blog: https://simonwillison.net/2019/Jun/23/datasette-asgi/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504844339 https://github.com/simonw/datasette/issues/272#issuecomment-504844339 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDg0NDMzOQ== simonw 9599 2019-06-24T03:33:06Z 2019-06-24T03:33:06Z OWNER

It's alive! Here's the first deployed version: https://a559123.datasette.io/

You can confirm it's running under ASGI by viewing https://a559123.datasette.io/-/versions and looking for the "asgi" key.

Compare to the last version of master running on Sanic here: http://aa91112.datasette.io/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504761039 https://github.com/simonw/datasette/issues/272#issuecomment-504761039 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDc2MTAzOQ== simonw 9599 2019-06-23T15:15:41Z 2019-06-23T15:18:36Z OWNER

And now the tests are all passing!

Still to do:

  • Use raw_path so table names containing / can work correctly
  • Get ?_trace=1 working again
  • Replacement for @app.listener("before_server_start")
  • Replace Sanic request object with my own request class, so I can remove Sanic dependency
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504761165 https://github.com/simonw/datasette/issues/272#issuecomment-504761165 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDc2MTE2NQ== simonw 9599 2019-06-23T15:17:07Z 2019-06-23T15:17:07Z OWNER

I'm going to move the remaining work into a pull request.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504716988 https://github.com/simonw/datasette/issues/272#issuecomment-504716988 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDcxNjk4OA== simonw 9599 2019-06-23T03:43:46Z 2019-06-23T15:15:26Z OWNER

OK, it's beginning to shape up now. Next steps:

  • Static file support (including for plugins) - plus tests
  • Streaming support so the CSV tests will pass
  • Ability to download the database file
  • Implement missing-slash redirects
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504760061 https://github.com/simonw/datasette/issues/272#issuecomment-504760061 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDc2MDA2MQ== simonw 9599 2019-06-23T15:02:52Z 2019-06-23T15:02:52Z OWNER

Tests are failing on Python 3.5: https://travis-ci.org/simonw/datasette/jobs/549380098 - error is TypeError: the JSON object must be str, not 'bytes'

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504759842 https://github.com/simonw/datasette/issues/272#issuecomment-504759842 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDc1OTg0Mg== simonw 9599 2019-06-23T15:00:06Z 2019-06-23T15:00:06Z OWNER

I also need to actually take advantage of raw_path such that pages like https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators can be correctly served.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504759683 https://github.com/simonw/datasette/issues/272#issuecomment-504759683 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDc1OTY4Mw== simonw 9599 2019-06-23T14:57:50Z 2019-06-23T14:57:50Z OWNER

All of the tests are now passing!

I still need a solution for this:

https://github.com/simonw/datasette/blob/5bd510b01adae3f719e4426b9bfbc346a946ba5c/datasette/app.py#L706-L714

I think the answer is ASGI lifespan, which is supported by Uvicorn. https://asgi.readthedocs.io/en/latest/specs/lifespan.html#startup

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504754552 https://github.com/simonw/datasette/issues/272#issuecomment-504754552 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDc1NDU1Mg== simonw 9599 2019-06-23T13:53:39Z 2019-06-23T13:53:39Z OWNER

Next test to fix (because by new test harness doesn't actually obey the allow_redirects= parameter):

_____________ test_database_page_redirects_with_url_hash _____________

app_client_with_hash = <tests.fixtures.TestClient object at 0x10981f240>

    def test_database_page_redirects_with_url_hash(app_client_with_hash):
        response = app_client_with_hash.get("/fixtures", allow_redirects=False)
        assert response.status == 302
        response = app_client_with_hash.get("/fixtures")
>       assert "fixtures" in response.text
E       AssertionError: assert 'fixtures' in ''
E        +  where '' = <tests.fixtures.TestResponse object at 0x10981f550>.text
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504754433 https://github.com/simonw/datasette/issues/272#issuecomment-504754433 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDc1NDQzMw== simonw 9599 2019-06-23T13:51:53Z 2019-06-23T13:51:53Z OWNER

CSV tests all pass as of https://github.com/simonw/datasette/commit/ff9efa668ebc33f17ef9b30139960e29906a18fb

This code could be a lot neater though. At the very least I'm going to refactor datasette/utils.py into a datasette/utils package and put all of my new ASGI utilities in datasette/utils/asgi.py

The way I implemented streaming on top of a writer object (inspired by Sanic) is a bit of a weird hack. I think I'd rather use an abstraction where my view functions can yield chunks of body data.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504711468 https://github.com/simonw/datasette/issues/272#issuecomment-504711468 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDcxMTQ2OA== simonw 9599 2019-06-23T01:36:33Z 2019-06-23T01:36:33Z OWNER

Published an in-progress demo:

datasette publish now fixtures.db -n datasette-asgi-early-demo --branch=asgi

Here it is: https://datasette-asgi-early-demo-qahhxctqpw.now.sh/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504710331 https://github.com/simonw/datasette/issues/272#issuecomment-504710331 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDcxMDMzMQ== simonw 9599 2019-06-23T01:08:45Z 2019-06-23T01:08:45Z OWNER

Lots still to do:

  • Static files are not being served
  • Streaming CSV files don't work
  • Tests all fail
  • Some URLs (e.g. the 'next' link on tables) are incorrect

I'm going to work on getting the unit test framework to be ASGI-compatible next.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
504697742 https://github.com/simonw/datasette/issues/272#issuecomment-504697742 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwNDY5Nzc0Mg== simonw 9599 2019-06-22T20:55:59Z 2019-06-22T20:56:22Z OWNER

Getting this to work with both Sanic AND ASGI at the same time (via the classes described previously with an --asgi command-line option) is proving way trickier than I expected, mainly because of the complexity of the current Datasette.app() method.

I'm going to drop the compatibility path for a bit and see if I can make progress on a pure-ASGI port.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
503369834 https://github.com/simonw/datasette/issues/272#issuecomment-503369834 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMzM2OTgzNA== simonw 9599 2019-06-19T01:26:24Z 2019-06-19T01:26:24Z OWNER

I need to be able to define the URL routes once and have them work for both Sanic and ASGI.

I'm going to extract the web application bits out of the Datasette class into a DatasetteServer class. Then I can have a add_route() method on that class, then have DatasetteSanic and DatasetteAsgi subclasses which redefine that method.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
502393107 https://github.com/simonw/datasette/issues/272#issuecomment-502393107 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMjM5MzEwNw== simonw 9599 2019-06-15T19:25:54Z 2019-06-19T01:20:14Z OWNER

OK, time for a solid implementation plan.

As soon as https://github.com/django/asgiref/pull/92 is merged (hopefully very soon) the ASGI spec will have support for an optional raw_path - which means we can continue to use table%2Fnames with embedded / without being unable to tell if a path has been decoded or not.

Steps to implement:

Refactor classes, then add .asgi() method to BaseView

Add a .asgi(self, scope, receive, send) method to my base view class. This will expose an ASGI interface to the outside world: the method itself will construct a request object and call the existing .get() method.

My only true shared base class is actually RenderMixin because the IndexView doesn't extend BaseView. I'm going to refactor the class hierarchy a bit here - AsgiView will be my top level class with the .asgi() method on it. RenderMixin will be renamed BaseView(AsgiView), while existing BaseView will be renamed DataView(BaseView) since it mainly exists to introduce the handy .data() abstraction.

So...

  • AsgiView - has .asgi() method, extends Sanic HTTPMethodView (for the moment)
  • BaseView(AsgiView) - defines utility methods currently on RenderMixin
  • IndexView(BaseView) - the current IndexView
  • DataView(BaseView) - defines the utilities currently on BaseView, including data()
  • Everything else subclasses DataView

Extract routing logic out into a new DatasetteView

I considered calling this RouteView, but one of the goals of this project is to allow other ASGI apps to import Datasette itself and reuse it as its own ASGI function.

So DatasetteView will subclass BaseView and will do all of the routing logic. That logic currently lives here:

https://github.com/simonw/datasette/blob/aa911122feab13f8e65875c98edb00fd3832b7b8/datasette/app.py#L594-L640

For tests: Implement a version of app_client.get() that calls ASGI instead

Almost all of the unit tests currently use app_client.get("/path..."). I want to be able to run tests against both ASGI and existing-Sanic, so for the moment I'm going to teach app_client.get() to use ASGI instead but only in the presence of a new environment variable. I can then have Travis run the tests twice - once with that environement variable and once without.

Make datasette serve --asgi run ASGI and uvicorn

Uvicorn supports Python 3.5 again as of https://github.com/encode/uvicorn/issues/330 - so it's going to be the new dependency for Datasette.

Do some comparative testing of ASGI and non-ASGI

Just some sanity checking to make sure there aren't any weird issues.

Final step: refactor out Sanic

Hopefully this will just involve changes being made to the AsgiView base class, since that subclasses Sanic HTTPMethodView.

Bonus: It looks like dropping Sanic as a dependency in favour of Uvicorn should give us Windows support! https://github.com/encode/uvicorn/issues/82

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
503351966 https://github.com/simonw/datasette/issues/272#issuecomment-503351966 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMzM1MTk2Ng== simonw 9599 2019-06-18T23:45:17Z 2019-06-18T23:45:17Z OWNER

Uvicorn 0.8.1 is our and supports raw_path!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
503195217 https://github.com/simonw/datasette/issues/272#issuecomment-503195217 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMzE5NTIxNw== simonw 9599 2019-06-18T15:46:31Z 2019-06-18T15:54:18Z OWNER

How should file serving work?

Starlette and Sanic both use aiofiles - https://github.com/Tinche/aiofiles - which is a small wrapper around file operations which runs them all in an executor thread. It doesn't have any C dependencies so it looks like a good option. Quart uses it too.

aiohttp does things differently: it has an implementation based on sendfile with an alternative fallback which reads chunks from a file object and yields them one chunk at a time,

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
502466466 https://github.com/simonw/datasette/issues/272#issuecomment-502466466 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMjQ2NjQ2Ng== simonw 9599 2019-06-16T16:28:10Z 2019-06-16T16:28:10Z OWNER

I have an open pull request to Uvicorn with an implementation of raw_path: https://github.com/encode/uvicorn/pull/372

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
502401078 https://github.com/simonw/datasette/issues/272#issuecomment-502401078 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMjQwMTA3OA== simonw 9599 2019-06-15T21:35:26Z 2019-06-15T21:35:26Z OWNER

Started sketching out the router in the asgi branch: https://github.com/simonw/datasette/commit/7cdc55c6836fe246b1ca8a13a965a39991c9ffec

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
502395689 https://github.com/simonw/datasette/issues/272#issuecomment-502395689 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMjM5NTY4OQ== simonw 9599 2019-06-15T20:05:26Z 2019-06-15T20:05:26Z OWNER

For the routing component: I'm going to base my implementation on the one from Django Channels.

https://github.com/django/channels/blob/507cb54fcb36df63282dd19653ea743036e7d63c/channels/routing.py#L123-L149

Documented here: https://channels.readthedocs.io/en/latest/topics/routing.html#urlrouter

Particularly relevant: my view classes need access to the components that were already parsed out of the URL by the router. I'm going to copy the Django Channels mechanism of stashing those in scope["url_route"]["kwargs"].

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
502394420 https://github.com/simonw/datasette/issues/272#issuecomment-502394420 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMjM5NDQyMA== simonw 9599 2019-06-15T19:45:46Z 2019-06-15T19:45:46Z OWNER

For reference, here's some WIP code I wrote last year against the old ASGI 2 spec: https://github.com/simonw/datasette/commit/4fd36ba2f3f91da7258859808616078e3464fb97

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
502393267 https://github.com/simonw/datasette/issues/272#issuecomment-502393267 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDUwMjM5MzI2Nw== simonw 9599 2019-06-15T19:28:27Z 2019-06-15T19:28:27Z OWNER

I'll probably revert 9fdb47ca952b93b7b60adddb965ea6642b1ff523 from https://github.com/simonw/datasette/issues/272#issuecomment-494192779 since I won't need it now that ASGI is getting raw_path support.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
494297022 https://github.com/simonw/datasette/issues/272#issuecomment-494297022 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQ5NDI5NzAyMg== tomchristie 647359 2019-05-21T08:39:17Z 2019-05-21T08:39:17Z NONE

Useful context stuff:

ASGI decodes %2F encoded slashes in URLs automatically

raw_path for ASGI looks to be under consideration: https://github.com/django/asgiref/issues/87

uvicorn doesn't support Python 3.5

That was an issue specifically against the <=3.5.2 minor point releases of Python, now resolved: https://github.com/encode/uvicorn/issues/330 👍

Starlette for things like form parsing - but it's 3.6+ only!

Yeah - the bits that require 3.6 are anywhere with the "async for" syntax. If it wasn't for that I'd downport it, but that one's a pain. It's the one bit of syntax to watch out for if you're looking to bring any bits of implementation across to Datasette.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
494192779 https://github.com/simonw/datasette/issues/272#issuecomment-494192779 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQ5NDE5Mjc3OQ== simonw 9599 2019-05-21T00:10:47Z 2019-05-21T00:10:47Z OWNER

https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 added decode_path_component() and encode_path_component() functions because ASGI decodes %2F encoded slashes in URLs automatically. The new encoding scheme looks like this:

"table/and/slashes" => "tableU+002FandU+002Fslashes"
"~table" => "U+007Etable"
"+bobcats!" => "U+002Bbobcats!"
"U+007Etable" => "UU+002B007Etable"

For background see this comment: https://github.com/django/asgiref/issues/51#issuecomment-450603464

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
494192163 https://github.com/simonw/datasette/issues/272#issuecomment-494192163 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQ5NDE5MjE2Mw== simonw 9599 2019-05-21T00:07:25Z 2019-05-21T00:07:25Z OWNER

Bah, I'd much rather depend on Starlette for things like form parsing - but it's 3.6+ only!

https://github.com/encode/starlette/blob/ab86530eddfcf56e0f7e5ca56f6ab69c15594a7d/setup.py#L39

Maybe I could require Python 3.6 or higher if you want to handle POST data? This would make my internals far too complicated though I think.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
494191738 https://github.com/simonw/datasette/issues/272#issuecomment-494191738 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQ5NDE5MTczOA== simonw 9599 2019-05-21T00:05:02Z 2019-05-21T00:05:02Z OWNER

While I'm not depending on Starlette any more I will need to instead depend on https://github.com/andrew-d/python-multipart for POST form parsing - as used by Starlette here https://github.com/encode/starlette/blob/ab86530eddfcf56e0f7e5ca56f6ab69c15594a7d/starlette/requests.py#L178-L193

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
494191378 https://github.com/simonw/datasette/issues/272#issuecomment-494191378 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQ5NDE5MTM3OA== simonw 9599 2019-05-21T00:02:48Z 2019-05-21T00:02:48Z OWNER

I said earlier that I only need to support GET - I actually need to be able to support POST too, mainly to support plugins (e.g. a plugin that allows authenticated login before you can view Datasette, but potentially also plugins that let you write data directly to SQLite as well).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
494190922 https://github.com/simonw/datasette/issues/272#issuecomment-494190922 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQ5NDE5MDkyMg== simonw 9599 2019-05-21T00:00:40Z 2019-05-21T00:01:09Z OWNER

Wow, this issue has been open for a full year now!

I've been thinking about this a lot. I've decided I want Datasette to use ASGI 3.0 internally with no dependencies on anything else - then I want the option to run Datasette under both daphne and uvicorn - because uvicorn doesn't support Python 3.5 but Datasette still needs to (primarily for Glitch), and daphne works with 3.5.

So I'm going to try to go the following route:

  • Every Datasette view becomes an ASGI app
  • The Datasette application itself is an ASGI app that routes to those views
  • When you pip install datasette you get Daphne as a dependency (I'd like you to be able to opt-out of installing Daphne, I'm not yet sure how that would work)
  • A new asgi_serve plugin hook allows a plugin to serve Datasette using uvicorn (or hypercorn) instead
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
418695115 https://github.com/simonw/datasette/issues/272#issuecomment-418695115 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQxODY5NTExNQ== tomchristie 647359 2018-09-05T11:21:25Z 2018-09-05T11:21:25Z NONE

Some notes:

  • Starlette just got a bump to 0.3.0 - there's some renamings in there. It's got enough functionality now that you can treat it either as a framework or as a toolkit. Either way the component design is all just here's an ASGI app all the way through.
  • Uvicorn got a bump to 0.3.3 - Removed some cyclical references that were causing garbage collection to impact performance. Ought to be a decent speed bump.
  • Wrt. passing config - Either use a single envvar that points to a config, or use multiple envvars for the config. Uvicorn could get a flag to read a .env file, but I don't see ASGI itself having a specific interface there.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
408478935 https://github.com/simonw/datasette/issues/272#issuecomment-408478935 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQwODQ3ODkzNQ== simonw 9599 2018-07-27T17:00:08Z 2018-07-27T17:00:08Z OWNER

Refs
https://github.com/encode/uvicorn/issues/168

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
408105251 https://github.com/simonw/datasette/issues/272#issuecomment-408105251 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQwODEwNTI1MQ== simonw 9599 2018-07-26T13:54:06Z 2018-07-26T13:54:06Z OWNER

Tom shipped my fix for that bug already, so https://datasette-starlette-demo.now.sh/ is now serving CSS!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
408093480 https://github.com/simonw/datasette/issues/272#issuecomment-408093480 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQwODA5MzQ4MA== simonw 9599 2018-07-26T13:15:55Z 2018-07-26T13:46:40Z OWNER

I'm now hacking around with an initial version of this in the starlette branch.

Here's my work in progress, deployed using datasette publish now fixtures.db -n datasette-starlette-demo --branch=starlette --extra-options="--asgi"

https://datasette-starlette-demo.now.sh/

Lots more work to do - the CSS isn't being served correctly for example, it's showing this error when I hit /-/static/app.css:

INFO: 127.0.0.1 - "GET /-/static/app.css HTTP/1.1" 200
ERROR: Exception in ASGI application
Traceback (most recent call last):
  File "/Users/simonw/Dropbox/Development/datasette/venv/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 363, in run_asgi
    result = await asgi(self.receive, self.send)
  File "/Users/simonw/Dropbox/Development/datasette/venv/lib/python3.6/site-packages/starlette/staticfiles.py", line 91, in __call__
    await response(receive, send)
  File "/Users/simonw/Dropbox/Development/datasette/venv/lib/python3.6/site-packages/starlette/response.py", line 180, in __call__
    {"type": "http.response.body", "body": chunk, "more_body": False}
  File "/Users/simonw/Dropbox/Development/datasette/venv/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 483, in send
    raise RuntimeError("Response content shorter than Content-Length")
RuntimeError: Response content shorter than Content-Length
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
408097719 https://github.com/simonw/datasette/issues/272#issuecomment-408097719 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQwODA5NzcxOQ== simonw 9599 2018-07-26T13:29:38Z 2018-07-26T13:29:38Z OWNER

It looks like that's a bug in Starlette - filed here: https://github.com/encode/starlette/issues/32

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
404514973 https://github.com/simonw/datasette/issues/272#issuecomment-404514973 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQwNDUxNDk3Mw== tomchristie 647359 2018-07-12T13:38:24Z 2018-07-12T13:38:24Z NONE

Okay. I reckon the latest version should have all the kinds of components you'd need:

Recently added ASGI components for Routing and Static Files support, as well as making few tweaks to make sure requests and responses are instantiated efficiently.

Don't have any redirect-to-slash / redirect-to-non-slash stuff out of the box yet, which it looks like you might miss.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
403959704 https://github.com/simonw/datasette/issues/272#issuecomment-403959704 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQwMzk1OTcwNA== simonw 9599 2018-07-10T20:44:47Z 2018-07-10T20:44:47Z OWNER

No cookies or sessions - no POST requests in fact, Datasette just cares about GET (path and querystring) and being able to return custom HTTP headers.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
400571521 https://github.com/simonw/datasette/issues/272#issuecomment-400571521 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQwMDU3MTUyMQ== tomchristie 647359 2018-06-27T07:30:07Z 2018-06-27T07:30:07Z NONE

I’m up for helping with this.

Looks like you’d need static files support, which I’m planning on adding a component for. Anything else obviously missing?

For a quick overview it looks very doable - the test client ought to me your test cases stay roughly the same.

Are you using any middleware or other components for the Sanic ecosystem? Do you use cookies or sessions at all?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
400166540 https://github.com/simonw/datasette/issues/272#issuecomment-400166540 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDQwMDE2NjU0MA== simonw 9599 2018-06-26T03:29:43Z 2018-06-26T03:29:43Z OWNER

This looks VERY relevant: https://github.com/encode/starlette

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
392118755 https://github.com/simonw/datasette/issues/272#issuecomment-392118755 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDM5MjExODc1NQ== simonw 9599 2018-05-25T16:56:40Z 2018-06-05T16:01:13Z OWNER

Thinking about this further, maybe I should embrace ASGI turtles-all-the-way-down and teach each datasette view class to take a scope to the constructor and act entirely as an ASGI component. Would be a nice way of diving deep into ASGI and I can add utility helpers for things like querystring evaluation as I need them.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
394764713 https://github.com/simonw/datasette/issues/272#issuecomment-394764713 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDM5NDc2NDcxMw== simonw 9599 2018-06-05T15:58:54Z 2018-06-05T16:00:40Z OWNER

https://github.com/encode/uvicorn/blob/572b5fe6c811b63298d5350a06b664839624c860/uvicorn/run.py#L63 is how you start a Uvicorn server from code as opposed to the uvicorn CLI

from uvicorn.run import UvicornServer
UvicornServer().run(app, host=host, port=port)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
394503399 https://github.com/simonw/datasette/issues/272#issuecomment-394503399 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDM5NDUwMzM5OQ== simonw 9599 2018-06-04T21:20:14Z 2018-06-04T21:20:14Z OWNER

Results of an extremely simple micro-benchmark comparing the two shows that uvicorn is at least as fast as Sanic (benchmarks a little faster with a very simple payload): https://gist.github.com/simonw/418950af178c01c416363cc057420851

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
394431323 https://github.com/simonw/datasette/issues/272#issuecomment-394431323 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDM5NDQzMTMyMw== simonw 9599 2018-06-04T17:17:37Z 2018-06-04T17:17:37Z OWNER

I built this ASGI debugging tool to help with this migration: https://asgi-scope.now.sh/fivethirtyeight-34d6604/most-common-name%2Fsurnames.json?foo=bar&bazoeuto=onetuh&a=.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953
391011268 https://github.com/simonw/datasette/issues/272#issuecomment-391011268 https://api.github.com/repos/simonw/datasette/issues/272 MDEyOklzc3VlQ29tbWVudDM5MTAxMTI2OA== simonw 9599 2018-05-22T14:28:12Z 2018-05-22T14:28:12Z OWNER

I think I can do this almost entirely within my existing BaseView class structure.

First, decouple the async data() methods by teaching them to take a querystring object as an argument instead of a Sanic request object. The get() method can then send that new object instead of a request.

Next teach the base class how to obey the ASGI protocol.

I should be able to get support for both Sanic and uvicorn/daphne working in the same codebase, which will make it easy to compare their performance.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Port Datasette to ASGI 324188953

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 44.781ms · About: github-to-sqlite