issue_comments

Last thing is to replace sanic.response:

• response.text("")
• response.html()
• response.redirect(path)
• response.HTTPResponse

Implementations here: https://github.com/huge-success/sanic/blob/0.7.0/sanic/response.py#L175-L285

For the request object.... what are the fields of it I actually use?

• request.url
• request.query_string
• request.path
• request.method
• request.args
• request.raw_args

ALL of those are things that can be derived from the scope - so I think my new Request class (in utils/asgi.py) is just going to be a wrapper around a scope.

You can use a database query against sqlite_master to find all the FTS tables:

http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+name+from+sqlite_master+where+name+like+%22%25_fts%22+and+type+%3D+%22table%22

select name from sqlite_master where name like "%_fts" and type = "table"

You could then construct a crafty UNION query to get results back from all of those tables at once:

http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+%27PlantType_value_fts%27%2C+rowid%2C+value+from+PlantType_value_fts+where+PlantType_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCareAssistant_value_fts%27%2C+rowid%2C+value+from+qCareAssistant_value_fts+where+qCareAssistant_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCaretaker_value_fts%27%2C+rowid%2C+value+from+qCaretaker_value_fts+where+qCaretaker_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qLegalStatus_value_fts%27%2C+rowid%2C+value+from+qLegalStatus_value_fts+where+qLegalStatus_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSiteInfo_value_fts%27%2C+rowid%2C+value+from+qSiteInfo_value_fts+where+qSiteInfo_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSpecies_value_fts%27%2C+rowid%2C+value+from+qSpecies_value_fts+where+qSpecies_value_fts+match+%3Asearch&search=fi%2A

(I'm searching for fi* here to demonstrate wildcards)

The problem, as discussed earlier, is relevance: there's no way to compare the scores you're getting across different tables, so you won't be able to order by anything.

I suggest trying a full path in ExecStart like this:

ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 /home/chris/digital-library/databases/*.db --cors --metadata /home/chris/digital-library/metadata.json

That should eliminate the chance of some kind of path confusion.

The InvalidUsage exception is thrown by Sanic when it gets an unhandled request - usually a HEAD. It was added in efbb4e83374a2c795e436c72fa79f70da72309b8

I'm going to replace it with specific handling for HEAD requests plus a unit test.

The big one: Replace Sanic request and response objects with my own classes, so I can remove Sanic dependency

@russss

Thanks, just one more thing.

I edited datasette.service:

``` [Unit] Description=Datasette After=network.target

[Service] Type=simple User=chris WorkingDirectory=/home/chris/digital-library ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 databases/*.db --cors --metadata metadata.json Restart=on-failure

[Install] WantedBy=multi-user.target ```

Then ran:

$ sudo systemctl daemon-reload $ sudo systemctl enable datasette $ sudo systemctl start datasette

But the logs from journalctl show this datasette error:

Jun 23 23:31:41 ns331247 datasette[1771]: Error: Invalid value for "[FILES]...": Path "databases/*.db" does not exist. Jun 23 23:31:44 ns331247 datasette[1778]: Usage: datasette serve [OPTIONS] [FILES]... Jun 23 23:31:44 ns331247 datasette[1778]: Try "datasette serve --help" for help.

But the databases directory does exist in the directory specified by WorkingDirectory. Is this a datasette problem or did I write something incorrectly in the .service file?

@russss

Actually, here's what I've got in /etc/systemd/system/datasette.service

``` [Unit] Description=Datasette After=network.target

[Service] Type=simple User=chris WorkingDirectory=/home/chris/digital-library ExecStart=/home/chris/Env/datasette/lib/python3.7/site-packages/datasette serve -h 0.0.0.0 databases/*.db --cors --metadata metadata.json Restart=on-failure

[Install] WantedBy=multi-user.target ```

I ran: $ sudo systemctl daemon-reload $ sudo systemctl enable datasette $ sudo systemctl start datasette Then I ran: $ journalctl -u datasette -f

Got this message.

Hint: You are currently not seeing messages from other users and the system. Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages. Pass -q to turn off this notice. -- Logs begin at Thu 2019-06-20 00:05:23 CEST. -- Jun 22 19:55:57 ns331247 systemd[16176]: datasette.service: Failed to execute command: Permission denied Jun 22 19:55:57 ns331247 systemd[16176]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied Jun 22 19:55:57 ns331247 systemd[16184]: datasette.service: Failed to execute command: Permission denied Jun 22 19:55:57 ns331247 systemd[16184]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied Jun 22 19:55:58 ns331247 systemd[16186]: datasette.service: Failed to execute command: Permission denied Jun 22 19:55:58 ns331247 systemd[16186]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied Jun 22 19:55:58 ns331247 systemd[16190]: datasette.service: Failed to execute command: Permission denied Jun 22 19:55:58 ns331247 systemd[16190]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied Jun 22 19:55:58 ns331247 systemd[16191]: datasette.service: Failed to execute command: Permission denied Jun 22 19:55:58 ns331247 systemd[16191]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied When I go to the address for my server, I am met with the standard "Welcome to nginx" message:

``` Welcome to nginx! If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org. Commercial support is available at nginx.com.

Thank you for using nginx. ```

Very cool, thank you.

Using http://search-24ways.herokuapp.com as an example, let's say I want to search all FTS columns in all tables in all databases for the word "web."

Here's a link to the query I'd need to run to search "web" on FTS columns in articles table of the 24ways database.

And here's a link to the JSON version of the above result. I'd like to get the JSON result of that query for each FTS table of each database in my datasette project.

Is it possible in Javascript to automate the construction of query URLs like the one I linked, but for every FTS table in my datasette project?

Replacement for @app.listener("before_server_start") - this is what the ASGI lifespan protocol is for.

I know Uvicorn supports this because it keeps saying ASGI 'lifespan' protocol appears unsupported on the console.

I think the solution here will be to introduce another ASGI wrapper class similar to AsgiTracer. I'll model this on the example in the ASGI lifespan spec.

These plugins will need access to configuration and the ability to execute SQL - which means we need to make the datasette instance available to them.

The Starlette lint and test scripts do this, and also apply autoflake to remove any unnecessary imports: https://github.com/encode/starlette/tree/8c8cc2ec0a5cb834a9a15b871ae8b480503abb67/scripts

OK, for Get ?_trace=1 working again. The old code lives in two places:

https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L546-L560

And then:

https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L653-L672

So it's stashing something on the request to tell the rest of the code it should be tracing, then using that collected data from the request to add information to the final body.

One possible shape for the replacement is a new ASGI middleware that wraps everything else. We don't have a mutable request object here though, so we will need to untangle this entirely from the request object.

Also tricky is that in ASGI land we handle streams - we don't usually wait around for the entire response body to be compiled for us. This means the code that modifies the response (adding to the JSON or appending inside the </body>) needs to be reconsidered.

As usual, Starlette seems to have figured this out: https://github.com/encode/starlette/blob/8c8cc2ec0a5cb834a9a15b871ae8b480503abb67/starlette/middleware/gzip.py

Another bug: JSON is being served without a content-type header: ``` ~ $ curl -i 'http://127.0.0.1:8001/fivethirtyeight/ahca-polls%2Fahca_polls.json' HTTP/1.1 200 OK date: Sun, 23 Jun 2019 16:04:01 GMT server: uvicorn referrer-policy: no-referrer transfer-encoding: chunked

{"database": "fivethirtyeight", "table": "ahca-polls/ahca_polls", ... ```

Weird new bug: http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv?_format=json is downloading CSV for me now.

https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json does the right thing.

One interesting way to approach this could be to do it entirely in JavaScript. I've had a lot of success building small apps on top of Datasette's JavaScript API - I wrote up one example here: https://24ways.org/2018/fast-autocomplete-search-for-your-website/

Once #272 is done I'll be adding a plugin hook that allows plugins to define entirely new pages within the Datasette application, which may also be a good way to work on this.

Mystery solved: that's because I'm constructing my own scope object and testing via ApplicationCommunicator rather than exercising Uvicorn directly.

https://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/fixtures.py#L42-L57

I don't want to introduce the complexity of launching a real Uvicorn as part of the tests, so I guess I'll have to carefully update my ApplicationCommunicator test harness to more correctly emulate real life.

This is strange: on my local machine http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv is returning a 404 BUT there's a test for that which is passing under pytest:

https://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/test_api.py#L721-L727

And now the tests are all passing!

Still to do:

• Use raw_path so table names containing / can work correctly
• Get ?_trace=1 working again
• Replacement for @app.listener("before_server_start")
• Replace Sanic request object with my own request class, so I can remove Sanic dependency

I'm going to move the remaining work into a pull request.

OK, it's beginning to shape up now. Next steps:

• [x] Static file support (including for plugins) - plus tests
• [x] Streaming support so the CSV tests will pass
• [x] Ability to download the database file
• [x] Implement missing-slash redirects

Tests are failing on Python 3.5: https://travis-ci.org/simonw/datasette/jobs/549380098 - error is TypeError: the JSON object must be str, not 'bytes'

I also need to actually take advantage of raw_path such that pages like https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators can be correctly served.

All of the tests are now passing!

I still need a solution for this:

https://github.com/simonw/datasette/blob/5bd510b01adae3f719e4426b9bfbc346a946ba5c/datasette/app.py#L706-L714

I think the answer is ASGI lifespan, which is supported by Uvicorn. https://asgi.readthedocs.io/en/latest/specs/lifespan.html#startup

Next test to fix (because by new test harness doesn't actually obey the allow_redirects= parameter): ``` ___ test_database_page_redirects_with_url_hash _____

app_client_with_hash = <tests.fixtures.TestClient object at 0x10981f240>

def test_database_page_redirects_with_url_hash(app_client_with_hash):
    response = app_client_with_hash.get("/fixtures", allow_redirects=False)
    assert response.status == 302
    response = app_client_with_hash.get("/fixtures")

  assert "fixtures" in response.text

E AssertionError: assert 'fixtures' in '' E + where '' = <tests.fixtures.TestResponse object at 0x10981f550>.text ```

CSV tests all pass as of https://github.com/simonw/datasette/commit/ff9efa668ebc33f17ef9b30139960e29906a18fb

This code could be a lot neater though. At the very least I'm going to refactor datasette/utils.py into a datasette/utils package and put all of my new ASGI utilities in datasette/utils/asgi.py

The way I implemented streaming on top of a writer object (inspired by Sanic) is a bit of a weird hack. I think I'd rather use an abstraction where my view functions can yield chunks of body data.

The mechanism I described here - having a tests/example_plugin folder - is probably the right solution for #517

See also #226 - "Unit tests for installable plugins"

Published an in-progress demo:

datasette publish now fixtures.db -n datasette-asgi-early-demo --branch=asgi

Here it is: https://datasette-asgi-early-demo-qahhxctqpw.now.sh/

Lots still to do:

• Static files are not being served
• Streaming CSV files don't work
• Tests all fail
• Some URLs (e.g. the 'next' link on tables) are incorrect

I'm going to work on getting the unit test framework to be ASGI-compatible next.