html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/514#issuecomment-504686266,https://api.github.com/repos/simonw/datasette/issues/514,504686266,MDEyOklzc3VlQ29tbWVudDUwNDY4NjI2Ng==,7936571,2019-06-22T17:58:50Z,2019-06-23T21:21:57Z,NONE,"@russss
Actually, here's what I've got in `/etc/systemd/system/datasette.service`
```
[Unit]
Description=Datasette
After=network.target
[Service]
Type=simple
User=chris
WorkingDirectory=/home/chris/digital-library
ExecStart=/home/chris/Env/datasette/lib/python3.7/site-packages/datasette serve -h 0.0.0.0 databases/*.db --cors --metadata metadata.json
Restart=on-failure
[Install]
WantedBy=multi-user.target
```
I ran:
```
$ sudo systemctl daemon-reload
$ sudo systemctl enable datasette
$ sudo systemctl start datasette
```
Then I ran:
`$ journalctl -u datasette -f`
Got this message.
```
Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
Pass -q to turn off this notice.
-- Logs begin at Thu 2019-06-20 00:05:23 CEST. --
Jun 22 19:55:57 ns331247 systemd[16176]: datasette.service: Failed to execute command: Permission denied
Jun 22 19:55:57 ns331247 systemd[16176]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
Jun 22 19:55:57 ns331247 systemd[16184]: datasette.service: Failed to execute command: Permission denied
Jun 22 19:55:57 ns331247 systemd[16184]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
Jun 22 19:55:58 ns331247 systemd[16186]: datasette.service: Failed to execute command: Permission denied
Jun 22 19:55:58 ns331247 systemd[16186]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
Jun 22 19:55:58 ns331247 systemd[16190]: datasette.service: Failed to execute command: Permission denied
Jun 22 19:55:58 ns331247 systemd[16190]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
Jun 22 19:55:58 ns331247 systemd[16191]: datasette.service: Failed to execute command: Permission denied
Jun 22 19:55:58 ns331247 systemd[16191]: datasette.service: Failed at step EXEC spawning /home/chris/Env/datasette/lib/python3.7/site-packages/datasette: Permission denied
```
When I go to the address for my server, I am met with the standard ""Welcome to nginx"" message:
```
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.
Thank you for using nginx.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459397625,
https://github.com/simonw/datasette/issues/272#issuecomment-504754433,https://api.github.com/repos/simonw/datasette/issues/272,504754433,MDEyOklzc3VlQ29tbWVudDUwNDc1NDQzMw==,9599,2019-06-23T13:51:53Z,2019-06-23T13:51:53Z,OWNER,"CSV tests all pass as of https://github.com/simonw/datasette/commit/ff9efa668ebc33f17ef9b30139960e29906a18fb
This code could be a lot neater though. At the very least I'm going to refactor `datasette/utils.py` into a `datasette/utils` package and put all of my new ASGI utilities in `datasette/utils/asgi.py`
The way I implemented streaming on top of a writer object (inspired by Sanic) is a bit of a weird hack. I think I'd rather use an abstraction where my view functions can yield chunks of body data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504754552,https://api.github.com/repos/simonw/datasette/issues/272,504754552,MDEyOklzc3VlQ29tbWVudDUwNDc1NDU1Mg==,9599,2019-06-23T13:53:39Z,2019-06-23T13:53:39Z,OWNER,"Next test to fix (because by new test harness doesn't actually obey the `allow_redirects=` parameter):
```
_____________ test_database_page_redirects_with_url_hash _____________
app_client_with_hash =
def test_database_page_redirects_with_url_hash(app_client_with_hash):
response = app_client_with_hash.get(""/fixtures"", allow_redirects=False)
assert response.status == 302
response = app_client_with_hash.get(""/fixtures"")
> assert ""fixtures"" in response.text
E AssertionError: assert 'fixtures' in ''
E + where '' = .text
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504759683,https://api.github.com/repos/simonw/datasette/issues/272,504759683,MDEyOklzc3VlQ29tbWVudDUwNDc1OTY4Mw==,9599,2019-06-23T14:57:50Z,2019-06-23T14:57:50Z,OWNER,"All of the tests are now passing!
I still need a solution for this:
https://github.com/simonw/datasette/blob/5bd510b01adae3f719e4426b9bfbc346a946ba5c/datasette/app.py#L706-L714
I think the answer is ASGI lifespan, which is supported by Uvicorn. https://asgi.readthedocs.io/en/latest/specs/lifespan.html#startup","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504759842,https://api.github.com/repos/simonw/datasette/issues/272,504759842,MDEyOklzc3VlQ29tbWVudDUwNDc1OTg0Mg==,9599,2019-06-23T15:00:06Z,2019-06-23T15:00:06Z,OWNER,I also need to actually take advantage of `raw_path` such that pages like https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators can be correctly served.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504760061,https://api.github.com/repos/simonw/datasette/issues/272,504760061,MDEyOklzc3VlQ29tbWVudDUwNDc2MDA2MQ==,9599,2019-06-23T15:02:52Z,2019-06-23T15:02:52Z,OWNER,"Tests are failing on Python 3.5: https://travis-ci.org/simonw/datasette/jobs/549380098 - error is `TypeError: the JSON object must be str, not 'bytes'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504761039,https://api.github.com/repos/simonw/datasette/issues/272,504761039,MDEyOklzc3VlQ29tbWVudDUwNDc2MTAzOQ==,9599,2019-06-23T15:15:41Z,2019-06-23T15:18:36Z,OWNER,"And now the tests are all passing!
Still to do:
* Use `raw_path` so table names containing `/` can work correctly
* Get ?_trace=1 working again
* Replacement for `@app.listener(""before_server_start"")`
* Replace Sanic request object with my own request class, so I can remove Sanic dependency","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504761165,https://api.github.com/repos/simonw/datasette/issues/272,504761165,MDEyOklzc3VlQ29tbWVudDUwNDc2MTE2NQ==,9599,2019-06-23T15:17:07Z,2019-06-23T15:17:07Z,OWNER,I'm going to move the remaining work into a pull request.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/pull/518#issuecomment-504762769,https://api.github.com/repos/simonw/datasette/issues/518,504762769,MDEyOklzc3VlQ29tbWVudDUwNDc2Mjc2OQ==,9599,2019-06-23T15:37:26Z,2019-06-23T15:37:26Z,OWNER,"This is strange: on my local machine `http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv` is returning a 404 BUT there's a test for that which is passing under pytest:
https://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/test_api.py#L721-L727","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504762887,https://api.github.com/repos/simonw/datasette/issues/518,504762887,MDEyOklzc3VlQ29tbWVudDUwNDc2Mjg4Nw==,9599,2019-06-23T15:38:58Z,2019-06-23T15:38:58Z,OWNER,"Mystery solved: that's because I'm constructing my own scope object and testing via `ApplicationCommunicator` rather than exercising Uvicorn directly.
https://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/fixtures.py#L42-L57
I don't want to introduce the complexity of launching a real Uvicorn as part of the tests, so I guess I'll have to carefully update my `ApplicationCommunicator` test harness to more correctly emulate real life.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/issues/498#issuecomment-504763919,https://api.github.com/repos/simonw/datasette/issues/498,504763919,MDEyOklzc3VlQ29tbWVudDUwNDc2MzkxOQ==,9599,2019-06-23T15:50:49Z,2019-06-23T15:50:49Z,OWNER,"One interesting way to approach this could be to do it entirely in JavaScript. I've had a lot of success building small apps on top of Datasette's JavaScript API - I wrote up one example here: https://24ways.org/2018/fast-autocomplete-search-for-your-website/
Once #272 is done I'll be adding a plugin hook that allows plugins to define entirely new pages within the Datasette application, which may also be a good way to work on this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",451513541,
https://github.com/simonw/datasette/pull/518#issuecomment-504765018,https://api.github.com/repos/simonw/datasette/issues/518,504765018,MDEyOklzc3VlQ29tbWVudDUwNDc2NTAxOA==,9599,2019-06-23T16:03:20Z,2019-06-23T16:03:20Z,OWNER,"Weird new bug: http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv?_format=json is downloading CSV for me now.
https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json does the right thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504765145,https://api.github.com/repos/simonw/datasette/issues/518,504765145,MDEyOklzc3VlQ29tbWVudDUwNDc2NTE0NQ==,9599,2019-06-23T16:04:37Z,2019-06-23T16:04:37Z,OWNER,"Another bug: JSON is being served without a content-type header:
```
~ $ curl -i 'http://127.0.0.1:8001/fivethirtyeight/ahca-polls%2Fahca_polls.json'
HTTP/1.1 200 OK
date: Sun, 23 Jun 2019 16:04:01 GMT
server: uvicorn
referrer-policy: no-referrer
transfer-encoding: chunked
{""database"": ""fivethirtyeight"", ""table"": ""ahca-polls/ahca_polls"", ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504765738,https://api.github.com/repos/simonw/datasette/issues/518,504765738,MDEyOklzc3VlQ29tbWVudDUwNDc2NTczOA==,9599,2019-06-23T16:11:49Z,2019-06-23T16:20:44Z,OWNER,"OK, for **Get ?_trace=1 working again**. The old code lives in two places:
https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L546-L560
And then:
https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L653-L672
So it's stashing something on the request to tell the rest of the code it should be tracing, then using that collected data from the request to add information to the final body.
One possible shape for the replacement is a new ASGI middleware that wraps everything else. We don't have a mutable request object here though, so we will need to untangle this entirely from the request object.
Also tricky is that in ASGI land we handle streams - we don't usually wait around for the entire response body to be compiled for us. This means the code that modifies the response (adding to the JSON or appending inside the `