html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/pull/518#issuecomment-504798977,https://api.github.com/repos/simonw/datasette/issues/518,504798977,MDEyOklzc3VlQ29tbWVudDUwNDc5ODk3Nw==,9599,2019-06-23T23:52:38Z,2019-06-23T23:52:38Z,OWNER,"Last thing is to replace `sanic.response`:
* `response.text("""")`
* `response.html()`
* `response.redirect(path)`
* `response.HTTPResponse`
Implementations here: https://github.com/huge-success/sanic/blob/0.7.0/sanic/response.py#L175-L285","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504795648,https://api.github.com/repos/simonw/datasette/issues/518,504795648,MDEyOklzc3VlQ29tbWVudDUwNDc5NTY0OA==,9599,2019-06-23T23:07:06Z,2019-06-23T23:07:06Z,OWNER,"For the request object.... what are the fields of it I actually use?
* `request.url`
* `request.query_string`
* `request.path`
* `request.method`
* `request.args`
* `request.raw_args`
ALL of those are things that can be derived from the `scope` - so I think my new `Request` class (in `utils/asgi.py`) is just going to be a wrapper around a `scope`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/issues/498#issuecomment-504793988,https://api.github.com/repos/simonw/datasette/issues/498,504793988,MDEyOklzc3VlQ29tbWVudDUwNDc5Mzk4OA==,9599,2019-06-23T22:40:25Z,2019-06-23T22:41:25Z,OWNER,"You can use a database query against `sqlite_master` to find all the FTS tables:
http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+name+from+sqlite_master+where+name+like+%22%25_fts%22+and+type+%3D+%22table%22
`select name from sqlite_master where name like ""%_fts"" and type = ""table""`
You could then construct a crafty UNION query to get results back from all of those tables at once:
http://sf-trees.datasettes.com/trees-b64f0cb?sql=select+%27PlantType_value_fts%27%2C+rowid%2C+value+from+PlantType_value_fts+where+PlantType_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCareAssistant_value_fts%27%2C+rowid%2C+value+from+qCareAssistant_value_fts+where+qCareAssistant_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qCaretaker_value_fts%27%2C+rowid%2C+value+from+qCaretaker_value_fts+where+qCaretaker_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qLegalStatus_value_fts%27%2C+rowid%2C+value+from+qLegalStatus_value_fts+where+qLegalStatus_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSiteInfo_value_fts%27%2C+rowid%2C+value+from+qSiteInfo_value_fts+where+qSiteInfo_value_fts+match+%3Asearch%0D%0A++union%0D%0Aselect+%27qSpecies_value_fts%27%2C+rowid%2C+value+from+qSpecies_value_fts+where+qSpecies_value_fts+match+%3Asearch&search=fi%2A
(I'm searching for `fi*` here to demonstrate wildcards)
The problem, as discussed earlier, is relevance: there's no way to compare the scores you're getting across different tables, so you won't be able to order by anything.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",451513541,
https://github.com/simonw/datasette/issues/514#issuecomment-504793379,https://api.github.com/repos/simonw/datasette/issues/514,504793379,MDEyOklzc3VlQ29tbWVudDUwNDc5MzM3OQ==,9599,2019-06-23T22:31:29Z,2019-06-23T22:31:48Z,OWNER,"I suggest trying a full path in `ExecStart` like this:
`ExecStart=/home/chris/Env/datasette/bin/datasette serve -h 0.0.0.0 /home/chris/digital-library/databases/*.db --cors --metadata /home/chris/digital-library/metadata.json`
That should eliminate the chance of some kind of path confusion.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459397625,
https://github.com/simonw/datasette/pull/518#issuecomment-504791053,https://api.github.com/repos/simonw/datasette/issues/518,504791053,MDEyOklzc3VlQ29tbWVudDUwNDc5MTA1Mw==,9599,2019-06-23T22:00:56Z,2019-06-23T22:00:56Z,OWNER,"The `InvalidUsage` exception is thrown by Sanic when it gets an unhandled request - usually a HEAD. It was added in efbb4e83374a2c795e436c72fa79f70da72309b8
I'm going to replace it with specific handling for HEAD requests plus a unit test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504790825,https://api.github.com/repos/simonw/datasette/issues/518,504790825,MDEyOklzc3VlQ29tbWVudDUwNDc5MDgyNQ==,9599,2019-06-23T21:57:58Z,2019-06-23T21:57:58Z,OWNER,"The big one: **Replace Sanic request and response objects with my own classes, so I can remove Sanic dependency**","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504782618,https://api.github.com/repos/simonw/datasette/issues/518,504782618,MDEyOklzc3VlQ29tbWVudDUwNDc4MjYxOA==,9599,2019-06-23T20:05:44Z,2019-06-23T20:05:59Z,OWNER,"**Replacement for @app.listener(""before_server_start"")** - this is what the [ASGI lifespan protocol](https://asgi.readthedocs.io/en/latest/specs/lifespan.html) is for.
I know Uvicorn supports this because it keeps saying `ASGI 'lifespan' protocol appears unsupported` on the console.
I think the solution here will be to introduce another ASGI wrapper class similar to `AsgiTracer`. I'll model this on the example in the ASGI lifespan spec.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/issues/520#issuecomment-504772599,https://api.github.com/repos/simonw/datasette/issues/520,504772599,MDEyOklzc3VlQ29tbWVudDUwNDc3MjU5OQ==,9599,2019-06-23T17:44:36Z,2019-06-23T17:44:36Z,OWNER,These plugins will need access to configuration and the ability to execute SQL - which means we need to make the `datasette` instance available to them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459598080,
https://github.com/simonw/datasette/issues/516#issuecomment-504768147,https://api.github.com/repos/simonw/datasette/issues/516,504768147,MDEyOklzc3VlQ29tbWVudDUwNDc2ODE0Nw==,9599,2019-06-23T16:43:23Z,2019-06-23T16:43:23Z,OWNER,"The Starlette lint and test scripts do this, and also apply autoflake to remove any unnecessary imports: https://github.com/encode/starlette/tree/8c8cc2ec0a5cb834a9a15b871ae8b480503abb67/scripts","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459509126,
https://github.com/simonw/datasette/pull/518#issuecomment-504765738,https://api.github.com/repos/simonw/datasette/issues/518,504765738,MDEyOklzc3VlQ29tbWVudDUwNDc2NTczOA==,9599,2019-06-23T16:11:49Z,2019-06-23T16:20:44Z,OWNER,"OK, for **Get ?_trace=1 working again**. The old code lives in two places:
https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L546-L560
And then:
https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/app.py#L653-L672
So it's stashing something on the request to tell the rest of the code it should be tracing, then using that collected data from the request to add information to the final body.
One possible shape for the replacement is a new ASGI middleware that wraps everything else. We don't have a mutable request object here though, so we will need to untangle this entirely from the request object.
Also tricky is that in ASGI land we handle streams - we don't usually wait around for the entire response body to be compiled for us. This means the code that modifies the response (adding to the JSON or appending inside the ``) needs to be reconsidered.
As usual, Starlette seems to have figured this out: https://github.com/encode/starlette/blob/8c8cc2ec0a5cb834a9a15b871ae8b480503abb67/starlette/middleware/gzip.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504765145,https://api.github.com/repos/simonw/datasette/issues/518,504765145,MDEyOklzc3VlQ29tbWVudDUwNDc2NTE0NQ==,9599,2019-06-23T16:04:37Z,2019-06-23T16:04:37Z,OWNER,"Another bug: JSON is being served without a content-type header:
```
~ $ curl -i 'http://127.0.0.1:8001/fivethirtyeight/ahca-polls%2Fahca_polls.json'
HTTP/1.1 200 OK
date: Sun, 23 Jun 2019 16:04:01 GMT
server: uvicorn
referrer-policy: no-referrer
transfer-encoding: chunked
{""database"": ""fivethirtyeight"", ""table"": ""ahca-polls/ahca_polls"", ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504765018,https://api.github.com/repos/simonw/datasette/issues/518,504765018,MDEyOklzc3VlQ29tbWVudDUwNDc2NTAxOA==,9599,2019-06-23T16:03:20Z,2019-06-23T16:03:20Z,OWNER,"Weird new bug: http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv?_format=json is downloading CSV for me now.
https://latest.datasette.io/fixtures/table%2Fwith%2Fslashes.csv?_format=json does the right thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/issues/498#issuecomment-504763919,https://api.github.com/repos/simonw/datasette/issues/498,504763919,MDEyOklzc3VlQ29tbWVudDUwNDc2MzkxOQ==,9599,2019-06-23T15:50:49Z,2019-06-23T15:50:49Z,OWNER,"One interesting way to approach this could be to do it entirely in JavaScript. I've had a lot of success building small apps on top of Datasette's JavaScript API - I wrote up one example here: https://24ways.org/2018/fast-autocomplete-search-for-your-website/
Once #272 is done I'll be adding a plugin hook that allows plugins to define entirely new pages within the Datasette application, which may also be a good way to work on this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",451513541,
https://github.com/simonw/datasette/pull/518#issuecomment-504762887,https://api.github.com/repos/simonw/datasette/issues/518,504762887,MDEyOklzc3VlQ29tbWVudDUwNDc2Mjg4Nw==,9599,2019-06-23T15:38:58Z,2019-06-23T15:38:58Z,OWNER,"Mystery solved: that's because I'm constructing my own scope object and testing via `ApplicationCommunicator` rather than exercising Uvicorn directly.
https://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/fixtures.py#L42-L57
I don't want to introduce the complexity of launching a real Uvicorn as part of the tests, so I guess I'll have to carefully update my `ApplicationCommunicator` test harness to more correctly emulate real life.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/pull/518#issuecomment-504762769,https://api.github.com/repos/simonw/datasette/issues/518,504762769,MDEyOklzc3VlQ29tbWVudDUwNDc2Mjc2OQ==,9599,2019-06-23T15:37:26Z,2019-06-23T15:37:26Z,OWNER,"This is strange: on my local machine `http://127.0.0.1:8001/fixtures/table%2Fwith%2Fslashes.csv` is returning a 404 BUT there's a test for that which is passing under pytest:
https://github.com/simonw/datasette/blob/d60fbfcae2658e71cab6d7b3b9f53f8d895064ef/tests/test_api.py#L721-L727","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459587155,
https://github.com/simonw/datasette/issues/272#issuecomment-504761039,https://api.github.com/repos/simonw/datasette/issues/272,504761039,MDEyOklzc3VlQ29tbWVudDUwNDc2MTAzOQ==,9599,2019-06-23T15:15:41Z,2019-06-23T15:18:36Z,OWNER,"And now the tests are all passing!
Still to do:
* Use `raw_path` so table names containing `/` can work correctly
* Get ?_trace=1 working again
* Replacement for `@app.listener(""before_server_start"")`
* Replace Sanic request object with my own request class, so I can remove Sanic dependency","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504761165,https://api.github.com/repos/simonw/datasette/issues/272,504761165,MDEyOklzc3VlQ29tbWVudDUwNDc2MTE2NQ==,9599,2019-06-23T15:17:07Z,2019-06-23T15:17:07Z,OWNER,I'm going to move the remaining work into a pull request.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504716988,https://api.github.com/repos/simonw/datasette/issues/272,504716988,MDEyOklzc3VlQ29tbWVudDUwNDcxNjk4OA==,9599,2019-06-23T03:43:46Z,2019-06-23T15:15:26Z,OWNER,"OK, it's beginning to shape up now. Next steps:
- [x] Static file support (including for plugins) - plus tests
- [x] Streaming support so the CSV tests will pass
- [x] Ability to download the database file
- [x] Implement missing-slash redirects
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504760061,https://api.github.com/repos/simonw/datasette/issues/272,504760061,MDEyOklzc3VlQ29tbWVudDUwNDc2MDA2MQ==,9599,2019-06-23T15:02:52Z,2019-06-23T15:02:52Z,OWNER,"Tests are failing on Python 3.5: https://travis-ci.org/simonw/datasette/jobs/549380098 - error is `TypeError: the JSON object must be str, not 'bytes'`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504759842,https://api.github.com/repos/simonw/datasette/issues/272,504759842,MDEyOklzc3VlQ29tbWVudDUwNDc1OTg0Mg==,9599,2019-06-23T15:00:06Z,2019-06-23T15:00:06Z,OWNER,I also need to actually take advantage of `raw_path` such that pages like https://fivethirtyeight.datasettes.com/fivethirtyeight/twitter-ratio%2Fsenators can be correctly served.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504759683,https://api.github.com/repos/simonw/datasette/issues/272,504759683,MDEyOklzc3VlQ29tbWVudDUwNDc1OTY4Mw==,9599,2019-06-23T14:57:50Z,2019-06-23T14:57:50Z,OWNER,"All of the tests are now passing!
I still need a solution for this:
https://github.com/simonw/datasette/blob/5bd510b01adae3f719e4426b9bfbc346a946ba5c/datasette/app.py#L706-L714
I think the answer is ASGI lifespan, which is supported by Uvicorn. https://asgi.readthedocs.io/en/latest/specs/lifespan.html#startup","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324188953,
https://github.com/simonw/datasette/issues/272#issuecomment-504754552,https://api.github.com/repos/simonw/datasette/issues/272,504754552,MDEyOklzc3VlQ29tbWVudDUwNDc1NDU1Mg==,9599,2019-06-23T13:53:39Z,2019-06-23T13:53:39Z,OWNER,"Next test to fix (because by new test harness doesn't actually obey the `allow_redirects=` parameter):
```
_____________ test_database_page_redirects_with_url_hash _____________
app_client_with_hash =