html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/pull/1000#issuecomment-705889120,https://api.github.com/repos/simonw/datasette/issues/1000,705889120,MDEyOklzc3VlQ29tbWVudDcwNTg4OTEyMA==,9599,2020-10-08T23:59:01Z,2020-10-08T23:59:01Z,OWNER,Needs tests and documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705890365,https://api.github.com/repos/simonw/datasette/issues/1000,705890365,MDEyOklzc3VlQ29tbWVudDcwNTg5MDM2NQ==,22429695,2020-10-09T00:03:29Z,2020-10-09T16:07:03Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=h1) Report > Merging [#1000](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/7249ac5ca04b5ddc6517750326ee7e522cc49145?el=desc) will **increase** coverage by `0.15%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1000/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1000 +/- ## ========================================== + Coverage 84.37% 84.52% +0.15% ========================================== Files 28 28 Lines 3871 3878 +7 ========================================== + Hits 3266 3278 +12 + Misses 605 600 -5 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.34% <100.00%> (+0.02%)` | :arrow_up: | | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `74.35% <100.00%> (ø)` | | | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.16% <100.00%> (-4.84%)` | :arrow_down: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.94% <100.00%> (+0.11%)` | :arrow_up: | | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.92% <0.00%> (ø)` | | | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `93.51% <0.00%> (+8.33%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=footer). Last update [7249ac5...8a80c79](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705899629,https://api.github.com/repos/simonw/datasette/issues/1000,705899629,MDEyOklzc3VlQ29tbWVudDcwNTg5OTYyOQ==,9599,2020-10-09T00:37:02Z,2020-10-09T00:37:02Z,OWNER,I'm going to route the existing `TestClient` through this mechanism to exercise it during the tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705902902,https://api.github.com/repos/simonw/datasette/issues/1000,705902902,MDEyOklzc3VlQ29tbWVudDcwNTkwMjkwMg==,9599,2020-10-09T00:50:49Z,2020-10-09T00:50:49Z,OWNER,"Almost all of the tests are passing: ``` =========================== short test summary info ============================ FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200 FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200 FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert... FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5] FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyE... FAILED tests/test_plugins.py::test_hook_register_magic_parameters - Assertion... ============ 11 failed, 718 passed, 6 warnings in 225.77s (0:03:45) ============ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705918844,https://api.github.com/repos/simonw/datasette/issues/1000,705918844,MDEyOklzc3VlQ29tbWVudDcwNTkxODg0NA==,9599,2020-10-09T01:46:06Z,2020-10-09T01:46:06Z,OWNER,"For this failing test I'm suspicious that the AsyncClient may be persisting cookies in between requests: ``` def test_actor_cookie(app_client): ""A valid actor cookie sets request.scope['actor']"" cookie = app_client.actor_cookie({""id"": ""test""}) response = app_client.get(""/"", cookies={""ds_actor"": cookie}) > assert {""id"": ""test""} == app_client.ds._last_request.scope[""actor""] E AssertionError: assert {'id': 'test'} == {'id': 'root'} E Differing items: E {'id': 'test'} != {'id': 'root'} E Use -v to get the full diff ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705920055,https://api.github.com/repos/simonw/datasette/issues/1000,705920055,MDEyOklzc3VlQ29tbWVudDcwNTkyMDA1NQ==,9599,2020-10-09T01:51:05Z,2020-10-09T01:51:05Z,OWNER,"The topic of disabling cookie persistence is discussed a little here: https://github.com/encode/httpx/issues/422#issuecomment-537906693 > We have a cookiejar abstraction, I think setting it to an always-empty jar like you describe is best. :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705920228,https://api.github.com/repos/simonw/datasette/issues/1000,705920228,MDEyOklzc3VlQ29tbWVudDcwNTkyMDIyOA==,9599,2020-10-09T01:51:44Z,2020-10-09T01:51:44Z,OWNER,I'm going to switch back to having each request run through a new client. I'm worried about the impact on test performance though. I'll run a microbenchmark before and after.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705921006,https://api.github.com/repos/simonw/datasette/issues/1000,705921006,MDEyOklzc3VlQ29tbWVudDcwNTkyMTAwNg==,9599,2020-10-09T01:55:01Z,2020-10-09T01:55:01Z,OWNER,"With the single client that is reused for all tests: ``` % time pytest tests/test_api.py ... 6.73s user 9.91s system 81% cpu 20.365 total ``` After switching back to this class: ```python class DatasetteClient: def __init__(self, ds): self.app = ds.app() def _fix(self, path): if path.startswith(""/""): path = ""http://localhost{}"".format(path) return path async def get(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.get(self._fix(path), **kwargs) async def options(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.options(self._fix(path), **kwargs) async def head(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.head(self._fix(path), **kwargs) async def post(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.post(self._fix(path), **kwargs) async def put(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.put(self._fix(path), **kwargs) async def patch(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.patch(self._fix(path), **kwargs) async def delete(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.delete(self._fix(path), **kwargs) async def request(self, method, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.request(method, self._fix(path), **kwargs) ``` The time taken is: ``` % time pytest tests/test_api.py ... 7.26s user 10.02s system 82% cpu 21.014 total ``` That's close enough that I don't feel I need to investigate this further.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705926035,https://api.github.com/repos/simonw/datasette/issues/1000,705926035,MDEyOklzc3VlQ29tbWVudDcwNTkyNjAzNQ==,9599,2020-10-09T02:14:14Z,2020-10-09T02:14:14Z,OWNER,"Still need to handle these six failing tests: ``` FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'elemen... FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', '... FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<... FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '

FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages' That one is caused by `response.cookies` skipping cookies that were set to the empty string. Same fix as this: https://github.com/simonw/datasette/blob/a1687351fb75b01f737fda4ad07e0781029de05c/tests/test_auth.py#L90-L95","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705937696,https://api.github.com/repos/simonw/datasette/issues/1000,705937696,MDEyOklzc3VlQ29tbWVudDcwNTkzNzY5Ng==,9599,2020-10-09T02:52:53Z,2020-10-09T02:52:53Z,OWNER,"These failures are giving me a severe ""how did this ever work in the first place?"" vibe: ``` FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] ``` I have a fix for them, no idea why they weren't already failing though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705940507,https://api.github.com/repos/simonw/datasette/issues/1000,705940507,MDEyOklzc3VlQ29tbWVudDcwNTk0MDUwNw==,9599,2020-10-09T03:04:15Z,2020-10-09T03:04:15Z,OWNER,"This is really weird: new set of test failures that I wasn't seeing before, and those tests aren't failing on my laptop: ``` =========================== short test summary info ============================ FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200 FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200 FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert... FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5] FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] ================== 5 failed, 738 passed in 194.73s (0:03:14) =================== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705941580,https://api.github.com/repos/simonw/datasette/issues/1000,705941580,MDEyOklzc3VlQ29tbWVudDcwNTk0MTU4MA==,9599,2020-10-09T03:08:43Z,2020-10-09T03:08:43Z,OWNER,"Most likely reason for those failures is that `path` and `raw_path` are not being simulated correctly. I used to do those here: https://github.com/simonw/datasette/blob/402cf870b7d65f9b5fba9e23aa99433294bd4523/datasette/utils/testing.py#L116-L125 But now I'm delegating that to `httpx` to handle. WEIRD that it passes on my laptop but fails in GitHub Actions CI though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705945591,https://api.github.com/repos/simonw/datasette/issues/1000,705945591,MDEyOklzc3VlQ29tbWVudDcwNTk0NTU5MQ==,9599,2020-10-09T03:24:48Z,2020-10-09T03:24:48Z,OWNER,"I'm testing this with a `print(scope)` and `pytest -k test_table_with_slashes_in_name -s`. Against the `main` branch: `{'type': 'http', 'http_version': '1.0', 'method': 'GET', 'path': '/fixtures/table/with/slashes.csv', 'raw_path': b'/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'headers': [[b'host', b'localhost']], 'csrftoken': ._asgi_csrf_decorator..app_wrapped_with_csrf..get_csrftoken at 0x10e2e6040>}` Against the broken branch: `tests/test_api.py {'type': 'http', 'asgi': {'version': '3.0'}, 'http_version': '1.1', 'method': 'GET', 'headers': [(b'host', b'localhost'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'user-agent', b'python-httpx/0.15.0')], 'scheme': 'http', 'path': '/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'server': ('localhost', None), 'client': ('127.0.0.1', 123), 'root_path': '', 'csrftoken': ._asgi_csrf_decorator..app_wrapped_with_csrf..get_csrftoken at 0x109e0eca0>}` This is on my laptop though so both of those pass the tests. Key difference: the `httpx` version doesn't set a `raw_path` at all. BUT.. it does set `path` and sets it to `'/fixtures/table%2Fwith%2Fslashes.csv'` The non-httpx version sets `raw_path` to `b'/fixtures/table%2Fwith%2Fslashes.csv'` and `path` to `'/fixtures/table/with/slashes.csv'`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705946120,https://api.github.com/repos/simonw/datasette/issues/1000,705946120,MDEyOklzc3VlQ29tbWVudDcwNTk0NjEyMA==,9599,2020-10-09T03:27:05Z,2020-10-09T03:27:05Z,OWNER,"I may need to fuss around with how the `httpx` client sends things to the ASGI app. https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L26 is relevant: Alternatively, you can setup the transport instance explicitly. This allows you to include any additional configuration arguments specific to the ASGITransport class: ``` transport = httpx.ASGITransport( app=app, root_path=""/submount"", client=(""1.2.3.4"", 123) ) client = httpx.AsyncClient(transport=transport) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-705946360,https://api.github.com/repos/simonw/datasette/issues/1000,705946360,MDEyOklzc3VlQ29tbWVudDcwNTk0NjM2MA==,9599,2020-10-09T03:28:08Z,2020-10-09T03:28:08Z,OWNER,"Here's where `httpx` sets up the ASGI scope: https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L82-L97 ```python # ASGI scope. scheme, host, port, full_path = url path, _, query = full_path.partition(b""?"") scope = { ""type"": ""http"", ""asgi"": {""version"": ""3.0""}, ""http_version"": ""1.1"", ""method"": method.decode(), ""headers"": [(k.lower(), v) for (k, v) in headers], ""scheme"": scheme.decode(""ascii""), ""path"": unquote(path.decode(""ascii"")), ""query_string"": query, ""server"": (host.decode(""ascii""), port), ""client"": self.client, ""root_path"": self.root_path, } ``` Sure enough, it doesn't set the `raw_path`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-706263157,https://api.github.com/repos/simonw/datasette/issues/1000,706263157,MDEyOklzc3VlQ29tbWVudDcwNjI2MzE1Nw==,9599,2020-10-09T15:57:15Z,2020-10-09T15:57:15Z,OWNER,"My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet. I'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043, https://github.com/simonw/datasette/pull/1000#issuecomment-706269271,https://api.github.com/repos/simonw/datasette/issues/1000,706269271,MDEyOklzc3VlQ29tbWVudDcwNjI2OTI3MQ==,9599,2020-10-09T16:08:49Z,2020-10-09T16:08:49Z,OWNER,I'm going to document this in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717746043,